DualContrast: Unsupervised Disentangling of Content and Transformations with Implicit Parameterization

Authors: Mostofa Rafid Uddin, Min Xu
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Unsupervised Learning, Shape Analysis, Identifiability in Representation Learning, Disentangled Representation Learning, ML in Biology

Abstract:
Unsupervised disentanglement of content and transformation is significantly important for analyzing shape focused scientific image datasets, given their efficacy in solving downstream image-based shape-analyses tasks. The existing relevant works address the problem by explicitly parameterizing the transformation latent codes in a generative model, significantly reducing their expressiveness. Moreover, they are not applicable in cases where transformations can not be readily parametrized. An alternative to such explicit approaches is contrastive methods with data augmentation, which implicitly disentangles transformations and content. However, the existing contrastive strategies are insufficient to this end. Therefore, we developed a novel contrastive method with generative modeling, DualContrast, specifically for unsupervised disentanglement of content and transformations in shape focused image datasets. DualContrast creates positive and negative pairs for content and transformation from data and latent spaces. Our extensive experiments showcase the efficacy of DualContrast over existing self-supervised and explicit parameterization approaches. With DualContrast, we disentangled protein composition and conformations in cellular 3D protein images, which was unattainable with existing disentanglement approaches.

Decision: reject
Reviewer scores: 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730121989051

thread_type:
official_review

<thread_object>
creator: reviewer_aqpg

<content>
summary:
In this paper, the authors propose a model that learn unsupervised representation for  "shape-focused images". In particular, their method, DualContrast, learn to disentangle "content" and "transformation" in an unsupervised fashion. The model is trained with an a combination of 2 contrastive losses (one for context, one for transformations) and a VAE loss (the VAE is used to sample positive samples for the CL of transformations). The authors show results of the proposed model on multiple small/toysh datasets.

strengths:
+ The paper is well written and easy to follow
+ The idea of disentangling features is an important problem in many applications of machine learning
+ The proposed approach is simple and well motivated

weaknesses:
-  The author often mention that the work focus on "shape-focused real-world images", but they only applied in very simplified, toysh settings, very far from "real-world images". Even the CryoEM task is a very simplified task.
- The choice of positive/negative samples for each factor is very ad-hoc. The paper lacks explanation and empirical validation on why this choice makes sense versus others. 
- I found very strange the choice of using VAE generated samples as data to train the contrastive loss. This idea of using generated smples to train a model is not well understood. This approach might have worked in the very simplified tasks tested on the paper, but It is very unlikely that the proposed model would work on any real-world dataset.
- I also find the experimental results a bit weak. First, the datasets utilized in this work are very simple and results on them probably wont guarantee their utility on real-world problems. Second, the metrics utilized on Table 1 are not particularly significant. Third, most of the results are qualitative and based on one or two images that can potentially be cherry picked.

questions:
- On L147, the authors say that they model "is highly effective in disentangling wide range of transformations from the content in various shape-focused image datasets by only using simple rotation for creating contrastive pairs since the representation for disentangled rotation generalizes over other shape transformations." Could they elaborate on this? Why is it the case? Where is it shown on the paper? How can we be sure it will work to other modalities besides the toy tasks tested?
- Is the VAE trained at the smae time as the contrastive losses? Since the VAE is used to generated samples for the CLs, how do training jointly vs training in two stages (VAE followed by CLs) change the performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587458718

thread_type:
official_review

<thread_object>
creator: reviewer_uj45

<content>
summary:
Unsupervised disentanglement of transformations and content is a challenging task that was previously approached primarily through using separate ad-hoc transformation methods, or by self-supervised contrastive-based methods. Ad-hoc transformations suffer from being limited to the given parameterization chosen, while self-supervised methods do not tackle this disentanglement problem directly. In this work, DualContrast is proposed, which consists of a VAE with additional contrastive losses designed to disentangle content and transformation. The hardest challenge is obtaining positive pairs of samples with respect to transformations: changing the content while keeping the transformation constant. In this work, this has been done by decoding two random samples from the prior of the transformation latent space while feeding different permutations of the content latent representation to obtain similar transformations with different content. The method is applied to MNIST, LineMod, Starmen Shapes, and Cryo-ET subtomograms with positive results.

strengths:
Originality.
In this work, a novel method to address the problem of creating positive pairs of transformations under content change has been proposed. The core of this work is original.

Quality.
The method proposed was evaluated on a sufficient number of datasets. Although not very complex, they could suffice in showing the potential for this approach. The baselines chosen are also relevant to the method proposed.

Clarity.
The figures in the experiment section allow for a quick qualitative assessment of the performance of the methods. The method explanation is quite clear.

Significance.
This work and the proposed method have shown some potential for successful applications.

weaknesses:
The explanation of the method in the abstract and introduction is especially unclear. This is also a problem because Figure 2 fails to properly and intuitively show the method. Reading the method section explains this more. To improve, I would suggest clearly highlighting  the role of the latent space in the creation of the positive pair that would otherwise be impossible. This could be done similarly to how it was done in Figure 3. Figure 2 would then become useful. Additionally, Figure 2 lacks proper annotations such as labeling of all elements present, and proper caption explaining what happens in the figure in a more complete way. There is some inconsistency in how things are called. In the figure, style, and content are mentioned, however, in the text it is clear that "style" is supposed to be "transformation", please pick one and stick with it in the whole manuscript, either one would suffice, however, transformation is likely to be more accurate.

The contributions are a bit bold. The first contribution, especially, is more context for the work than a contribution and could be removed entirely. Please consider reworking the contributions to be more reflective of the actual content.

The related work section should expand a bit more on the protein part, which is currently very unclear for somebody who is not a practitioner. Please provide more examples, even referring to the appendix to understand the data and the context better.

The method section has a few mistakes and the explanation is very wordy, which makes it hard to follow. I wrote a few observations in the questions section of this review.

The manuscript should include the limitations of this method, especially regarding the latent space-based approach to creating positive transformation pairs. For example, the limitations should address whether this approach could be extended to real-world datasets or whether this approach should be limited to specific types of datasets. 

The experiments lack in quantitative results. Although disentanglement is very hard to measure, given the ability to choose datasets, it would be much more convincing to have datasets where a quantitative assessment is possible either in the form of direct supervision (similar to what the disentanglement metric is currently doing), or through some downstream tasks where the disentanglement would be useful. Such tasks could be segmentation, or visual question answer.
The choice of the "human deformation" as a dataset is very confusing, and the results reported are also very underwhelming. Although the generated shapes are better, the data appears to be very trivial, so some more information on the training and the difficulty of fitting such samples would be more convincing. The number of parameters, ablation performed, and results from more baselines would be a step in the right direction. It is especially important to keep including all baselines for all datasets used, the results on human deformation appear to be unfinished. If so, it would have been better to simply exclude the dataset from the manuscript.
Additionally, plots of the latent space are only available for the cellular dataset

questions:
In the method section, condition 1 is very confusing. I think it was meant to be "for all $T \in T$ and $x \in X, h_c(T(x)) = h_c(x)$", but please correct me if I misunderstood this.

In the method section, many terms are used seemingly interchangeably, such as "latent space", "factor", "representation", "transformation", "content". Please clarify these terms. For example, at line 206, "transformation" is used, however, I think it was meant to be "transformation representation" or "transformation factor", unless I misunderstood.

There are a few grammatical and syntactical mistakes, such as inconsistencies in the use of uppercase and lowercase, sometimes writing "shape focused" while other times "shape-focused".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730964353940

thread_type:
official_review

<thread_object>
creator: reviewer_b3mr

<content>
summary:
This paper proposes an unsupervised disentangling method to disentangle content and transformation of the input. Specifically, this paper first proposes two conditions that disentanglement of content and transformation should satisfy. Then, this paper proposes a method to construct positive and negative samples with respect to both content and transformation. The key idea is to utilize a variational autoencoder to construct these samples. The experiments are conducted on four datasets, i.e., three of them (mnist, linemod, and starmen) are pure images, and one is protein subtomogram. One quantitative result and several qualitative results are shown to prove the effectiveness of the proposed method.

strengths:
1. This paper proposes well-defined conditions for the disentanglement of content and transformation. 
2. The experiments are conducted on four datasets, and comprehensive qualitative results are shown.

weaknesses:
The main concern of this paper is evaluation, which is insufficient and less significant.
1. The first three datasets (mnist, linemod, starmen) are somehow toy datasets, which is less significant in real-world applications.
2. I agree that protein conformation is one meaningful real-world application, but other than map visualizations, it fails to produce convincing evaluation results.
3. There lacks some widely used evaluating metrics in Table 1 to demonstrate the application of the disentanglement.
4. This paper also does not provide comparisons with other baseline methods or state-of-the-art methods.

--------------------------------------

Post-rebuttal:
Thanks for the authors' response. The revision includes more quantitative results and qualitative comparisons, which is appreciated. These additions partially resolved my concerns. However, I noticed that these compared methods are still up to date in 2022, which cannot be considered state-of-the-art to some extent. 

Another issue to mention is that the metrics D(c|c) and D(c|z) were used in the initial submission, and the protein experiments are the most important one (in my personal view); it is usually discouraged not to report the main quantitative results of the critical experiments in the initial submission, but added during rebuttal.

Considering the rebuttal and revision, I raised my rating to 5. Please use it sparsely.

questions:
Please refer to the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732784773789

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank the area chairs and reviewers for their efforts in reviewing our paper and providing suggestions. Their helpful comments have greatly enhanced our work.

We have revised the manuscript according to the reviewers' suggestions and marked them in blue in the new version of the manuscript. The main differences include:

1. **We included an additional Fig. 6 demonstrating convincing application-specific evaluation results of DualContrast.** The figure clearly shows how DualContrast can identify subtle conformational changes from a protein mixture cryo-ET subtomogram dataset, highlighting its significance in real-world applications. 

2. **We included additional evaluation metrics in Table 1.** In the table, we also reported values for all the evaluation metrics for the protein subtomogram dataset.

3. **We added a Discussions & Limitations section (Section 5)**, discussing when DualContrast is expected to work and when not. We observed that the method disentangles the transformations, causing small pixel-space changes, e.g., subtle conformational changes in proteins, viewpoint changes in LineMod, etc. Identifying subtle changes is vital in scientific image datasets, and our method applies to such cases. However, it is not expected to disentangle transformations causing significant pixel-space changes or transformations not present in the dataset, which is also not feasible in a completely unsupervised manner. We discussed these issues in the newly added section. 

4. As suggested by Reviewer UJ45, **we excluded the StarMen dataset** and all its results from the manuscript to maintain consistency of evaluation across all the datasets and the page limit. Moreover, all the reviewers regarded the dataset as insignificant, so we used the space to demonstrate highly significant subtomogram results (Fig. 6). 

5. **We changed Fig. 2 to make it more comprehensive and self-explanatory**. Consequently, we felt that the previous Fig. 3, which showed contrastive pair-making for a batch of MNIST digits, was optional in the main manuscript, so we moved it to the Appendix as Fig. 9.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733287987006

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarifying Remarks

comment:
Dear reviewers and chairs,

We again greatly appreciate your time and effort in reviewing our work and providing suggestions. Since the deadline for authors' comments is today, we would like to mention several clarifying remarks regarding the work. This may resolve several confusions regarding the work. 

**Type of contribution- Fundamental vs Incremental**: We view our work as a fundamental contribution rather than an incremental one. In cryo-EM/ET, many incremental works focus on tasks like particle picking or determining the location of protein complexes from raw images, aiming to improve state-of-the-art performance. In contrast, our work addresses a novel task: disentangling content and transformations where the transformations, such as conformational changes, lack a well-defined parametric form. This is similar to works like SpatialVAE [1], which introduced the task of disentangling content from 2D rotations and translations, and Harmony [2], which extended this to disentangling content from parameterized transformations, including 2D and 3D affine transformations. By addressing this new challenge, our work expands the boundaries of what is achievable in the field.

**Takeaway from cryo-ET results**: The main takeaway from the cryo-ET part of this paper is the demonstration that protein complexes with varying compositions and conformations can be identified from a collection of cryo-ET subtomograms in an unsupervised manner- a capability that previous methods lacked. Earlier approaches were limited to either identifying a few distinct protein complexes from a collection of images or analyzing a few conformations of a single protein complex within a dedicated dataset. However, when presented with collections containing multiple distinct protein complexes and multiple conformations, these methods could, at best, identify only a subset of the complexes and failed entirely to distinguish their conformations. This limitation is clearly illustrated in Fig. 6.


**Apart from cryo-ET, use of simple datasets- MNIST and LineMod**: We set out to disentangle the composition (semantic content) and conformation (transformations causing subtle voxel-level changes) of protein complexes in cryo-ET subtomogram datasets by framing the problem as an unsupervised content-transformation disentanglement task. These datasets present protein complexes in diverse 3D poses, with varying compositions and conformations, making the task inherently complex. To address this, we adopted a strategy of starting with simpler datasets, such as MNIST and LineMod (please note starmen was excluded in the revised paper), which share similar problem contexts. This strategy is consistent with prior works. For instance, SpatialVAE [1] and Harmony [2], while targeting content-transformation disentanglement in cryo-EM and cryo-ET, first validated their methods on MNIST. Similarly, NAISR [3], designed for interpretable shape analysis in medical imaging, initially tested its approach on the starmen dataset to establish its effectiveness on simpler cases. Following this strategy, we demonstrated our method’s success on MNIST and LineMod before applying it to cryo-ET subtomograms for the final experiments. 

[1] Bepler, Tristan, et al. "Explicitly disentangling image content from translation and rotation with spatial-VAE." Advances in Neural Information Processing Systems 32 (2019).

[2] Uddin, Mostofa Rafid, et al. "Harmony: a generic unsupervised approach for disentangling semantic content from parameterized transformations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Jiao, Yining, et al. "$\texttt {NAISR} $: A 3D Neural Additive Model for Interpretable Shape Representation." The Twelfth International Conference on Learning Representations.

- ICLR 2025 Conference Submission7350 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper utilizes a VAE with dual latent spaces for disentangling of content and transformations of cellular 3D protein images. Apart from standard VAE loss, the novel part is contrastive learning loss based on positive and negative pairs in terms of content and transformation. The method here is unsupervised, and those positive and negative pairs are generated without supervision or labels. Clustering of content and transformation codes clearly shows the effectiveness of the method as shown in Fig. 6 in the revised version.
However, reviewers were concerned about real-world impact on downstream tasks. Clustering of latents is another way of visualization for the latents, but not really an application with a profound impact. (There might be a knowledge gap for the reviewers and AC who do not work on biological science).
Another weakness is the lack of comparison to more recent methods on disentangled representation learning as mentioned by the reviewer. Another metric was added in the rebuttal, but overall the amount of quantitative results seems insufficient to be convincing.
The way to construct positive pairs and negative pairs is highly unreliable. Randomly selecting pairs of samples can lead to positive pairs with the same content. Also, it is not clear how to bootstrap the joint network to produce synthetic images with the same content but different transformations, in particular at the beginning of training when the network cannot produce images with high fidelity.

The authors are encouraged to improve the empirical significance of the method and resubmit it to the next conference or to a more domain-specific venue where the impact can be better appreciated.