Supervised Disentanglement Under Hidden Correlations

Authors: Rong Hu, Jiawei Chen, Ling Chen
Primary area: learning theory
Keywords: Disentangled representation learning, Supervised representation learning, Mutual information, Causal graph analysis, Hidden Correlations

Abstract:
Disentangled representation learning (DRL) methods are often leveraged to improve the generalization of representations. Recent DRL methods have tried to handle attribute correlations by enforcing conditional independence based on attributes. However, the complex multi-modal data distributions and hidden correlations under attributes remain unexplored. Existing methods are theoretically shown to cause the loss of mode information under such hidden correlations. To solve this problem, we propose Supervised Disentanglement under Hidden Correlations (SD-HC), which discovers data modes under certain attributes and minimizes mode-based conditional mutual information to achieve disentanglement. Theoretically, we prove that SD-HC is sufficient for disentanglement under hidden correlations, preserving mode information and attribute information. Empirically, extensive experiments on one toy dataset and five real-world datasets demonstrate improved generalization against the state-of-the-art baselines. Codes are available at anonymous Github https://anonymous.4open.science/r/SD-HC.

Decision: reject
Reviewer scores: 6, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730533410160

thread_type:
official_review

<thread_object>
creator: reviewer_hxxe

<content>
summary:
This paper addresses disentangled representation learning for data with correlated, multimodal attributes and proposes Supervised Disentanglement under Hidden Correlations (SD-HC), which infers hidden modes within attributes to achieve disentanglement while preserving essential information by minimizing mode-based conditional mutual information (CMI). The proposed approach is validated through theoretical analysis and experiments on illustrative toy data, image data (colored MNIST), and time series datasets (UCI-HAR, RealWorld, HHAR, and MFD), showing improved performance over existing methods.

strengths:
(Disclosure: I've reviewed this paper before, and I used a PDF comparison tool to check the differences.)

- This paper studies a challenging problem in disentangled representation learning.
- The motivating example in Figure 1 is easy to follow and highlights the need for this method. It also links to the experiments on the human activity recognition task in Section 5.
- The symbols and technical terms are properly defined. The definitions of disentangled data generation process and disentangled representations are clearly stated in Definitions 1 and 2.
- The paper provided sufficient empirical evidence on both image and time series data.

weaknesses:
- The connections between the theory and learning are still unclear to me. A better way to characterize the proposed method is to say something like "Under x assumptions, given x supervision, when x loss is optimized or x condition is satisfied, the learned representations are disentangled in the sense of definition x".
- The propositions and corollaries can be written in a more self-contained manner, even if the symbols have been defined in the text above.

questions:
- (minor) Why renaming? The current name SD-HC sounds more like a problem setting, not a specific method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693993174

thread_type:
official_review

<thread_object>
creator: reviewer_wrrv

<content>
summary:
The paper introduces a method called Supervised Disentanglement under Hidden Correlations (SD-HC) to improve disentangled representation learning (DRL). Traditional DRL methods struggle with hidden correlations in multi-modal data. SD-HC addresses this by discovering data modes under certain attributes and minimizing mode-based conditional mutual information. This approach ensures disentanglement while preserving mode and attribute information. The paper provides theoretical proofs and empirical evidence showing SD-HC outperforms existing methods on various datasets.

strengths:
1. The studied problem is open and important -- latent confounding is prevalent and proper efforts are deserved to this problem.
2. The experiments are extensive and the proposal looks promising.

weaknesses:
1. The paper may benefit from a clear introduction of the task, especially the instantiation with real-world examples. Although I appreciate several examples present in the paper, a coherent running example would aid the readability much more if it includes the objective of the task, the problem of the hidden correlation, and the instantiation of the proposal in this example.
2. The graphical model Figure 2 is not well-defined for me. $z$ variables are functions of the observed data $x$. How would they appear in the true data-generating process? To me, this is not even a causal graph -- if I intervene on attribute $a_{i}$ in the true data-generating process, this influences will not be blocked by estimated variables $z$'s.
3. The main arguments are elusive to me. Especially, what does Proposition 2 imply? Its connection to the disentanglement is unclear. To me, it states that if you can find the modality variable $m_{1}$ and condition on it, the associated estimated variable $z_{1}$ will be d-separated from other attributes $a_{-1}$. First, there is no guarantee to discover the true modality variable $m_{1}$ (the proposal is somehow heuristic); second, how would this lead to the disentanglement conclusion?

questions:
Please refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730739879480

thread_type:
official_review

<thread_object>
creator: reviewer_dyti

<content>
summary:
This paper addresses the problem of disentangled representation learning in the presence of hidden correlations, which are common in real-world data. The authors consider a multi-modal data distribution where certain attributes may induce complex modes correlated with other attributes. To tackle this issue, they first investigate the necessary and sufficient conditions for supervised disentanglement under the proposed data-generating process. Subsequently, they introduce the SD-HC method for learning disentangled representations with hidden correlations. The proposed method is validated on both synthetic data and five real-world datasets.

strengths:
1. The paper studies an important problem in disentangled representation learning by considering hidden correlations, which are prevalent in real-world data.
2. The experimental evaluation is thorough, including both synthetic and real-world datasets.

weaknesses:
1. The major issue lies in the lack of detailed theoretical explanation:
   - The causal graph presented in Figure 3 is confusing. Since $z = f(x)$ is a function of $x$, there should be a directed edge from $x$ to $z$. However, the graph shows $z$ as a parent of $x$, which is misleading.
   - The claim that "mode-based conditional mutual information (CMI) minimization is the necessary and sufficient condition for supervised disentanglement" lacks a detailed formal list of necessary assumptions. This claim heavily depends on the data-generating process, and the specific assumptions should be clearly highlighted.
   - The link between Proposition 2 and the independence requirement in Definition 2 is unclear. Proposition 2 appears to be a mutual information result that is not directly related to causality. Although the authors attempt to bridge this gap in Lines 242 to 244, a more rigorous analysis with detailed assumptions and proofs is needed.
2. The mode label estimation step is confusing. In this step, mode labels are estimated by clustering on the corresponding attribute. Consequently, there would be no confounders that affect $m_k$ and $a_{-k}$ while not affecting $a_k$, which violates the assumptions depicted in Figures 2 and 3.
3. The paper lacks detailed explanations regarding the choice of attributes and the modes in the real-world datasets. Providing more context would help in understanding the experimental results better.
4. There are typos in Lines 332 and 336. "Figure 7a" and "Figure 7b" should be replaced with "Figure 5a" and "Figure 5b," respectively.

questions:
See the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730794719607

thread_type:
official_review

<thread_object>
creator: reviewer_uijj

<content>
summary:
The paper introduces a novel supervised disentanglement method, SD-HC, designed to address hidden correlations in multi-modal data, which traditional DRL methods often overlook. SD-HC focuses on preserving essential mode information within attributes by minimizing mode-based conditional mutual information, thus achieving better disentanglement and predictive accuracy under both hidden and attribute correlations. The authors theoretically prove that mode-based conditional mutual information minimization is both necessary and sufficient for effective disentanglement in complex, correlated datasets.

strengths:
- This paper establishes a novel theory for supervised disentanglement with hidden correlations. It consists of both necessary and sufficient conditions, serving as a significant improvememt for existing works.
- The experiments are comprehensive and the results are good enough.

weaknesses:
- The motivation of the problem setting is not clearly discussed. In Figure 2, to study the hidden correlations among attributions, why not directly study $a_k$ and $a_{-k}$? The role or advantage of introducing the variable $m_k$ is not clear for me.
- It seems that the performance of SD-HC heavily relies on the mode label estimation. First, in real applications, how to determine the number of modes $N_m$? In addition, for complex data pattern, the data representations are not high-quality and the clutering is not accurate accordingly. In such case, is there any techniques to guarantee good performance?
- There are some ambiguous concepts in this paper. For example, 1) "Data mode". I do not find its concrete definition. This term is not common in machine learning and its meaning varies with contexts. 2) "Multi-modal". It seems not to refer to the commonly used meaning, such as text and image. More strict definitions is required to make this paper more clear.
- The writting needs to be further improved. For example, in line 155, $c^a$ and $c^m$ should be boldface.

questions:
- From Section 3 and 4, only one attribution mode is considered. For more complex or practical scenarios, all attributions may have their mode variables. Whether the SD-HC can efficiently and effectively address them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732277258712

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions and Global Response

comment:
We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for us to improve the clarity and rigor of our paper.

**Towards handling hidden correlations under certain attributes in disentangled representation learning**, our paper proposes Supervised Disentanglement under Hidden Correlations (SD-HC), which discovers data modes under certain attributes and minimizes mode-based conditional mutual information to achieve disentanglement. SD-HC is equipped with both **solid theoretical guarantees from the perspective of mutual information and causality**, and brings **promising experimental results on synthetic data and real-world image and time series data**. In addition, SD-HC can be **flexibly** tailored when different numbers of attributes need to be disentangled, and applies to **multiple attributes, various correlation types, and simple uncorrelated and uni-modal cases**. 

We are delighted that the reviewers generally held positive opinions of our paper, in that the investigated problem is "**challenging**", "**open**", "**important**" and "**prevalent in real-world data**" (Reviewer dyTi, wrRV, and HxXe), the proposed method and theory are "**properly defined**", "**clearly stated**", "**novel**", "**serving as a significant improvement for existing works**" (Reviewer UiJJ and HxXe),  the empirical evaluation is "**thorough**","**sufficient**", "**extensive**" and "**promising**" (Reviewer UiJJ, dyTi, wrRV, and HxXe).

The reviewers also raised insightful and constructive concerns. We have made every effort to address all the concerns by providing detailed clarifications, sufficient evidence, and in-depth analysis. We have uploaded the latest revised version, with **blue** markings indicating new or modified content. Here is the summary of the major revisions:

* **Enhance the rigor, completeness, and clarity of our theoretical results (Reviewers UiJJ, dyTi, wrRV)**: Our main theoretical results and contributions remain unchanged. We further enhance the proposed theory by adding definitions, differentiating true and learned representations, highlighting assumptions, and adding one more proof for rigor, completeness, and clarity throughout Section 3.
* **Better connect our theoretical results with practice (Reviewer UiJJ, dyTi, HxXe)**: We highlight the connections between learning and theory in "Theoretical Contributions", Section 3.3, and add generalization discussions to simple cases in "Generalization to Multiple Attributes and Simple Cases", Section 3.3, with more discussions added in Appendix B.5. In addition, we add toy experiments with multiple attributes and complex multi-modality and correlations to validate our method in complex scenarios, which will be added to our paper. These clarify how to implement our learning theory, and in what scenarios are they applicable.
* **Add practical guidance on the choice of clustering methods and mode estimation methods (Reviewer UiJJ)**: As the clustering method is a choice of design in our pre-training stage, we add guidance on how to choose appropriate clustering and mode estimation methods in "Clustering Evaluation" and "Parameter Sensitivity of $N_m$", Appendix I as mentioned in "Mode Label Estimation", Section 4.
* **Add the reasons for the settings on real-world data (Reviewer dyTi)**: We add explanations on the real-world phenomena that support our choices of attributes and modes on each real-world dataset.
* **Add coherent examples (Reviewer wrRV)**: We add examples in "Intuitive Example", Section 3.1. Along with the explanations on real-world datasets, the coherent examples in the same context of activity recognition are more complete and aid the understanding of our method.
* **Correcting a few typos and refining some expressions (Reviewer dyTi, HxXe)**: We check our paper thoroughly and correct a few typos regarding the symbols and figure references. Also, we refined some expressions to be more concise and save space for the added content.

The valuable suggestions from the four reviewers are highly valued. **We eagerly await your feedback and will gladly answer further questions.**

Best Regards!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733233541031

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of discussion

comment:
We would like to provide a summary of the discussion phase to offer greater clarity.

The four reviewers generally recognized the strengths of SD-HC as follows:

* **Importance of the studied problem**. **Reviewers dyTi, wrRV, and HxXe** agreed that SD-HC studies an **important and realistic** problem for disentangled representation learning, i.e., the disentanglement of attributes with multi-modality and hidden correlations.
* **Theoretical contributions**. Although **Reviewer UiJJ** raised some doubts about our problem setting, Reviewer UiJJ recognized our theoretical results as "novel" and "a significant improvement for existing works". Specifically, based on conditional mutual information, we give the *necessary and sufficient conditions* for disentanglement under hidden correlations and attribute correlations, which is also the **first** sufficient condition for disentanglement under correlations. Our results can be generalized to various cases and provide theoretical foundations for various disentangling methods with conditional independence.
* **Empirical soundness**. **ALL reviewers** agreed that our comprehensive experiments on toy data and real-world *image and time series* datasets provide sufficient evidence to support the advantage of SD-HC. In addition to the manually introduced correlations as commonly studied, we experiment under **natural hidden correlations** within real-world data, **bringing our theoretical results into diverse real applications**. Extensive analyses are carried out to study the behaviors of different methods, the impact of various factors, and the learned representation distributions and decision boundaries.

While the reviewers recognized the above strengths of SD-HC, they also expressed some concerns:

* **Reviewer UiJJ** felt that the problem setting and mode-related concepts need to be further clarified, and also asked about the effectiveness of SD-HC under complex scenarios when mode label estimation is challenging and when multiple attributes exhibit multi-modality.

* **Reviewer dyTi** mainly required more theoretical explanations, including the meaning of representations $z$ in Figure 3, the assumptions, and the link between our theoretical results to disentanglement, and also required explanations regarding the mode label estimation step and experiment settings on real-world datasets.

* **Reviewer wrRV** raised similar concerns as Reviewer dyTi regarding the meaning of $z$ in our causal graphs and the link to disentanglement, and raised similar concerns as Reviewer UiJJ about the effectiveness of SD-HC under potential mode label estimation errors. More coherent examples were also required to aid readability.

* **Reviewer HxXe** felt that the connections between theory and learning could be better established, and the theoretical results could be more self-contained.

During the discussion phase, we diligently and thoroughly addressed each reviewer's concerns. The revisions are summarized in our **global response**.

* **Theoretically**, we added formal definitions of mode, differentiated true and learned representations $z$, highlighted the assumptions, and added one more proof that links to disentanglement. We also clarified the connection between our theory and learning and refined our results to be more self-contained. 

* **Empirically**, we highlighted the analyses regarding the clustering performance for mode label estimation and the impact on the performance of SD-HC, which had been a part of our comprehensive experiments but might not be prominent to readers. We also provided practical guidance for choosing suitable clustering methods. In addition, we add toy experiments to validate SD-HC in a complex multi-attribute setting.

Eventually:
* **Reviewer UiJJ** acknowledged the understanding of the concepts, and raised the rating.
* **Reviewer HxXe** appreciated our clarifications, considered the other reviews/responses, and maintained a positive rating. 
* **Reviewer dyTi** acknowledged that our new definitions $z, z^l$ were clear, and our added proof seemed correct, but raised new concerns. We further offered clarifications that the required graph was in the appendix and promised to move it to our main paper, and that the change of the equation in Definition 2 was part of the thorough adjustment regarding the new definition of $z$. Although we didn't receive further feedback, we sincerely hope this addresses the concerns.
* **Reviewer wrRV** might have been occupied during the discussion phase and was unable to engage in discussions.

We thank the reviewers for their effort and time in reviewing our paper, and the Area Chairs' effort in encouraging discussions. We hope this summary will assist all reviewers and the Area Chairs in making the final decision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper aims to address disentangled representation learning under hidden correlations and multi-modal attribute distributions. 
During the review, reviewers raised theoretical concerns, particularly regarding assumptions, definitional clarity, and the rigor behind the necessity and sufficiency claims. Reviewer dyTi and Reviewer wrRV still harbored concerns about theoretical rigor, the modifications of key definitions during the rebuttal, and the underlying assumptions needed to fully justify the claims.  No reviewers were willing to champion this paper.  we recommend rejecting this paper.