Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning

Authors: Zijian Li, Shunxing Fan, Yujia Zheng, Ignavier Ng, Shaoan Xie, Guangyi Chen, Xinshuai Dong, Ruichu Cai, Kun Zhang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Disentangled Representation Learning, Identifiability

Abstract:
Disentangled representation learning aims to uncover the latent variables underlying observed data, yet identifying these variables under mild assumptions remains challenging. Some methods rely on sufficient changes in the distribution of latent variables indicated by auxiliary variables, such as domain indices, but acquiring enough domains is often impractical. Alternative approaches exploit the structural sparsity assumption on mixing processes, but this constraint may not hold in practice. Interestingly, we find that these two seemingly unrelated assumptions can actually complement each other. Specifically, when conditioned on auxiliary variables, the sparse mixing process induces independence between latent and observed variables, which simplifies the mapping from estimated to true latent variables and hence compensates for deficiencies of auxiliary variables. Building on this insight, we propose an identifiability theory with less restrictive constraints regarding the auxiliary variables and the sparse mixing process, enhancing applicability to real-world scenarios. Additionally, we develop a generative model framework incorporating a domain encoding network and a sparse mixing constraint and provide two implementations based on variational autoencoders and generative adversarial networks. Experiment results on synthetic and real-world datasets support our theoretical results.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730128481704

thread_type:
official_review

<thread_object>
creator: reviewer_qelo

<content>
summary:
In this paper, authors combine two assumptions for disentangling latent variables (latent identifiability). Thew whole work is based on non-linear ICA (identifiable VAE) with auxiliary variables $u$. Authors show that sufficient changes on the distribution of latent variables and exploiting structural sparsity assumptions on the mixing procedure can be complement each other to achieve better identifiability. Theorems are illustrated and proved in this paper. With both synthetic and real-world experiments, authors show that there proposed approach based on VAE and GAN can achieve good latent indentifiability.

strengths:
* The theories in this paper is presented and illustrated in a clear way. The schematic graphical model is good. Besides, there are solid prooves of those theorems.
* The contribution is good, I can visually understand the usefulness of the proposed method.
* There are both synthetic results validating the method, and real-world applications to show that the new method is useful.

weaknesses:
* The sentence of Theorem A3 is a bit confusing. It is a bit hard to read that long sentence.
* Line 158, why $x_k$ contribute $z_j$? And the derivative $\frac{\partial z_j}{\partial x_k}$. Could the author explain why the relationship is not $z_j$ contribute $x_k$.
* Line 349, "$q(z|x)$  is the an encoder"
* Eq. 9, why loss $L_s$ is chosen to be the absolute sum rather than the squared sum? Will a squared sum be easier to train and more stable? Any reason or trade-off analysis on this choice?
Or does absolute sum have sparsity?
* What is CG-VAE-S? It seems like it was used before it was introduced.
* For synthetic results, why not apply CG-GAN? Also, there are no other baseline methods for comparison.
* It is also very common to compare the generated image with the true by MSE, $R^2$, or cross-entropy (log-likelihood). They are intuitive metrics that are worth seeing.
* Same for the real-world, why not try CG-VAE. If there is some simple reason, authors should state in one sentence why VAE is not suitable for real-world datasets. Also, are there any other alternative methods for real-world, since all four methods are actually some method combinations? Please correct me if my understanding is wrong.
* To me, it would be better to have more experimental results in the main content. The current experiment section is a bit simple. I'd like to see for example, more comprehensive comparison.

questions:
/
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618767505

thread_type:
official_review

<thread_object>
creator: reviewer_4kgp

<content>
summary:
In this paper, the authors investigated disentangled representation learning and proposed a new framework by combining two existing methods: sufficient changes and sparse mixing procedures. This framework can guarantee identifiability of the models, because the sparse mixing procedure can induce conditional independence between latent variables , which can simplify the mapping from estimated latent to ground truth. On the other hand, sufficient changes can further improve component-wise identifiability of latent variables. They implemented this framework for two types of deep generative models: VAEs and GANs. They evaluated on multiple datasets and show that their approach can result in better disentangled representations.

strengths:
Quality and Clarity: In general this paper is in good quality. The paper is well structured and the motivation is significant, and the writing is clear. The authors provide very detailed math derivations for the framework, and to my best knowledge, the derivations are sound. 

Originality: While the paper leverages two existing methods for disentangled representation learning, it is interesting that they explore a framework that combines them in a compatible way, and this framework will also guarantee the identifiability of representation learning.

weaknesses:
While they validated their framework on multiple datasets, they only provided qualitative results in terms of the disentanglement of learned representations. I would hope that they can evaluated the results more quantitatively and make comparison based on metrics of disentanglement, e.g. the metrics used in [1] and [2]. I guess the reason I want to see this is it is difficult to see the disentanglement only based on t-SNE plots, because t-SNE itself will employ an algorithm that perform dimension reduction into 2D space, so it is likely that this algorithm introduces certain level of disentanglement.

[1] Eastwood, Cian, and Christopher KI Williams. "A framework for the quantitative evaluation of disentangled representations." 6th International Conference on Learning Representations. 2018.

[2] Kim, Hyunjik, and Andriy Mnih. "Disentangling by factorising." International conference on machine learning. PMLR, 2018.

questions:
What is the reason that the authors choose VAEs and GANs as the two specific types of models that they want to implement the framework with? I mean my main concern is that, since disentangled representation learning has been studied for almost ten years from now, there have been a lot of variants of VAEs that were developed for learning disentangled representations. So I wonder 1) whether their framework can be applied to those variants of VAEs, and 2) if it is possible to compare against those models, such as TC-VAE[3], FactorVAE[4], and HFVAE[5]


[3] Chen, Ricky TQ, et al. "Isolating sources of disentanglement in variational autoencoders." Advances in neural information processing systems 31 (2018).

[4] Kim, Hyunjik, and Andriy Mnih. "Disentangling by factorising." International conference on machine learning. PMLR, 2018.

[5] Esmaeili, Babak, et al. "Structured disentangled representations." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684658965

thread_type:
official_review

<thread_object>
creator: reviewer_fw6r

<content>
summary:
A disentangled representation learning framework with two types of identifiability guarantees has been proposed. By forcing a sparse mixing procedure, the subspace identifiability is ensured, while sufficient changes in the distribution of latent variables promote componentwise identifiability of latent variables.

strengths:
- Theoretical understanding of identifiability issue in the disentangled representation
- Proposing two generative implementations based on VAE and GAN using a domain encoding network and sparse mixing constraints
- Synthetic and real-world experiments

weaknesses:
The notations are heavy and hard to follow. Please provide a comprehensive notation table at the beginning of the paper. and explicitly define each symbol when it's first introduced. For example, 
- In equation (2), what is the meaning of two subscripts of $\mathbf{x}_{k, (1)}$. 
- Other notation also needs to be defined carefully, $\backslash k$ (all index but $k$) ?
- In equation (6), it seems there is a missing integral (the second integral has been omitted). 

There are English typos and errors, e.g., line 157 (contributions --> contribute)

In synthetic experiments, there is no comparison with any nonlinear ICA methods. Comparing the CG-VAE, and CG-GAn with one or two relevant nonlinear ICA methods can be beneficial for readers to understand the advantages of incorporating domain encoding network and sparse mixing constraints in the disentangled representation.

questions:
- In line 220, the index in $\mathbf{z}_{o1}$ should be $o_1$ ?
- Can you provide some intuition why Sufficient Changes for Component-wise Identification require the second derivative assumption, while this is not the case for the subspace Identification? This could help readers better understand the theoretical foundations of the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732551627970

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We hope AC can encourage Reviewers 4KGP for discussion

comment:
We thank the ACs for the time and efforts dedicated to taking care of our paper. Despite your very busy schedule, we hope you can help encourage Reviewers 4KGP for discussion.

We believe our modifications can address the concerns raised by Reviewers 4KGP well. As Reviewer qELo said, our modification can well solve his/her similar questions.

We kindly request you consider these points in your final decision.

Thank you once again.

With best regards,

Authors of submission 4906
</content>
</thread_object>

</thread>
<thread>
creation date:
1732969842156

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear AC,

Thank you very much for the time and effort you dedicated to handling the submissions.  All reviewers responded to our response, which we highly appreciate.  We would like to bring your attention to Reviewer Fw6r, who ever said he/she would like to increase the rating to 7, but later deleted the message.  Perhaps it is because there is no rating of 7 as an option.  We hope you can take this into account when making your decision.  

Many thanks,

Authors of submission 4906
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a disentangled representation learning framework that combines two existing methods—sufficient changes in the latent variable distribution and sparse mixing procedures—to ensure identifiability in deep generative models like VAEs and GANs. The approach is validated with both synthetic and real-world experiments, showing that it leads to better latent identifiability.

Strengths: The paper presents a solid theoretical framework for identifiability in disentangled representation learning and provides clear mathematical derivations, supported by experimental results. The combination of sparse mixing and sufficient changes in latent distributions represents an innovative and effective approach to improving identifiability.

Weaknesses: The paper could benefit from clearer notation and definitions, as well as a more comprehensive experimental evaluation, including comparisons with other relevant models like nonlinear ICA methods. Additionally, while the results are promising, further quantitative analysis of disentanglement metrics would strengthen the conclusions. Most were addressed during the rebuttal. 

Recommendation: I recommend this paper for acceptance, as it makes a meaningful contribution to the field of disentangled representation learning, though improvements in presentation and additional experimental comparisons would enhance its impact.