Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning
Authors: Zijian Li
Shunxing Fan
Yujia Zheng
Ignavier Ng
Shaoan Xie
Guangyi Chen
Xinshuai Dong
Ruichu Cai
Kun Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Disentangled Representation Learning
Identifiability
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Disentangled representation learning aims to uncover the latent variables underlying observed data, yet identifying these variables under mild assumptions remains challenging. Some methods rely on sufficient changes in the distribution of latent variables indicated by auxiliary variables, such as domain indices, but acquiring enough domains is often impractical. Alternative approaches exploit the structural sparsity assumption on mixing processes, but this constraint may not hold in practice. Interestingly, we find that these two seemingly unrelated assumptions can actually complement each other. Specifically, when conditioned on auxiliary variables, the sparse mixing process induces independence between latent and observed variables, which simplifies the mapping from estimated to true latent variables and hence compensates for deficiencies of auxiliary variables. Building on this insight, we propose an identifiability theory with less restrictive constraints regarding the auxiliary variables and the sparse mixing process, enhancing applicability to real-world scenarios. Additionally, we develop a generative model framework incorporating a domain encoding network and a sparse mixing constraint and provide two implementations based on variational autoencoders and generative adversarial networks. Experiment results on synthetic and real-world datasets support our theoretical results.

Published as a conference paper at ICLR 2025
SYNERGY BETWEEN SUFFICIENT CHANGES AND SPARSE MIXING PROCEDURE FOR DISENTANGLED REPRESENTATION LEARNING
Zijian Lit*" Shunxing Fan"* Yujia Zheng' Ignavier Ng Shaoan Xie Guangyi Chen
Xinshuai Dong' Ruichu Cai Kun Zhang
*Camegie Mellon University, Pittsburgh PA, USA
*Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
+Guangdong University of Technology, Guangzhou, China
ABSTRACT
Disentangled representation learning aims to uncover latent variables underlying the observed data, and generally speaking, rather strong assumptions are needed to ensure identifiability. Some approaches rely on sufficient changes on the distri- bution of latent variables indicated by auxiliary variables such as domain indices, but acquiring enough domains is often challenging. Alternative approaches exploit structural sparsity assumptions on the mixing procedure, but such constraints are usually (partially) violated in practice. Interestingly, we find that these two seem- ingly unrelated assumptions can actually complement each other to achieve identi- fiability. Specifically, when conditioned on auxiliary variables, the sparse mixing procedure assumption provides structural constraints on the mapping from esti- mated to true latent variables and hence compensates for potentially insufficient distribution changes. Building on this insight, we propose an identifiability theory with less restrictive constraints regarding distribution changes and the sparse mix- ing procedure, enhancing applicability to real-world scenarios. Additionally, we develop estimation framework incorporating domain encoding network and a sparse mixing constraint and provide two implementations, based on variational autoencoders and generative adversarial networks, respectively. Experiment re- sults on synthetic and real-world datasets support our theoretical results. The code is available hhtps///stttbb.coommoorrreesesyr htwp:///wwooooooooiiiiinnnnnnnne Disentanglement
INTRODUCTION
Disentangled representation learning (Schölkopf et al. 2021 is learning paradigm that seeks to learn meaningful and explanatory representation from observed data, Mathematically, suppose that observed variables x are generated from latent variables z through an unknown mixing function f. i.e., x = [(z). The primary goal of disentangled representation learning is to recover the latent variables with certain identifiability guarantees, i.c., to estimate them up particular indetermina- cies such as component-wiss transformations. A thcoretical foundation in this field is related to independent component analysis (ICA) (Hyvärinen al. 2001 2023). Earlier methods (Hyvärinen & Ojal 2000) Tichavsky et al.) (006) achieved identifiability with the linear mixing assumption. To address more complex real-world tasks, nonlinear ICA (Hyvarinen & Moriokal 2016) allows the mixing function f to be an unknown nonlinear function. However, the identifiability of nonlinear ICA remains significant challenge, primarily because without additional assumptions, there can be infinitely many solutions with independent variables which are mixtures of the true latent variables (Hyvärinen & Pajunen 1999).
To overcome these challenges, several assumptions have been proposed, such as distribution suffi- cient changes (see, e.g.. Hyvarinen& Morioka (2016)) and mixing procedure restrictions (see, e.g., Zheng & Zhang (2023). For example, the methods based on sufficient changes exploit the auxiliary valuables u and assume that the latent variables have changing distributions across different values
"These authors contributed equally to this work.

---

Published as a conference paper at ICLR 2025
of u but are conditionally independent given (Hyvarinen & Morioka 2016 Hyvarinen et al. 2019
Khemakhem et al. (020aa). Under this assumption, many works use domain indices or the observed
class labels as u Kong et al., 2022 Li et al., 2023). Additionally, other works harness the clustering-
based methods (Willetts & Paige 2021) or temporal dependency (Yao et al. 2021] 2022: Chen et al.,
2024 Li et al.
2024) for the identifiaiility of nonlinear ICA. Alternatively, y. one can impose suitable
restrictions on the mixing function (Gresele et al. 2021, Moran et al. 2021] Locatello et al. 2019b) to achieve identifiability. Specifically, [Hvärinenn & Pajunen| (1999); Buchholz et al. (2022) show that conformal maps are identifiable up to specific indeterminacies. Recently, Morioka & Hyvärinen
(2023) achieved identifiability by assuming that the observational mixing exhibits suitable group- ing of the observations. Zheng et al. (2022); Zheng & Zhang (2023) proposed the structural sparsity assumption, making it fully unsupervised manner for the identifiability of nonlinear ICA. Specifi- cally, this assumption restricts the connective structure from sources to observations, i.e., the support of the Jacobian matrix of the mixing function. One may refer to Appendix C for further discussion of related works, including the disentangled representation learning, identifiability of nonlinear ICA, and multi-domain image generation.
Despite advances in the identifiability of disentangled representation, these methods impose strin- gent conditions on the number of auxiliary variables or the mixing function, each of which might be violated in practice. On the one hand, the methods based on auxiliary variables Khemakhem let al. ) Kong et al. (2022) often require a large number of values of the auxiliary variables u. identifiabi- ity of the latent variables typically requires at least 2n + domains, where the dimension of latent variables is n. On the other hand, to achieve full identifiability, methods with a constrained mixing procedure have to enforce strong constraints on the mixing function. For example, the structural sparsity assumptions (Zheng et al., 2022] Zheng & Zhang, 2023)say that for each latent source 2;, there exists a set of observations such that the intersection of their parents is zj, which excludes scenarios where any latent variable densely connected to observed variables. Although each of the two types of constraints discussed above is generally strong. both of them may be partly true in practice, given a particular problem. Therefore, it is natural to ask this question: Is it possi- ble leverage both types of constraints in complementary principled way to learn disentangled representations with identifiability guarantees?
Specifically, in domain adaptation, achieving theoretical guarantees of component-win
The answer is yes unified framework is proposed in this paper to address it. Intuitively, the sparse mixing procedure assumption implies independence between certain latent variables and a particular subset of the observed variables. For example, zi JL xry uu, if z; is not adjacent to xyj (i.e., z does not contribute to .ry). This condition can then be used to constrain the mapping from the estimated latent variables å to the true latent variables z. In such cases, the sparsity in the mixing procedure compensates for a potential lack of sufficient domains. Additionally, even when the structural spar- sity assumption is partially violated, the sufficient changes introduced by the auxiliary variables u can still benefit the identifiability of latent variables. Therefore, the sparse mixing procedure and sufficient changes assumptions nicely complement each other-when one assumption is (partially) violated, the other can compensate, allowing for the identifiability of disentangled representations with milder assumptions.
Based on this intuition, we establish a principled identifiability framework leveraging both types of constraints. Moreover, we develop a general generative model framework with domain encoding networks and a sparse mixing constraint for disentangled representation learning benefiting from both constraints. Specifically, the domain encoding network is used to impose the assumption of sufficient changes by modeling the distribution of latent variables given auxiliary variables, and the sparse mixing constraint is used to enforce the sparsity of the estimated mixing procedure. Our method is validated through a simulation and several widely used multi-domain image generation datasets. The performance demonstrates the effectiveness of the proposed framework.
2 PRELIMINARIES
We start by giving a brief background on nonlinear independence component analysis (ICA) and identifiability. In typical nonlinear ICA setting, the data generation process is shown as follows:
p(z) = II p(z)), x= g(2),
(1)
i=1
2

---

Published as a conference paper at ICLR 2025
Table I: Notation Descriptions.
Symbol
Description
2
Scalar variable
Random vector
Xk
k-th component of the random vector x
(())
i-th instance realization of the random vector
XR,(0)
i-th instance of the k-th component of the random vector X
XiA
Random vector x excluding the k-th component
where x = (z1...... Fn.) and z (21...., 2n denote the observed variables and the independent latent variables with the dimension of n, respectively, function g : z + x denotes nonlinear mixing procedure. As mentioned in Hyvärinenn & Pajunen (1999), without any further assumptions, there is an infinite number of possible solutions and these have no trivial relation with each other. Therefore, previous works introduce the auxiliary variables u and assume that the latent variables are conditionally independent given u., i.c., z = =_ and p(z u) = I p(z] [), where e is the independent exogenous noise variables. The goal of nonlinear ICA is learn estimated unmixing function و x - 2, such that 2 = (f1,..., , 2n) consists of independent estimated sources.
To better understand our theoretical results, provide the description of notation as shown in Table as well as the definition of subspace-wise identifiability and component- wise identifiability.
Definition (Subspace-wise Identifiability of Latent Variables (Li et al. 2023)). The subspace- wise identifiability of E "A means that for ground-truth z, there exists 2 and an invertible function functiol h: 18 + R such that z h(ž).
Definition (Component-wiis Identifiability of Latent Variables (Hyvarinen & Morioka 2016 The component-wen wise identifiability of that for each 2,, E [n]. there exists a corresponding estimated component žj.j E [T)] and an invertible function h2 R - BR such that h((z))
3. IDENTIFIABILITY WITH COMPLEMENTARY GAINS FROM SUFFICIENT CHANGES AND SPARSE MIXING PROCEDURE
In this section, illustrate the identifiability results by leveraging the complementary gains from sufficient changes and sparse mixing procedures. Specifically, we first present the subspace identifi- cation result (Theorem DD. where certain latent variables are subspace-wise identification. Based on the aforementioned result, we further establish the component- wise identification result (Theorem 2). which uses conditional independence induced by sparse mixing procedure to constraint the solu- tion space of full-rank linear system. Additionally, we also show that the existing component wise identification results (Khemakhem et al., 2220aa) Kong et al. 2022) with 2n + auxiliary variables special cases of our approach (Corollary when the mixing procedure from latent sources to observations are fully-connected.
3.1 SUBSPACE IDENTIFIABILIYY WITH COMPLEMENTARY GAINS
Theorem 1. (Subspace Identification with Complementary Gains) Following the data generation process in Equation (7). we further make the following assumption.
Al (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, i.e., Pz/u > over Z and U.
A2 (Conditional Independence): Conditioned on u, each 2; is independent of any other zj for i,j E (I....,n),i. j, i.e., log (z/u(ziu) Ei log (z)
A3 (Generalized Sufficient Changes for Subspace Identification) Let Xe, be subset of x. X) be the left variables, and (1) (k,(0) be two different instance of xt- And vectors w(k, u) wikk 0) with I,..., [Pa(x;] are linearly independent, where vector w(k, u) is defined as:
w(k, u) Ologppxx... x/,uu)
2 log ((x. X1, (.,ulu)
dz
821 & log (Oxx( azn((a)
(2)
i log x:// azpu(za)
3

---

Published as a conference paper at ICLR 2025
Suppose that we learn 9 to achieve Equation 7 with the minimal number of edges of the mixing process. Then, for every pair of 2g and X2 in which X2 does not contribute to 2.g. i.e., 02, Ext = 0,
[2gingl
Zpa(xe) is not the function of Zj. i.e. BE,
O, where Epa(se) ) is the parents of Xx:
Intuition of the Theoretical Results. A proof of Theorem D can be found in Appendix A.1 For a better understanding of the proposed theory, provide intuitive example shown as Figure [ which includes the ground truth and estimated data generation processes with aux- iliary variables
The solid lines denote the true mixing edges. In the ground-truth gen- 0. Since the powerful neu- ral networks may choose to use the fully connected mixing procedure during training, lead- ing to the redundant estimated mixing edges, i.e., the red dashed line. We will show how
eration process, because Za is not adjacent to Xk: 8z.
0x
to remove these redundant estimated mixing edges via sparse mixing constraints in Section
As shown in Figure (a), we let X and 2y be subset of observed variables and its corresponding parents. Since zo do connect to XR. we have X Il Zo lu. Sequentially, suppose that we have two different values of Xg. i.e., XR,(0) and (1, then we have:
u
u
a
2.
2
2x
i log p(x,, xs, x Ø (log.((xa. N6. (1) (u)
Xx
R
Oz.
Oz.
(a) Ground-truth data generation process.
(b) Estimated data generation process.
(3)
Similarly, by constraining the sparsity of the esti-
mated mixing procedure, we can also achieve con- ditional independence ê. .lI & Therefore, we can construct a full-rank linear system as shown in Equation (4) by leveraging auxiliary variables.
Figure 1: Hxample for Theorem 1 with ground-truth and estimated data generation processes. The red dashed lines denote the dundant estimated mixing edges.
i Oogg((xa No, (1)())
Oz,
a pox, No, (x, /u) Oz
dz.
05, x4 (1)
Oz,
0E, x x
=0.
(4)
Therefore, with a sufficient number of values of the auxiliary variables u, the only solution of On. for 2, E ZØ and 2, E (E.: ), zero. This implies that {z.. Zb} is only the function of {2,,, 2n)) i.e. (za; 7b} is subspace-wise identifiable.
82,"
Discussion of Assumptions. We also provide detailed discussions of the assumptions and how they relate to real-world scenarios. The first two assumptions are standard in the componentwise identification existing nonlinear ICA (Li al. 2023 Kong et al. 2022 Khemakhem et al. 2020a). The smooth and positive density assumption means that the latent variables change continuously. For example, z₁ can be considered as the light directions of images, which are changing continuously. The conditional independence assumption implies that there no causal relationships among For instance, considering gender as an auxiliary variable, the light direction and the image resolution are conditionally independent. Finally, the sufficient changes assumption reflects that the conditional distributions are first-order differenntialee And the linear independence is the condition of a unique solution of the linear full-rank system.
Although existing works like domain adaptation (Li et al. 2023), and disentangled representation learning (Zheng Zhang 2023) also adopt the linear independence of first-order to achieve sub- space identification, the assumption in our work is more general and easier to meet in practice. First, the aforementioned methods achieve subspace identifiability by assuming that the auxiliary variables do not have influence on all the latent variables, e.g., partition the latent variables into domain-invariant and domain-specific variables. However, our result does not need this assump- tion. Moreover, the aforementioned methods only use adequate number of values of the auxiliary variables to meet the linear independence assumption, which might be hard to satisfy in practice. Mcanwhile, method further harnesses the independence between observed and latent variables, which constraints the solution space of the full-rank linear system and hence decreases the require- ment for the number of auxiliary variables.
4

---

Published as a conference paper at ICLR 2025
3.2 COMPONENT- WISE IDENTIFIABILITY WITH COMPLEMENTARY GAINS
Theorem 2. (Component- wise Identification with Complementary Gains) Let the observations be sampled from the data generating process in Equation 17. Suppose that Assumptions from Theorem hold and also that we learn 9s Pa/u to match the marginal distribution between x and & with the minimal number of edges of the mixing process. We further make the following assumption:
A4 (Sufficient Changes for Component-wise Identification)): Suppose N5, and Xa be subset of x and X2 Oxx= = We let the Za be the set of latent variables that connect to Xa but not connect to Xt, such that for all different values of Xa such as Xa (1) the vectors V(a k, us) - V(a, k. 0) with = 1....,,1.. 2/zua are linearty independent. And the vector V(a, k, u,) is defined as follows:
(8²
88 log ((za,(1). 2o, Zu Um.)
8 log p ((),,10)) 2o, Zu Um)
V(a, (0ckk,um)
(02:)²
(0z.)²
(5)
i log (1), Bb, BA uum)
ô log (() (0) Zu, 2k UL))
O2.
Oz.
2:Eaa
Suppose that we learn 9 to achieve Equation T
with the minimal number of edges of the mixing
process. Then is component-wise identifiable.
Intuition of the Theoretical Results. A proof can be found in Appendix A.2 For a better un- derstanding, we also provide an example as shown in Figure [ Besides assuming that x is not connected to za and Zb. we further assume that there exists another subset of observed variables x. that is connected to z. but not 2₁. Therefore, process in Theorem 1, suppose that we have different values of observed variables Xa. by further conducting second-orde -oddr partial derivatives w.r.t. 21. 1 [n] based Equation ( component-wise identifiability by reducing the solution space of the full-rank linear system.
have Zg IL /u/u. Similar to the derivation
achieve
Intuitively, Theory 2 tells about the degree which each latent variable can be identified in the case of partial identifiability. For example, if we only have M number of auxiliary variables, then when try to determine subset of latent variables 2, corresponding to selected subsets of observed variables X and X2. if M 2 2/z. + 1 (we let |z, be the dimension of z,). then each identifiable; otherwise, they are subspace identifiable. Moreover, if for any X2: and X., the corresponding 2, with the largest dimension satisfies M 2 2/z., + 1, then the number of auxiliary variables is sufficient to achieve component- latent variables. Since |z,,| S ni, we can use fewer values of the auxiliary variables to achieve component-wiss identifiability. Please refer to Appendix for further discussion of the theories.
24 E Za component
identifiability for all the
Compared with existing works for component-wis wise identification Kong et al. (2022), our result is also more general. First, our method does not require any assumptions about the distribution of latent variables, while Khemakhem et al. ; Hy Hyvarinen et al. (2019); Hyvarinen & Morioka (2016) assume that the latent variables follow the exponential family distributions. Second, we exploit the conditional independence brought by the sparse mixing procedure to constrain the solution space of the linear full-rank system, so fewer values of auxiliary variables are required.
3.3 SPECIAL CASE WITH FULLY-CONNECTED MIXING PROCEDURE
When the mixing procedure is fully connected, we show that the existing identification results with auxiliary variables are special cases of our theory, as shown in Corollary |
Corollary 1. (General Case for Component-wise Identification with Full-connected Mixing pro- cedure) We follow the data generation process in Equation 7 and make assumptions AI, A2, and A3. In addition, we make the following assumptions:
A5 (Fully Connected Mixing procedure): Each latent variable 2; is connected to each observed variable xj. where i, [|n].
Suppose that we learn 9 to achieve Equation 2; is component-wise identifiable with 2n + different values of auxiliary variables u.
Intuition of the Theoretical Results. Please refer to Appendix A.3 for proof. We provide the intuition behind the theoretical results follows. Because the mixing procedure is fully connected,
5

---

Published as a conference paper at ICLR 2025
LxL L1
Fu
10.1))
mm
2
Mask
Fu
Y
1.
Encoder
Decoder
Generator
X
Discriminator
X
(a) Implementation of CG-VAE
(b) Implementation of CG-GAN
Figure 2: Illustrations CGE and CG-GAN, respectively. é and the estimated and ground- truth noise variables, respectively. Fu denotes the domain- neural networks. Note that in the CG-GAN, we partition the latent variables into the domain-specific style 2,, The I. denotes the mask restriction for automatically optimal dimension determination. And Y and Y denote the ground-trute and predicted labels of real and fake samples.
content variables z, and
there are no pairs of latent variables and observed variables that are independent conditioned on u. so we cannot constrain the solution space of the linear equations obtained by the second orderr partial derivatives. As result, for n-dimension latent variables, 2n auxiliary variables are needed to obtain compoenent-wise identifiability.
4 GENERAL GENERATIVE MODEL FRAMEWORK FOR DISENTANGLED REPRESENTATION LEARNING
To levarage the developed theory for disentanglement, propose generative model framework for disentangled representation learning, which includes the domain-encoding neural networks and the sparse mixing constraint. Additionally, we provide two practical implementations based on vari- ational autoencoder ( (VAE) (Kingma 2013) and generative adversarial network (GAN) (Goodfellow) al. 2020 respectively. For clarity, we name these implementations based on complementary gains CG-VAE and CG-GAN, respectively. Moreover, We employ CG-VAF and CG-GAN for synthetic experiments and multi-domain image generation, respectively. Moreover, for a fair com- parison, we follow the backbone architecture of (Xie al. (023), which also leverage nonlinear ICA and minimal change assumption to achieve identifiability disentangled representation. The model architecture of the CG-VAE and CG-GAN are shown in Figure 2 () and (b), respectively.
4.1 IMPLEMENTATION OF CG-VAE
We begin with the derivation of the evidence lower bound (ELBO) in Equation (6)
log ((x/(x)u) = log p(x])))((zzu) = / /log p(x (2)))))((())(())))))))) dz
xz
(6)
Eggit))) p(g(() 2) + Egtx x) logp(ulzz) - (KL(((/
where KL denotes the Kullback-Leibler divergence. Since the reconstruction of is not the opti- mization goal, we remove the reconstruction of u. g(z]x) implemented as an encoder neural archi- tecture, which outputs the mean and variance of the posterior distribution and p(x(z) is parameter- ized as the decoder that takes latent variables z for reconstruct observed variables.
Domain-encoding Neural network. Similar to Kong et al. (2022 Zhang al. (2024), to model how auxiliary variables u influence the latent variables z, we employ the normalizing flow-based architecture (Dinh al., 2016 Huang al., 2018 Durkan et al., 2019) that takes the estimated latent variables and auxiliary variables to turn it into é. Specifically, we have:
e, det (2).
where F. is the normalizing flow and log det is the log determinant of the conditional flow transfor- mation on 2. Therefore, by assuming that the noise term follows a standard isotropic Gaussian, we use the change of variable formula compute the prior distribution of the latent variables:
log (1) logp(() + log det.
(8)

---

Published as a conference paper at ICLR 2025
Sparse Mixing Constraint. To enforce the conditional independence between the estimated latent variables and the observed variables, we further devise the sparse mixing constraint on the partial derivative of & with respect to 2, which shown as follows:
L
Σ
Finally, the total loss of the VAE-based implementation is shown as follows:
Livee = Ex(e)s) logp(x/z) + al, - 3KL(g(2/))l<vzlx) = L, + aL. 3LKL,
(10)
where and are the hyper-parameters.
4.2 IMPLEMENTATION OF CG-GAN
We further provide the implementation of CG-GAN as shown in Figure 2 (b) since it is suitable for the multi-domain image translation task. In this case, we consider more practical data generation process in Xie et al. (023)), where the latent variables are partitioned into the domain-invariant variable Ze and the domain-specific variables Z₂ for content and style, respectively.
Domain-encoding neural networks Similar to CG-VAE, we also the domain-specine flow function Fu to embed the domain information. Following Xie et al. (2023), we use learnable mask M shown in the gray block in Figure 2 (b) adaptively determine the dimension and location of Z, within the latent variable z.
= M F.)cc
(11)
where mask M is vector with the same shape as e. An entry of 0 in M indicates that the corre- sponding component belongs to ze and will not be affected by domain changes. If the entry is not O, it corresponds to 2,, which will vary according to domain changes.
Sparse Mixing Constraint For enforcing the sparsity of the mixing procedure, penalize the noorr of the Jacobian vector of x with respect to each 2g. which is similar to our VAE-based architecture. Specifically, we use forward finite differences (Kim Hong 2021) approximate the Jacobian vector in real-world experiments because the large dimensions of the image tensor make calculating the Jacobian computationally expensive.
G(2 + G(2+6)) G(z)
L
||aa|
Here, is the perturbation vector, with only the i-th entry being non-zero. Finally, the total GAN- based model loss are shown as follows:
Lyen La 7bm als
(12)
where y and a are hyperparameters. La is the traditional discriminator loss, Lm is 4 -orm of the mask matrix, which is introduced by Xie et al. (2023), and the L., is the sparse mixing constraint.
EXPERIMENTS
5.1
SYNTHETIC EXPERIMENTS
5.1.1
EXPERIMENT SETUP
Data Generation. We generate the synthetic data with multiple distributions. Specifically, we de- vise three datasets (Dataset A, B, and C) with 8, 9, and 11 numbers of domains, respectively. Please refer Figure for for causal graphs of these data generation processes. The noise variables sam- pled from a factorized Gaussian distribution for all datasets. We let the data generation process from latent variables to observed variables be MLPs with the LeaklyReLU activation function. Moreover, the dataset is randomly split into 90% for training and 10% for testing.
Evaluation Metrics. For evaluation, We use the mean correlation coefficient (MCC) (Hyvaiinen & Morioka 2016) between the true latent variables z and the estimated ones A higher MCC denotes the better identification performance the model achieve. We also consider other metrics for

---

Published as a conference paper at ICLR 2025
Dataset
Dataset
1.0
1.0
1.0
0.9
0.8
0.8
0.8
MCC
0.7
0.7
0.7
0.6
0.6
0.6
0.5
4
11
Number of Auxiliary Variables
Figure 3: Experiments results on synthetic datasets. The horizontal axis represents the number of auxiliary variables, and the vertical axis represents the values of MCC.
Table 2: Mean correlation coefficient results on dataset A of different methods
Number of Domain
CG-VAB
CG-GAN
iMSDA
iVAE
FactorVAE
Slow-VAB
B-VAE
0.953 (0.007)
0.935 (0.016)
0.869 (0.002)
0.756 (0.105)
0.728 (0.038)
0.856 (0.053)
0.810 (0.056)
0.952 (0.017)
0.927 (0:005)
0.881 (0.043)
0.807 (0.060)
0.804 (0.031)
0.798 (0.054)
0.706 (0.083)
0,956 (0.016)
0.955 (0:007)
0.936 (0.025)
0.815 (0.029)
0.755 (0.006)
0.913 (0.031)
0.795 (0.041)
0.971 (0.009)
0.944 (0.010)
0.959 (0.017)
0.920
0,805 (0.004)
0,921 (0.026)
0.881 (0.045)
disentanglement (Eastwood & Williams, 2018) like Completeness, Disentangle Score, Informative- ness, R², and MSE. Appendix provides the introduction of these metrics. Note that the higher the values of completeness, disentangle score, and R2, the better the performance of disentanglemennt, and the lower the values of informativeness, the better the performance of disentanglement. We repeat each experiment over random seeds for each experiment and report the mean and standard deviation. Please refer to Appendix B for the implementation of the synthetic experiments.
provides
Baselines: To evaluate the effectiveness of our method, we consider the following baselines. We first consider the standard 3-VAE (Higgins et al. (0017), FactorVAE (Kim & Mnih 2018). and HloWVEE (Klindt al., 2020). We further consider other nonlinear ICA-based methods like iVAE (Khemakhem et al. 2020a) and iMSDA (Kong et al. 2022). which use auxiliary variables.
5.1.2 RESULTS AND DISCUSSION
The experimental results of the synthetic datasets are shown in Figure 3 To evaluate the effec- tiveness of the proposed sparsity constraint, we remove L,, from CG- VAE and name CG-VAE-S, According the experiment results, we can obtain the following conclusions: 1) when the num- ber of auxiliary variables is sufficient, both CG-VAE and CG-VAE-S achieve relatively high MCC (around 0.95), demonstrating that sufficient changes brought by auxiliary variables benefit disentan- gled representation learning. 2) as the number of auxiliary variables decreases, CG-VAE-S shows a noticeable decline in performance across the three datasets, while CG- VAE remains stable. In par- ticular, for Dataset A, CG- VAE performs well even with just two domains, emphasizing the impor- tance of sparse mixing constraints for achieving identifiable disentangled representation. 3) When the number of domains is reduced to just one, both CG-VAE and CG-VAE-S perform poorly, indi- cating that sufficient variation essential for disentangled representation. At the same time, we find that even with only a single domain, CG-VAE still performs better than CG-VAE-S, this is because the mapping from the ground truth to estimated latent variables is constrained by sparse mixing pro- cedure, making certain latent variable subspaces identifiable. We also compare the proposed method with different baselines to evaluate the performance disentanglement. Experiment results with the MCC, completeness, and disentanglement (Fastwood & Williams) 2018) are shown in Tables 2 and 4 respectively. Compared with the baselines, we can find that the proposed method achieves the best disentanglement performance in different metrics even the the domain number is limited, which verifies our theoretical results. Please refer to Appendix for more experiment results.
Table 5: Results of two domain image generation on CelebA and MNIST datasets.
Dataset
Metries
TGAN
StyleGAN2-ADA
i-StyleGAN
CG-VAE
CG-GAN-M
CG-GAN
FID DIPD 1
4.89
3.57
2.65
3.02
2.60
2.57
CelebA
1.11
1.00
0.95
0.93
0.93
0.93
MNIST
FID Joint-FIID J.
67.45
117.64
16.6
18.18
31.74
8.74
155.21
386.19
107.39
111.76
81.53
67.04
8

---

Published as a conference paper at ICLR 2025
Table 3: Completeness results on dataset A of different methods.
Number of Domain
CG-VAB
CG-GAN
iMSDA
iVAE
FactorVAR
Slow-VAE
B-VAE
0.608 (0.115)
0,533 (0.074)
0.405 (0.063)
0.348 (0.035)
0.349 (0.057)
0.343 (0.072)
0,319 (0.008)
0.632 (0.054)
0.659 (0.132)
0.477 (0.028)
0.358 (0.148)
0.453 (0.069)
0.356 (0 083)
0.427 (0.095)
0.626 138)
0.681 (0.062)
0.663 (0.066)
0.368 (0.029)
(0.064)
0.596 (0. 107)
0.525 (0.008)
0.729 (0.064)
0.809 (0.017)
0.758 (0.042)
0.435 (0.008)
0.506 (0.057)
0.687 (0:039) 0.644 (0.113)
Table 4: Disentanglement score results on dataset A of different methods.
Number of Domain
CG-VAE
CG-GAN
iMSDA
IVAE
FactorVAE
Slow-VAE
(-VAE
0.586 (0. 188)
0.529 (0.066)
0.306 (0.032)
0.347 (0.089)
0.320 (0.091)
0.347 (0.068)
0.320 (0.009)
0.589 (0.076)
0.666 (0.127)
0.498 (0.033)
0.315 (0.002)
0.380 (0.061)
0.338 (0.080)
0.403 (0.094)
0.611 (0.125)
0.672 (0.048)
0.514 (0.045)
0.548 (0.042)
0.537 (0.078)
0.620 (0.089)
0.522 (0.036)
0.697 (0.019)
0.719 (0.013)
0.710 (0:026)
0.564 (0.076)
0.626 (0.011)
0.655 (0.073)
0.692 (0.084)
5.2
REAL-WORLD EXPERIMENTS
5.2. EXPERIMENT SETUP
Datasets. We use the CelebA ((Liu et al.) 2015)) and the MNIST dataset ((LeCun et al. 1998) for multi-domain image generation. To further evaluate the effecttivnness of our theoretical results in real-world applications, we choose two domains for all the datasets. Specifically, in CelebA, we create two domains based on the presence or absence of eyeglasses, subsampling the no-eyeglasses domain to balance the sample sizes between the two. For MNIST, we use the training portion of the dataset and generate two domains consisting of red and green digits.
We evaluate our method using the Frechet Inception Distance (FID), a widely used metric for measuring the distribution divergence between generated and real images, where lower FID scores indicate better performance. Since the CelebA datasets lack paired data, we use InvainnInvariant Perceptual Distance (DIPD) to assess semantic correspondence (Liu al. 2019). DIPD calculates the distance between nnstance-normalized Conv5 features of the VGG network. As for the MNIST dataset with ground truth tuples, we first compute the inception features of images in each tuple, and concatenate the features. Sequentially, we compute the Frechet distance between the features of the ground truth and generated tuples. referring to this as Joint-FID, as measures the divergence between joint distributions. More addition, we remove the restriction of mask M and name this model variant as CG-GAN-M.
Evaluation Metrics.
Implementation and Baselines. cial Py implementation of i-StyleGAN (Xie et al. 2023), which also follows the backbone networks of StyleGAN2-ADA (Karras et al., 2020). Moreover, to verify the effectiveness of our of i-StyleGAN and only change the
For fair comparison, we build our method based on the offi-
introduced modules, we employ the default values of As for the normalizing flow-based domain-encoding neural networks, we employ the deep sigmoid flow (DSF), which is designed to be component-wiss strictly increasing. Since our method is built on StyleGAN2-ADA (Karras al., 2020) and i-StyleGAN (Xie et al.. (003), we consider TGAN (Saabaaii et al. (2022) as the compared method to evaluate the effectiveness of the sparse mixing constraint. Moreover, we further consider the CG-VAE, which uses the same de- coder architecture as the generator of the i-StyleGAN. In practice, we train the Stylegan2-ADD and i-Stylegan for 25000k images, which is the default setting of Stylegan. For our method, to save the training time, we load the checkpoint of i-Stylegan at 20000k images, and train for 5000k images.
5.2.2 RESULTS AND ANALYSIS
Comparison with Baselines. Experimental results on the CelebaA and MNIST datasets with only two domains are shown in Table We also provide some generated samples in Figure 4 According to the experiment results, we can find that the proposed method outperforms all other baselines on all the image generation tasks even though there are only two domains. Specifically, our method out- performs StyleGAN2-ADA with a clear margin, which reflects the importance of sufficient changes and the spare mixing restriction for disentangled representation learning. Moreover, compared with i-StyleGAN, we can find that our method also achieves a better result. We also find that the perfor- mance of CG-VAE is the worst, this is because the VAE-based models usually employ the Gaussian prior assumption (Bredell et al. 2023). The qualitative results shown in Figure 5 are clear. For ex- ample, in the adding glasses image generation task, we find that the images from --StyleGAN change

---

Published as a conference paper at ICLR 2025
TGAN
Stylegan2-ADA
i-Stylegan
CG-VAE
CG-GAN
3
6 8
6
6
1
6
6
0
/ /
3 O
8
8
7
3
7
9
1
0
32
Figure Samples of multi-domain image generation on the CelebA and MNIST datasets. i- StyleGAN, CG-VAE, and CG-GAN share the same noise input We find that when the number domain is insufficient, iStyleGAN will produce unnecessary changes.
Generated Domain1 Digit
Generated Domain2 Digit
Real Domain1 Digit
Real Domain2 Digit
(a) CG-GAN
(b) i-StyleGAN
(c) StyleGAN2
Figure 5: The t-SNE visualization of different methods. Blue and red points represent features from generated images, while green and yellow points correspond to features from real images. A greater overlap between generated and real points within the same domain reflects improved performance.
the glasses and gender simultaneously. Mcanwhile, the image generated from method is stable, showing that our method can achieve better disentanglement under weak assumptions. Finally, we can find that the performance of CG-GAN-M is lower than that of standard CG-GAN, showing the necessity of the learnable mask for image generation. Similar to the several results of the VAE-based methods (Higgins al. 2017 Pandey al. (02I). the images generated by CG-VAE are less clear than those generated by CG-GAN due to their Gaussianity assumption.
Visualization. We also provide the t-SNE visualization, shown in Figure 5 According to the experimental results, within the same domain, the data generated by our method shows the highest overlap with the corresponding ground truth.
CONCLUSION
This paper introduces a disentangled representation learning framework with identifiability guaran- tees by harnessing the complementary nature of sufficient changes and sparse mixing procedures, boosting its applicability to wallworld scenarios. Specifically,, the conditional independence in- duced by the sparse mixing procedure simplifies the mapping from estimated to ground truth latent variables, ensuring subspace identifiability. Meanwhile, sufficient changes promote component- wise identifiability of latent variables. Theoretical findings are validated through both synthetic and real-world experiments. Future work aims to extend these results to related tasks, such as transfer learning and causal representation learning. The empirical study in the paper preliminarily focuses on visual wtwpaa/ggleeeeeappppppppiiiieeie more complex real-world scenarios are be given.
ACKNOWLEDGMENT
We would like to acknowledge the support from NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R01HL159805, and grants from Quris AI, Florin Court Capital, and MBZUAI-WIS Joint Program. Moreover, this research was supported in part by the National Science and Technology Major Project (20212D0111) 1,50)), Natural Science Foundation of China (U24A20233).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1588):179--1828, 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Gustav Bredell, Kyriakos Flouris, Krishna Chaitanya, Ertunc Erdil, and Ender Konukoglu. Ex- plicitly minimizing the blur error of variational autoencoders. arXiv preprint arXiv:2304.05939 2023.
Andrew Brock. Large scale gan training for high fidelity natural image synthesis. arXiv preprint urXiv:1809 1096, 2018.
Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable non- linear independent component analysis. Advances in Neural Information Processing Systems, 35: 11946-16961, 2022.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in bela-vae. arXiv preprint arXiv:1804.0359% 2018
Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentangled semantic representation for domain adaptation. In IICAI: proceedings of the conference, volume 2019, pp. 2060. NIH Public Access, 2019.
Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangehen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. Caring: Learning temporal causal representation under non-invertible generation process. arXiv preprint arXiv:2401.14533 2024.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud, Isolating sources of disen- tanglement in variational autoencoders. Advances neural information processing systems, 31, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 10Xiv:6605,088000 2016.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows.
Advances in neural information processing systems, 32, 2019.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In 6th International Conference on Learning Representations, 2018.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan- Willem van de Meent. Structured disentangled representa- tions. arXiv preprint aiv: 804.02086 2018.
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. Advances in Neural Information Processing Systems, 36:27682-27695 2023.
Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxilary classifiers gan. Advances in neural information processing systems, 32, 2019.

---

Published as a conference paper at ICLR 2025
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 114411139--4, 2020.
Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, new concept? Advances in neural information processing systems, 34:28233-28248, 2021.
Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pp. 939-948. PMLR, 2020.
Hermanni Hälvä. Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. Advances in Neural Information Processing Systems, 34:1624-1633 1633, 2021.
I Higgins, D Amos, Pfau, S Racaniere, I. Matthey, D Rezende, and Lerchner. Towards a definition of disentangled representations. arxiv 2018. arXiv preprint arXiv:1812.0223
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Chin-Wei Huang. David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International conference on machine learning, pp. 2078-2087. PMLR, 2018.
Aapo Hyvärinen. Independent component analysis: recent advances. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(4944)200105544 2013.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvarimen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460-469. PMLR, 2017.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications.
Neural networks, 13(4-5):411445) 2000.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999.
Aapo Hyvärinen, Juha Karhunen, and Erkki Oja. interscience publication, 2001.
Independent component analysis, a wiley-
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turne. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Aapo Hyvärinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear independent component anal- ysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10), 2023.
Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equaation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics, 76(1):1-33, 2024.
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation.
Advances in Neural Information Processing Systems, 33:21357-21369, 2020.
Minguk Kang, Woohyeon Shim, Minsu Cho, and Jaesik Park. Rebooting acgan: Auxiliary classifier gans with stable training. Advances in neural information processing systems, 34:23505-2355 2021.
12

---

Published as a conference paper at ICLR 2025
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33:12104-1211 2020.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International conference on artificial intelligence and statistics, pp. 2207-2217. PMLR. 2020a.
llys Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33: 17.6--887777 2020b.
Dahye Kim and Byung- Woo Hong. Unsupervised segmentation incorporating shape prior via gen- erative adversarial networks. In Proceedings the IEEE/CVE International Conference on Computer Vision, pp. 7324-7334, 2021.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conference machine learning, pp. 2649-2658. PMLR, 2018.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint 20Xv:20077.1930 2020.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Vic- tor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In Kama- lika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10p.1144 11455-11472. PMLR, 17-23 Jul 2022. URL htttss://proceeinng ml. wtw...peee//vv.///eeeee
htmll
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv:1711 1111177111008.... 2017.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pp. 428-484 PMLR, 2022.
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexan- dre Lacoste, and Simon Lacoste-Julien. Nonparametric partial disentanglement via mecha- nism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint N0t.:2401.04890, 2024.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2324, 1998.
Adam Li, Yushu Pan, and Elias Bareinboim. Disentangled representation learning in non-markovian causal systems.
Zijian Li, Ruichu Cai, Guangyi Chen, Boyang Sun, Zhifeng Hao, and Kun Zhang. Subspace identifi- cation for multi-source domain adaptation. Advances in Neural Information Processing Systems, 36, 2023.
Zijian Li, Yifan Shen, Kaitao Zheng, Ruichu Cai, Xiangchen Song. Mingming Gong, Zhifeng Hao, Zhengmao Zhu, Guangyi Chen, and Kun Zhang. On the identification of temporally causal rep- resentation with instantancous dependence. arXiv preprint arXiv:2405, 15325, 2024.
M. Liu, X. Huang, A. Mallya, T. Karras, T. Aila, J. Lehtinen, and J. Kautz. Few-shot unsupervised image-to tomitteeeeeea-- translation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10550-10559 Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10. 1109/CC (00/ICCV20019000000 URL htter:::sebb..eee.ccaarrrrrrroor
TCCV. 2019 61965
13

---

Published as a conference paper at ICLR 2025
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. Advances in neural information processing systems, 29, 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference machine learning, pp. 411 4124. PMIR, 2019a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258. 2019b.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A commentary the unsupervised learning of disentangled representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 13681-13684 2020a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look the unsupervised learning of disentangled repre- sentations and their evaluation. Journal of Machine Learning Research, 21(209): 122,, 2020b.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly Weakly-supervised disentanglement without compromises. In International conference on machine learning, pp. 6348 -6599. PMLR, 2020c.
Takeru Miyato and Masanori Koyama. egans with projection discriminator: ar Xiv preprint
arXiv:1802.05637, 2018.
Gemma E Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. Identifiable deep generative models via sparse decoding. arXiv preprint arXiv:2110.10804, 2021.
Hiroshi Morioka and Aapo Hyvärinen. Causal representation learning made identifiable by grouping of observational variables. arXiv preprint a:Xiv:2310.15709, 2023.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with aux- iliary classifier gans. In International conference on machine learning, pp. 2017.
PMLR,
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Vaes meet diffusion models: Efficient and high-fidelity generation. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Yunchen Pu, Shuyang Dai, Zhe Gan, Weiyao Wang, Guoyin Wang, Yizhe Zhang, Ricardo Henao, and Lawrence Carin Duke. Jointgan: Multi-domain joint distribution learning with generative adversarial nets. In International Conference on Machine Learning, pp. 4151-4160 PMLR, 2018.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109:5)x612-634 2021.
Mohamad Shahbazi, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool. Collapse by ditioning: Training class-conditional gans with limited data. arXiv preprint arXiv:2201.06578 2022.
Raphael Suter, Djordje Miladinovic, Stefan Bauer, and Bernhard Schölkopf. Interventional robust- ness of deep latent variable models, arXiv, pp. 181 1000007, 2018.
14

---

Published as a conference paper at ICLR 2025
Petr Tichavsky, Zbynek Koldovsky, and Erkki Oja. Performance analysis of the fastica algorithm and crame/spl acute/r-rao bounds for linear independent component analysis. IEEE transactions on Signal Processing, 54(4):1189- 2006.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. International conference on machine learning, pp. 10401-10412. PMLR, 2021.
Xin Wang. Hong Chen, Si'ao Tang. Zihao Wu, and Wenwu Zhu, Disentangled representation learn- ing. arXiv preprint arXiv:2211. 1695, 2022.
Matthew Willetts and Brooks Paige. I don't need Identifiable non-linear ica without side infor- mation. arviv preprint arXiv:2106.05238, 2021.
Shaoan Xie, Lingjig Kong, Mingming Gong, and Kum Zhang. Multi-domain image generation and translation with identifiability guarantees. In The Eleventh International Conference on Learning Representations, 2023.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data, arXiv preprint arXiv:2110.05428, 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning. Advances in Neural Information Processing Systems, 21226422-26505 2022.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. adversarial networks. In International conference on machine learning, pp. 7354-7363, PMLR,
Self-attention generative
2019.
Kun Zhang, Shaoan Xie, Ignavier Ng. and Yujia Zheng. Causal representation learning from multi- plc distributions: A general setting. arXiv preprint arXiv:2402.05052. 2024.
Yujia Zheng and Kun Zhang. Generalizing nonlinear ica beyond structural sparsity. Advances in Neural Information Processing Systems, 36:13326-13355 2023.
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. Advances neural information processing systems, 35:1641 1-66411114222 2022.
15

---

Published as a conference paper at ICLR 2025
Supplement to
"Synergy Between Sufficient Changes and Sparse Mixing Proce- dure for Disentangled Representation Learning"
Appendix organization:
Proof
17
A.1
Subspace Identification via Generalized Sufficient Changes
17
A.2
Component-wise Identification Sufficient Changes of Multiple Distributions
18
A.3
General Case for Component-wise Identification with Full-connected Mixing Process
20
Synthetic Experiments
B. 1 Simulation Datasets
21
B.2 Mean Correlation Coefficient
B.3
Implementation Details
21
Related Works
C.1
Disentangled Representation Learning
21
Nonlinear Independent component analysis
22
C,3 Multi-Domain Image Generation
22
Discussion of the Theoretical Results
22
D.I
Implications
22
D.2 Intuition of Subspace-wise wise and Component wise Identification
................................
23
D.3 Discussion of Assumptions of Theorem
23
More Experiment Results
23
Scalability of Different VAE Variants
23
Other Constraint on Mixing Procedure
23
Experiment Results Other Metrics
24
Metrics Introduction
24
16

---

Published as a conference paper at ICLR 2025
A PROOF
A.1 SUBSPACE IDENTIFICATION VIA GENERALIZED SUFFICIENT CHANGES
Theorem (Subspace Identification with Complementary Gains), Following the data genera- tion process in Equation ( we further make the following assumption.
Al (Smooth and Positive Density): The probability density function of latent variables is smooth and positive, ie., Pz/u 0 over Z and U.
A2 (Conditional Independence): Conditioned on each is independent of any other 23 for i,j [1.... ,n,, + j. i.e., log (z|u(z]u)= -i= log Ps, (n (z)u).
A3 (Generalized Sufficient Changes for Subspace Identification) Let Xx be subset of and Xk,(1): 1k,10) be two different instance of And vectors w(k, u) wolk, 0) with u 1,... [a((x)) are linearly independent, where vector w(k, u) is defined as:
(ks, u) = ( (
(llogp(xx ((),)(())))))........()))))))) Xx,llu) Ölog Mlogp(x>,(00 ((,0))))((1) x(alu)
Ozi
0z1 Öog (0) x,,(u)
(13)
log ((xx X.E
@zpa(xe)
0zp((x))
Suppose that we learn 9 to achieve Equation 7 with the minimal number of edges of the mixing process. Then, for every pair of 2j and that are not adjacent in the mixing process, have Zpa(xe) is not the function of 2j. i.e., 02,
0zpiagl O, where Zpa(xs) is the parents of Xx
Proof. We start from the matched marginal distribution condition to develop the relation between and & as follows: Yu E U
Pxlu
Px|u
Pā(a)Ju Pa(=) Po-loo(i) Jg-1 P2|u
(14)
Ph(2jju P2|u - logPs P3 log J p(/p((
where 9 2 + Z denotes the estimated invertible generating function, and h := g o 9 is the transformation between the true latent variable and the estimated one. | stands for the absolute value of Jacobian matrix determinant of g Note that as both ğ and g are invertible, [ + # and h is invertible.
By matching the marginal distribution between & and x, we further take the first order derivative with 2, and have:
i llog((x/u) 0&
" a logp(xx(u) dz:
(15)
Oz, 02j
Suppose there exists group of observations X that are independent of žÚ given so there exists different values of Xu. i.e., Xk,(1) and (x,(0), making (log does not vary with different d2, values of Xk. Sequentially, we further let the other observed variables be X Then we subtract Equation corresponding to Xx.(1) with that corresponding to Xk,(0) and have:
0 Ölogp(xx.(1) x;/uu)
0z:
Olog p( (xx.())) xxulu)
Ozi
02,
xi -1-(1
dz
alogp( X(xuu)
0z;
aog (x) X% (u)
Oz;
Oz
02,
x(=X)((1)
0z
02g
xx=xx,
X..
opPa(xx)
i log pepp(( (1) xi,lu)
0z;
1lgg((Xx,(0) (0///1)
0zi
Ozi
0ā,
x==Xx,1)
0zi
Oīg xx=Xi
(EPa(xk)
alog X; x
Ozi
i log pl (0) Xx
0z;
Σ
0z
02,
x1 x1=X;,(1)
0z4
02,
JEPa(x))
(16)
17

---

Published as a conference paper at ICLR 2025
where Pa(x;) denotes the indices set of the parents of X5. Similarly, we further have:
alog (2)))
Ozi
i log pl ((xx, x. k/
Ozi
0
Oz;
d2g x (2,
0z,
02, xr. xx,(0)
(EPa(x))
@logp(Xx.te (.,12)xxxuu1)) xv.lu)
0zi
log pl (1x, Xvx
dzi
0
Σ (EPa(xa)
0z:
02- x. (2,
0zg
02,
X4. (k,
(17)
Suppose that we let u No. [] then by combining Equation (16) and 17 leveraging the sufficient changes assumption (A3), the linear system is Pa (kk
and further
3 P((())
full-rank system. Therefore, the only solution is 8z.
0, X2,0)
85.
0, and
0E;
xe.(0)
0zs
= 0, E Pa(x;), implying that 82,
Os.
= 0, E Pa(x)).
(3)
As h(-) is smooth 2, its Jacobian can written as
+ 0
82.
Jn
(18)
O2e 82,
8%.
82. Therefore, 82.
+ and 2, is subspace identifiable.
A.2 COMPONENT WISE IDENTIFICATION VIA SUFFICIENT CHANGES OF MULTIPLE DISTRIBUTIONS
Theorem 2. (Component-wise Identification with Complementary Gains) Let the observations be sampled from the data generating process in Equation 7 Suppose that Assumptions from Theorem I hold and also that we learn 9. Ps/u to match the marginal distribution between x and & with the minimal number of edges of the mixing process. We further make the following assumption:
A4 (Sufficient Changes for Component-wise Identification.): Suppose X2 and Xa be subset of x and X2 nx, = We let the z, be the set of latent variables that connect to x, but not connect to Xs, such that for all different values of Xa, such as Xa,0, (a.(I). the vectors V(a, k, u,) V(a 0) with ..... 2/zul are linearly independent. And the vector V(a, k, u,) is defined follows:
88 (8llog log pl (Z., 2s, Zk (1m)
8 log pl (1) Za, Zu umm)
V(o, k, um.)
(82,)
(0z,)²
(19)
log (1); no, ze (1m)
8 bloogpzz....
(mm)
8%
8z:
2,ERa
Suppose that we learn to achieve Equation 7 with the minimal number of edges of the mixing process. Then Za is component-wid -iss identifiable.
Proof. Based on Theorem 1, given different values of z and [Pa())) different domains, Equation 15 can be further derived as follows:
i log (lx/p(x(u) 02,
2 Oog p(x|u)
Oz:
O p(x/u)
02g 02,"
Σ
(20)
Ozi
02,
Oz,
2, E2
Sequentially, if there exists x,, x, nx; = 0. that connects to & and z, but does not connect to Zμ- Similar to Theorem given different values of xx.(1) and (a) subtract Equation 20 that
18

---

Published as a conference paper at ICLR 2025
corresponding to (1) with that corresponding to X0,(0) and then we have:
((.,)( Xb. 2:(1)
log pl (x), X6, 2x 2
02j i Öog pl X (1). Xb, u
Oāz
02,
i log p (0), Xb. Zz. zx(uu)
0z,
Σ
Ōz:
03g Xx x1=
Ozi
O2g Xx
2, #2
i log p( (1)+ Xb. 2k ())
dzi
a log p (0), Xb. Z1 (
dzi
E
θzi
g2 Xa (1)
Oz
0ā, x== x==
(2z,add, ER.
(21)
By combining Equation Ō (lgg((?.((1)) 11, 20, (m) à Ôlog p (0); 2o, Rx (u)
[4) and (1). we have:
8 alog [Jn]
8 log [Ja]
02;
02j
08, log pl (0),
(1)
8hg
Ö (log ( (1); 20, &x [u)
0z:
25. 22/11)
de:
0% 2 log ((a,
02g
Na (1)
Oz
02, Na xamXa.(0)
20,, 21c [))
Ozi
2 log p (0)
20. 2:/11x
Oz
#, ( Exo
Ozi
(1)
Ozi
(1.. 2o, 2x (n)
Oz.
a log p(?,,,0)) 2s. 21/11)
de
0%
Oz:
08,
2:Eaa
(22)
where Jhe and Jn, denote the the Jacobian with the values of (1) and (a, respectively. And then we further take the second-order derivative w.r.t â, where + j and have:
Oogg [Ju]
Olog
jJa
+
X. x1)
0ē,00
Xa Xx.,0010) (0)
Σ
02 log p(z)p(2) u)
0z; 0z,
82 log p(z) u) (0z,) 1
Oz,
dzi
)+
(0z,)2
0āj
0ā
Oh x. 10)
(1)
2, ER
i
02
i Olog p(z/u) Øz,
022
(23)
Oz,
02,0%1
X. x xx,(1)
02,821 1x. (Xa(0))
0z,
0zi
0z.
0z,
Σ (((a,kiiuu) u)
v(aa, k, k,,
02j
0ā,
X. x1=
02,
Oā, Xx (0)
R
82
8z
+ T(a Taa,ki, u.)
v(a, u)
02,021
02,02, (o)
xx.11)
Then we consider u.. u and let Equation corresponding to uo, and have:
(3
corresponding to u, subtract with that
dz. dz,
Oz dz,
0 Σ k,,, ZEER
v(a, k,, u.)
0E,
0E xx xa.1)
Dā, 0E (o.
8z2
Fzi
v(a, k u,)
ki, us)
02 xx (aa(1)
02,02,
x,, (0)
(24)
dz: 0zi
0z
0z:
Σ (u(a,k, uo) Z
-vaa, u,))
dz,
xx xx=(0
0E, X. xx=X,,(0)
82 02 x.=Xa,(1)
82
+ T(a, k, i, un)
T(a,k, uo)
02,02 Xa.
Under the generalized sufficient changes assumptions and further let three different values of x,. i..., Xa,()), and X2. so the linear system is a zzll x 6/z,| full-rank system. Therefore, the only solution is dz az:
02 dz. 0 and 8= 0.
D2,02,
19

---

Published as a conference paper at ICLR 2025
Note that 85 85, 02, Therefore, there is only at most one non-zero entry each row indexed by 2 E Ze in the Jacobian Jn. As a result, (axx))((xx)] .. 2 number for the component-wise identifiability is max(
d2 # for at most one element & [n]. 05. 0 implies that for each 2 Za D5,
is component-wise identifiable with n different domains, where 7e x |zal + 1). Therefore, for each latent variable 2,, i [ni], the necessary domain
A.3 GENERAL CASE FOR COMPONENT-WISE IDENTIFICATION WITH FULL-CONNECTED MIXING PROCESS
Corollary 1. (General Case for Component-wise Identification with Full-connected Mixing Pro- cess) We follow the data generation process in Equation 1 and make assumptions A1, A2, and A3. In addition, we make the following assumptions:
A6 (Fully Connected Mixing Process): Each latent variable 2, is connected to each ob- served variable zj. where i.j E [n].
Suppose that we learn 9 to achieve Equation different values of auxiliary variables u.
2z is component-wise identifiable with 2n
Proof. We conduct second-order derivation of Equation 14 w.r.t 2g and 2 and have:
@log((2lu) 02
72
02 log p(z u) Oz.
Oz.
θ log p(z; u)
(0z,)²
Olog (Jul
+
(25)
(
02j
02:
02,02
02,02,
Suppose we have 2n different values of i.e. uo un Ugn let Equation 25 corre- sponding to u, subtract with that corresponding to up and have:
82 log logg(zz/ue p(z) 82 log (o:g((;(uu)) 0z: Oz +( (0z,)2
2 log p(zluu)) Ologp(z;(u))
(0z,)²
0= Σ
(0z,)2
0&g 02:
dz,
dz;
02,081
(26)
With assumption A4, the linear system is 2n x 2n full-rank system, and the unique solution is 0z. 0z: = 0 and (az,)² 0. Since Jn is invertible and full-rank, for each of Jr, there is only 02, 0s āž,0:, one non-zero element, implying that latent variables z are component-we identifiable.
20

---

Published as a conference paper at ICLR 2025
u
1
Z2
3
1
Z
33
Z4
Z1
Z2
Z3
Z4
Z4
11
2
X3
X1
2
X
X
11
X
X3
X
X
(a) Dataset
(a) Dataset
Dataset C
Figure 6: Data generation process of the three datasets for synthetic data.
B SYNTHETIC EXPERIMENTS
B.1 SIMULATION DATASETS
The synthetic datasets are constructed in accordance with the causal graphs shown in Figure 6 Specifically, the mixing functions are synthesized using multilayer perceptrons (MLPs) initialized with random weights. The mixing functions incorporate the LeakyReLU activation function to en- sure invertibility. For each domain, we set different noises to ensure that has different distributions.
B.2 MEAN CORRELATION COEFFICIENT
Mean Correlation Coefficient, a widely used metric in ICA research, evaluates how well latent fac- tors are recovered. It works by calculating the absolute correlation coefficients between each ground truth factor and the estimated latent variables. If the recovered factors involve component- wise in- vertible nonlinearities, either Pearson's or Spearman's 's rank correlation coefficients are used accord- ingly. The best matching between ground truth and estimated factors is found by solving linear sum assignment problem on the correlation matrix, which is efficiently solvable in polynomial time.
B.3 IMPLEMENTATION DETAILS
Both the encoder and decoder in the proposed CG- VAE are composed of 5-layer multilayer per- ception (MLP) with the LeakyReLU activate function. 2. We trained the VAE network using the AdamW optimizer with learning rate 3e 3 and a mini-batch size of 64, the The coefficients for a and 3 are 5e 2 and le 3, respectively.
C RELATED WORKS
C.1 DISENTANGLED REPRESENTATION LEARNING
Disentangled representation learning plays an important role in unsupervised learning Li et al. Locatello et al., 2020b Peters et al. 2017). The key intuition (Bengio al., 2013) is that the disentangled representations should separate the distinct, independent, and informative generative factors of variation in the data. And each latent variable is sensitive changes in single underlying generative factors while being relatively invariant to changes in other factors. Based on this intu- ition, researchers further extend this concept to groups of factors (Bouchacourt al. 2018 Suter et al. 22118) Cai et al.) 2019, Higgins et al.). Several methods (Wang et al.) (0222) are proposed to learn disentangled representations. For example, some researchers employ the variational autoen- coders Kingma (2013); Burgess et al. (2018); Kumar et al.) (2017); Kim & Mnih (2018) to learn disentangied representation. Other works employ the GAN-based structures to learn disentangled representations. For instance, InfoGAN Chen al. (2016) achieves disentanglement by leveraging an extra variational regularization of mutual information. Recent works (Locatello et al. 2019a 2020a b) propose that unsupervised learning of disentangled representations is impossible, without any inductive biases on both the models and the data. Therefore, Locatello et al, (2020c) leverages pairs of observations to learn disentangled representation in weakly-supervised. And Träuble al.
21

---

Published as a conference paper at ICLR 2025
2021 learns disentangled representation from correlations data. Recently, Fumeer et al. (2023) obtains disentangled representation in multi-task models by considering that the features activate sparsely with respect tasks and are maximally shared across tasks.
C.2 NONLINEAR INDEPENDENT COMPONENT A ANALYSIS
Independent component analysis (ICA) provides the theoretical foundation for disentangled repre- sentation (Hyvärinen et al. 2023) and identifiability is important challenge. Previously, several works achieve identifiability by assuming the mixing processes are linear (Comon) 1994] Hyvärinen & Pajunen 1999 Hyvärinen 2013). However, the nonlinear ICA is challenging task since the la- tent variables are not identifiable without any extra assumptions on the distribution of latent variables
or the generation process (Hyvärinen al. 2024 Khemakhem al. 2020b; Zheng et al. 2022).
To solve this problem, some works leverage the auxiliary variables e.g. domain indexes. time in- dexes, and class labels to achieve sufficient changes for identification (Hyvarinen Morioka 20166] Khemakhem et al. 2020a. Kong et al. 2022 [i et al. 2023 Hälvä & Hyvarinen| 2020) Hälvä et al., Hyvariinn & Morioka||2017
Recently, other works achieve identifiability by using the sparse causal model. Specifically, Lachapelle et al. (2022) 2024); Zhang et al. (2024); Li et al. (2024) em- ploy sparse interactions between latent variables to achieve identifiability. And Moran et al. (202I); Zheng & Zhang (203);; Zheng et al. (2022) achieve identifiability by using structural sparsity sumption. However, either the sufficient changes or the structural sparsity assumptions are hard to meet in worldworld scenarios. Fortunately, find the complementary benefits between the sufficient changes and sparse mixing procedure, so we can mitigate these assumptions.
C.3 MULTI-DOMAIN IMAGE GENERATION
Multi-domain image generation focuses learning the joint distribution of multi-domain image data, even without paired data (Liu & Tuzel 2016 Pu et al. 2018 Xie et al. 2023). CoGAN Liu & Tuzel (2066) uses different generators for each domain while sharing high-level weights transfer domain-invariant information across domains. JoinIGAN Pu et all (2018) factorizes the joint distri- bution into marginal and conditional distributions learn. i-StyleGAN Xie al. (2023) connects multi-domain generation with identifiability by dividing the latent input into Ze and 2g. where &e captures
information, and 2, captures domain-speetie
information. Unlike condi-
tional GANs (Gong et al. 2019 Brock 2018 Miyato & Koyama 2018: Odena et al. 2017 Kang
Park 2020 Kang et al. 2021 Zhang al. 2019), which aim to generate images that conform to provided label information and enhance diversity to approximate the marginal distribution of each domain, multi-domain image generation seeks to model the relationships between different domains within unified framework. This approach not only captures the individual domain distributions but also the shared and domain-specinf characteristics across domains. When transitioning between domains, the objective is make minimal adjustments necessary to adapt to the new domain while preserving as much Comain-invarian information as possible.
D DISCUSSION OF THE THEORETICAL RESULTS
D.1 IMPLICATIONS
Based on the aforementioned theoretical results, we can relax the assumptions required for dis- entangled representation learning, which holds practical significance in real-world scenarios. For example, in the task of multi-domain image generation, using previous methods, is challenging to obtain the necessary number of domains to identify all latent variables. Although existing work in- troduces the assumption of minimal change to alleviate this issue to some extent, due to the unknown dimensions of the latent variables, the required number of domains still does not meet theoretical needs, resulting in unsatisfactory practical outcomes.
Meanwhile, our method has two advantages in solving such problems. First, the conditional inde- pendence caused by the sparse mixing procedure constrains the solution space of the linear full-rank system, thus decreasing the requirement the auxiliary variables. Second, even with insuffi- cient number of values of the auxiliary variables, we can still disentangle the latent variables that are of interest and relevant to downstream tasks (e.g., Zay in Figure ??), even if other unrelated latent
22

---

Published as a conference paper at ICLR 2025
Table 6: Experiment results of different model variants of CG-VAF.
MCC
Completeness
Number of Domain
CG-VAE
CG-FactorVAE
CG-TOVAB
CG-HFVAE
VAE
CG-FactorVAE
CG-TCVAE
CG-HFVAE
0.953
0.882
0,946
0.944
0.608
0.814
0.553
0.633
0.952
0.876
0.900
0.932
0.632
0.732
0.661
0.497
6
0.956
0.837
0.918
0.907
0.620
0.626
0.600
0.612
0.971
0.869
0.729
0.793
variables are not identifiable. For instance, in multi-domain image generation (such as controlled image generation with or without glasses), it is sufficient to identify the latent variable related to particular feature, such as glasses, with small number of domains. Therefore, our method is more likely to meet practical needs.
D.2 INTUITION OF SUBSPACE-WIL WISE AND COMPONENT-WIS. IDENTIFICATION
Subspace identification weaker than component identification, and hence naturally requires less information in the data generation process and distributions. Let start with the subspace identifiability, suppose we have two sets of latent variables with changing distributions denoted by 2, and ze, respectively. And 2, and ăe are the corresponding estimated variables. Morcover, we let the i- and j- component of z, and Ze be 2 and z!. respectivelly In order to see whether z, is identifiable
up to a subspace, we just need show whether 0z To this end, only the first-order derivative of the logarithmic density distribution is needed. However, for the propose of component-wiss identifiability, we have to make additional conditions. such as conditional independence between z and 2g given the all the remaining components. These conditions impose stronger constraints compared to subspace identifiability. Mathematically. this conditional independence implies that the second-order cross derivative of the logarithmic density distribution with respect 2; and 2 is zero, which involves second-order derivatives.
D.3 DISCUSSION OF ASSUMPTIONS OF THEOREM 2
In contrast to subspace identification, here we assume that the conditional distributions are second- order differentiable, and linear independence is necessary for unique solution to the full-rank linear system. Additionally, while the conditional independence between 2,z and x, implies sparse mixing procedure, our assumption on the sparsity is weaker than structural sparsity (Zheng et al.) 2022 Zheng & Zhang 2023). For example, the case shown in Figure achieves component-wise identiniailiiyy but violates the structural sparsity assumption. How to further amalyze the relationship between structural sparsity and the sparse procedure proposed in this paper is an interesting future direction. We also note that sparse mixing procedures common in real-world tasks. In multi- domain image generation, for instance, when transforming an image of a man into a woman, the latent variable representing gender affects only certain observed features, such as facial hair and hairstyle, while leaving the background and other facial features unchanged.
E MORE EXPERIMENT RESULTS
E. SCALABILITY OF DIFFERENT VAE VARIANTS
We also extend our method other VAE variants like TC-VAE Chen al. (2018). FactorVAE Kim & Mnih (2018). and HFVAE Esmacili et al. (2018). We named them and CG HFVAE, respectively. experiment on synthetic are shown in Table 6 According the ex- periment, we can find that the all the model variants achieve the ideal disentanglement performance even the number of domains is 2, which verify our theoretical results.
CG-FactorVas VAE,
E.2 OTHER CONSTRAINT ON MIXING PROCEDURE
In this paper, choose L1 norm enforce sparsity of mixing procedure since the L1 penalty induces a "sharp" constraint, encouraging the penalized quantities (like the coefficients linear re- gression the partial derivatives in formulation) to become exactly zero during optimization.
23

---

Published as a conference paper at ICLR 2025
Table 7: MCC results of the CG-VAE and CG-VAE-L.2.
Number of Domains
CGVAE-LI
CGVAE-L2
2
0.953 (0.007)
0.887 (0,071)
4 0.952 (0.017)
0,924 (0.011)
0.956 (0.016)
0.892 (0.025)
0.971 (0.009)
0.911 (0.035)
Table 8: Informativeness results on dataset of different methods
Number of Domain
CG-VAR
CG-GAN
iMSDA
iVAR
FactorVAR
Slow-VAF
heta- MAR
2
0.265 (0.018)
0.291 (0.028)
0.317 (0.028)
0.402 (0.164)
0.525 (0.013)
0.350 (0.046)
0.343 (0.045)
4
0.256 (0.045)
0.351 (0.032)
0.361 (0.045)
0.288 (0.007)
0.532 (0.040)
0.399 (0.014)
0.428 (0.036)
6
0,212 (0.016)
0.259 (0.019)
0.282 (0.001)
0.251 (0.090)
0.480 (0.017)
0.288 (0.051)
0.318 (0.037)
8
0.206 (0.016)
0.265 (0.038)
0.225 (0.013)
0.216 (0.018)
0.471 (0.012)
0.322 (0.037)
0.278 (0.051)
This characteristic is particularly desirable in scenarios where we aim to simplify the model or iden- tify the most relevant components in the mixing process. In the meanwhile, the L2 norm (squared sum) tends to stabilize training by uniformly penalizing the magnitude of all coefficients, it does not promote sparsity as effectively. Instead, typically results in coefficients with small but non-zero values, which may not align with our objective in this context. Therefore, the L1-norm to induce sparsity of the estimated mixing procedure, which aligns with the goal of our theoretical results. We replaced L2-norm with LI-norm, and conducted experiments which are shown in Table 7 According to the experiment, we can find that disentanglents perfor- mance of the model with L2 2-norm lower than that of LI, evaluating our statement.
be used
the synthetic datasets,
E.3 EXPERIMENT RESULTS ON OTHER METRICS
Experiment results on the Dataset A of informativeness, n2, and MSE are shown in Tables g and
10
respectively.
F
METRICS INTRODUCTION
To quantitatively analyze the disentanglement performance of different methods and settings, we employ six metrics: MCC, Completeness, Disentanglement, Informativeness, MSE, and R-squared.
MCC (Mean Correlation Coeflicient) evaluates the correspondence between the latent variables and generative factors. It caleulates the correlation matrix between the latent representations and the generative factors and uses the Hungarian algorithm to maximize the absolute values of the diagonal elements after sorting. The final MCC score is the mean of the absolute values of the diagonal elements.
Completeness (Eastwood & Williams, 2018): This metrie measures the degree to which each generative factor is captured by a single latent variable. It requires a training regressor to predict generative factors from latent variables and computes the entropy of the importance weights of latent variables for each factor.
Disentanglement (Eastwood & Williams, 2018):
This metric assesses whether each latent variable is primarily associated with a single generative factor. It is computed by training regressors to predict latent variables from generative factors and calculating the entropy of their importance distribution.
Informativeness (Eastwood & Williams 2018): his metric quantifies the total amount of informa- tion about the generative factors contained in the latent variables. It is computed as the root mean squared error (RMSE) between the predicted generative factors and the ground truth z.
Mean Squared Error (MSE) (Li et al. 2023): : By using LASSO as the regressor to predict from the latent representation 2, we use MSE between he predicted generative factors and the ground truth values on the test set to evaluate the disentanglement performance.
R²: The coefficient of determination, which is also common metric for the regression task.
24

---

Published as a conference paper at ICLR 2025
Table 9: R² results on dataset A different methods.
Number of Domain
CG-VAE
CG-GAN
iMSDA
iVAE
FactorVAE
Slow-VAE
B-VAE
2
0.931 (0.002)
0.916 (0.014)
0.854 (0.019)
0.843 (0.090)
0,656 (0.012)
0.854 (0.015)
0.848 (0.003)
0,930 (0.023)
0.876 (0.014)
0.883 (0.015)
0.904 (0.008)
0.728 (0.063)
0.834 (0.014)
0.799 (0.085)
0.950 (0.006)
0.937 (0:009)
(0.002)
(0.004)
0.766 (0.016)
(0.049)
(0.013)
8
0.955 (0.002)
0.938 (0.034)
0.946 (0.016)
0.941 (0.006)
0.790 (0.005)
0.897 (0.038)
0.912 (0.021)
Table 10: MSE results on dataset A of different methods.
Number of Domain
CG-VAE
CG-GAN
iMSDA
FactorVAE
Slow-VAE VAE
B-VAE
2
0.072 (0.009)
0.084 (0.017)
0.105 (0.009)
0.122 (0.020)
0.325 (0.009)
0.141 (0.030)
0.158 (0.013)
(0.031)
0.154 (0.032)
0.120 (0.005)
(0.006)
0,311 (0.037)
0.158 (0.003)
0.215 (0.010)
6
0.055 (0.005)
0.072 (0:015)
0.080 (0.000)
0.067 (0.009)
0.233 (0.003)
0.118 (0.044)
0.137 (0.020)
8
0.048 (0.005)
0.086 (0.001)
0.052 (0.007)
0.052 (0.002)
(0.010)
0.118 (0.040)
0.081 (0.028)
25


Reviews:
[{"review_id": "ZQq4IaT3f8", "reviewer": "Reviewer_Fw6r", "summary": "A disentangled representation learning framework with two types of identifiability guarantees has been proposed. By forcing a sparse mixing procedure, the subspace identifiability is ensured, while sufficient changes in the distribution of latent variables promote componentwise identifiability of latent variables.", "questions": "- In line 220, the index in $\\mathbf{z}_{o1}$ should be $o_1$ ?\n- Can you provide some intuition why Sufficient Changes for Component-wise Identification require the second derivative assumption, while this is not the case for the subspace Identification? This could help readers better understand the theoretical foundations of the proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Theoretical understanding of identifiability issue in the disentangled representation\n- Proposing two generative implementations based on VAE and GAN using a domain encoding network and sparse mixing constraints\n- Synthetic and real-world experiments", "weaknesses": "The notations are heavy and hard to follow. Please provide a comprehensive notation table at the beginning of the paper. and explicitly define each symbol when it's first introduced. For example, \n- In equation (2), what is the meaning of two subscripts of $\\mathbf{x}_{k, (1)}$. \n- Other notation also needs to be defined carefully, $\\backslash k$ (all index but $k$) ?\n- In equation (6), it seems there is a missing integral (the second integral has been omitted). \n\nThere are English typos and errors, e.g., line 157 (contributions --> contribute)\n\nIn synthetic experiments, there is no comparison with any nonlinear ICA methods. Comparing the CG-VAE, and CG-GAn with one or two relevant nonlinear ICA methods can be beneficial for readers to understand the advantages of incorporating domain encoding network and sparse mixing constraints in the disentangled representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ho8YXWFYP0", "reviewer": "Reviewer_4KGP", "summary": "In this paper, the authors investigated disentangled representation learning and proposed a new framework by combining two existing methods: sufficient changes and sparse mixing procedures. This framework can guarantee identifiability of the models, because the sparse mixing procedure can induce conditional independence between latent variables , which can simplify the mapping from estimated latent to ground truth. On the other hand, sufficient changes can further improve component-wise identifiability of latent variables. They implemented this framework for two types of deep generative models: VAEs and GANs. They evaluated on multiple datasets and show that their approach can result in better disentangled representations.", "questions": "What is the reason that the authors choose VAEs and GANs as the two specific types of models that they want to implement the framework with? I mean my main concern is that, since disentangled representation learning has been studied for almost ten years from now, there have been a lot of variants of VAEs that were developed for learning disentangled representations. So I wonder 1) whether their framework can be applied to those variants of VAEs, and 2) if it is possible to compare against those models, such as TC-VAE[3], FactorVAE[4], and HFVAE[5]\n\n\n[3] Chen, Ricky TQ, et al. \"Isolating sources of disentanglement in variational autoencoders.\" Advances in neural information processing systems 31 (2018).\n\n[4] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International conference on machine learning. PMLR, 2018.\n\n[5] Esmaeili, Babak, et al. \"Structured disentangled representations.\" The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Quality and Clarity: In general this paper is in good quality. The paper is well structured and the motivation is significant, and the writing is clear. The authors provide very detailed math derivations for the framework, and to my best knowledge, the derivations are sound. \n\nOriginality: While the paper leverages two existing methods for disentangled representation learning, it is interesting that they explore a framework that combines them in a compatible way, and this framework will also guarantee the identifiability of representation learning.", "weaknesses": "While they validated their framework on multiple datasets, they only provided qualitative results in terms of the disentanglement of learned representations. I would hope that they can evaluated the results more quantitatively and make comparison based on metrics of disentanglement, e.g. the metrics used in [1] and [2]. I guess the reason I want to see this is it is difficult to see the disentanglement only based on t-SNE plots, because t-SNE itself will employ an algorithm that perform dimension reduction into 2D space, so it is likely that this algorithm introduces certain level of disentanglement.\n\n[1] Eastwood, Cian, and Christopher KI Williams. \"A framework for the quantitative evaluation of disentangled representations.\" 6th International Conference on Learning Representations. 2018.\n\n[2] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International conference on machine learning. PMLR, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ke0yxvRqWr", "reviewer": "Reviewer_qELo", "summary": "In this paper, authors combine two assumptions for disentangling latent variables (latent identifiability). Thew whole work is based on non-linear ICA (identifiable VAE) with auxiliary variables $u$. Authors show that sufficient changes on the distribution of latent variables and exploiting structural sparsity assumptions on the mixing procedure can be complement each other to achieve better identifiability. Theorems are illustrated and proved in this paper. With both synthetic and real-world experiments, authors show that there proposed approach based on VAE and GAN can achieve good latent indentifiability.", "questions": "/", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The theories in this paper is presented and illustrated in a clear way. The schematic graphical model is good. Besides, there are solid prooves of those theorems.\n* The contribution is good, I can visually understand the usefulness of the proposed method.\n* There are both synthetic results validating the method, and real-world applications to show that the new method is useful.", "weaknesses": "* The sentence of Theorem A3 is a bit confusing. It is a bit hard to read that long sentence.\n* Line 158, why $x_k$ contribute $z_j$? And the derivative $\\frac{\\partial z_j}{\\partial x_k}$. Could the author explain why the relationship is not $z_j$ contribute $x_k$.\n* Line 349, \"$q(z|x)$  is the an encoder\"\n* Eq. 9, why loss $L_s$ is chosen to be the absolute sum rather than the squared sum? Will a squared sum be easier to train and more stable? Any reason or trade-off analysis on this choice?\nOr does absolute sum have sparsity?\n* What is CG-VAE-S? It seems like it was used before it was introduced.\n* For synthetic results, why not apply CG-GAN? Also, there are no other baseline methods for comparison.\n* It is also very common to compare the generated image with the true by MSE, $R^2$, or cross-entropy (log-likelihood). They are intuitive metrics that are worth seeing.\n* Same for the real-world, why not try CG-VAE. If there is some simple reason, authors should state in one sentence why VAE is not suitable for real-world datasets. Also, are there any other alternative methods for real-world, since all four methods are actually some method combinations? Please correct me if my understanding is wrong.\n* To me, it would be better to have more experimental results in the main content. The current experiment section is a bit simple. I'd like to see for example, more comprehensive comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)