Nonlinear ICA Using Volume-Preserving Transformations
Authors: Xiaojiang Yang
Yi Wang
Jiacheng Sun
Xing Zhang
Shifeng Zhang
Zhenguo Li
Junchi Yan
Venue: ICLR 2022 Poster (2022)
Keywords: Independent Component Analysis
Nonlinear ICA
Identifiability

Abstract: Nonlinear ICA is a fundamental problem in machine learning, aiming to identify the underlying independent components (sources) from data which is assumed to be a nonlinear function (mixing function) of these sources. Recent works prove that if the sources have some particular structures (e.g. temporal structure), they are theoretically identifiable even if the mixing function is arbitrary. However, in many cases such restrictions on the sources are difficult to satisfy or even verify, hence it inhibits the applicability of the proposed methods. Different from these works, we propose a general framework for nonlinear ICA, in which the mixing function is assumed to be a volume-preserving transformation, and meanwhile the conditions on the sources can be much looser. We provide an insightful proof of the identifiability of the proposed framework. We implement the framework by volume-preserving Flow-based models, and verify our theory by experiments on artificial data and synthesized images. Moreover, results on real-world images indicate that our framework can disentangle interpretable features.

Published as a conference paper at ICLR 2022
NONLINEAR ICA TRANSFORMATIONS
USING VOLUME-PRESERVING
Xiaojiang Yang'; Yi Wang', Jiacheng Sun?, Xing Zhang?, Shifeng Zhang", Zhenguo Li & Junchi Yan'i
Shanghai Jiao Tong University, 2 Huawei Noah's Ark Lab
bangxiaojiang, refzaction334, janjunchi } jtu. edu. en [sun junjhence,gl, zhangxing85, zhangshifeng,, li w...zhenguu thhanggio/euawwe..com
ABSTRACT
Nonlinear ICA is a fundamental problem in machine learning, aiming to identify the underlying independent components (sources) from data which assumed be nonlinear function (mixing function) of these sources. Recent works prove that if the sources have some particular structures (e.g. temporal structure), they are theoretically identifiable even if the mixing function is arbitrary. However, in many cases such restrictions on the sources are difficult to satisfy or even verify, hence it inhibits the applicability of the proposed methods. Different from these works, we propose a general framework for nonlinear ICA, in which the mixing function is assumed be aelume-preservingg transformation, and meanwhile the conditions on the sources can be much looser. We provide an insightful proof of the identifiability of the proposed framework. We implement the framework by preneeeing flow-based models, and verify our theory by experiments on artificial data and synthesized images. Moreover, results on real- wolld images indicate that our framework can disentangle interpretable features.
INTRODUCTION
Independent component analysis (ICA) is of the most fundamental problems in machine learn- ing. The earlier works concentrate on linear ICA (Comon, 1994), in which the observed data is assumed be linear and invertible transformation (called "mixing function")) of several indepen- dent components (called "sources").. and the goal is to identify the independent components from data points. Recently, there are increasing interests in nonlinear ICA (Hyvärinen & Pajunen, 1999), in which the mixing function is generalized to be nonlinear. This nonlinear problem is crucial, as it is the theoretical foundation of many important tasks. For example, one task disentangle- ment (Locatello et al., 2019; Sorrenson et al., 2020; Locatello et al., 2020), which aims at learning some explanatory factors of variation from data (Bengio et al., 2013) and hence facilitates the inter- pretability of representation learning well as its downstream tasks (Peters al., 2017; Lake al., 2017). Another example is controlling the generation process generative models using semantic factors (Karras al., 2019; Abdal et al., 2021). These tasks essentially require their latent variables to be identifiable in highly nonlinear latent models. Otherwise the latent variables can be mixing functions of explanatory semantic factors, hence are probably not explanatory or semantic.
The central problem in nonlinear ICA is the unidentifiability. Specifically, if the observed data points are independent and identically distributed (i.i.d. in short), i.e. there is temporal or similar structure in the data, then the sources are not identifiable essentially (Hyvärinen & Pajunen, 1999). This motivates many works to involve some structures data for identifiability guarantee.
To deal with the unidentifiability problem, most existing works of nonlicar ICA involve structures to data by restricting the sources in some plausible ways. One popular solution is to assume the sources have some kinds of temporal structures (Hyvarinen & Morioka, 2016; 2017). These works almost have no restriction on the mixing functions, but limit the data in the form of time series. Recent
This work was done when Xiaojiang Yang was research intern in Huawei Noah's 's Ark Lab. 1 "ooresponniig author is Junchi Yan. The SJTU authors with MoB Key Lab of Artificial Intelligence, Shanghai Jiao Tong University.

---

Published as a conference paper at ICLR 2022
works (Hyvarinen et al., 2019; Khemakhem et al., 2020) extend the applicability of nonlinear ICA by involving auxiliary variables like labels of classes, and assume that the priors of sources are members of exponential family. However, their works require many classes in a dataset and each class should not be isolated in data space (Sorrenson et al., 2020), which is difficult to be satisfied. To see this, note that many datasets (e.g. CIFAR-I0 (Krizhevsky et al., 2009)) just have very few classes, and usually each class is isolated in data space.
Another direction of involving structures to data is restrict the mixing function. One way is to assume that the mixing function is post-nonlinear (Taleb & Jutten, 1999), i.e. each data variable is a nonlinear function of linear combination of the sources. This is obviously overly restrictive, and hence is impractical. Some works provide few heuristic settings on learning algorithms to facilitate identifiability (Watters et al., 2019; Sorrenson et al., 2020), e.g. Sorrenson et al. (2020) empirically report that volume preservation of flow-based models promotes disentanglement.
In this work, we explore a balancing direction: involve some natural structures to the mixing func- tion, and meanwhile loosen the restrictions the sources. Specifically, motivated by the empirical evidences by (Sorrenson et al., 2020), we assume the mixing function to be volume-preserving (and weaker condition is provided in Appendix B), and seek some natural restrictions on the sources for identifiability guarantee. With mild conditions, we establish novel and fundamental identifiability theorems, which mainly show that if there exist two distinet classes in the datasets, the true sources are identifiable. The proofs are essentially different to existing works, and provide some insights for nonlinear ICA: the main indeterminacy of nonlinear ICA framework using volume-preserving mixing functions is the rotation of latent variables.
Based on our identifiability theorems, we implement the proposed nonlinear ICA framework using volume-preserving flow-based model (Dinh et al., 2014), which requires merely two classes of data according to our theory. First, we empirically show that with the most essential conditions on sources (two distinct classes with non-zero overlap) and mixing functions, the sources can be well identified, which implies that there might exist some extensions of our theorems with weaker conditions. Then we compare our implementation with the state-of-the -het identifiable model iVAE (Khemakhem et al., 2020) on artificial and synthetic image datasets, and show that our framework remarkably better results in terms of mean correlation coefficient. More importantly, we empirically show that besides the given two classes, involving more classes cannot improve identifiability, Moreover, ex- periments MNIST (LeCun et al., 1998) and CelebA (Liu et al., 2015) indicate that the implemen- tation is able to disentangle interpretable features using only two classes of data, which demonstrates the applicability of our theory. Our contributions are as follows:
1) We present a new solution to the unidentifiability problem of nonlinear ICA by using the general golume-preserving transformation, retical results lessen the requirement by existing works on the sources by reducing the number of required classes 2, at the slight cost of adding some natural conditions on the mixing function. Accordingly, all the restrictions are moderate, which facilitates the applicability of nonlinear ICA.
widely used in flow-based models. Specifically,
theo-
2) We establish two novel identifiability theorems. Specifically, by using the friendly and general polume-preservive property, they guarantee the sources can be identified up to a point-wise non- linearity and a point-wise linearity, respectively. The proofs suggest that the main indeterminacy of a nonlinear ICA framework with some natural conditions on mixing functions is simply the rotation of latent variables. The hope is to lay the foundation for a new path to identifiability.
3) We conduct experiments on both synthetic and real-world data to verify our theory, and point out that as long as the most essential conditions on sources and mixing functions are satisfied, our framework can well identify the true sources. This means that our theory can be further extended. Moreover, our framework remarkably excels the state-of-the- nonlinear ICA method, which indi- cates that our framework is more powerful to identify the true sources.
BACKGROUNDS
In this section, provide some backgrounds nonlinear ICA theory. We first give a general definition of nonlinear ICA, and then explain its central problem, namely unidentifiability. Based on this, we discuss some recent proposed solutions, including their advantages and disadvantages.
2

---

Published as a conference paper at ICLR 2022
2.1 DEFINITION AND THE KEY PROBLEM OF NONLINEAR ICA
Consider an observed random vector x E Rd, we assume it generated by an invertible nonlinear transformation f (called mixing function) using independent latent variables s (81, 8p) (called independent components or sources) as
x f(s),
where n S d (Khemakhem et al., 2020). In earlier ICA theory it is commonly assumed that n = d, L.e. the number of independent components should be the same as the number of observed variables. As it is commonly believed that n < d for real world data (Cayton, 2005; Narayanan & Mitter, 2010; Rifai et al., 2011), the definition above is much more practical.
The goal of nonlinear ICA theory is to identify (or recover) the independent components s; based on the observations of using estimating function g- Given the data distribution p(x), if there exists a transformation g such that for arbitrary f, g o f is a point-wise linear transformation, then the mixing model (Eq. 1) is said to be identifiable.
Unfortunately, if there are no more restrictions on the mixing model (Eq. 1), it is seriously unidenti- fiable (Hyvärinen & Pajunen, 1999; Locatello et al., 2019). Specifically, even if the components of the estimated latent variables z = g()(s) are independent, Z; can be the mixing of [s:]/=1- In other words, independence of components does not guarantee identifiability of the mixing model. E.g. suppose the prior p(s) is a factorial multivariate Gaussian, take a point-wise scaling on s such that p(s) becomes an isotropic multivariate Gaussian, then take rotation around the center, the inde- pendence of components is unchanged, but the obtained components are the mixing of the original sources. As arbitrary density functions can be transformed to a Gaussian (Locatello et al., 2019) 9), this example shows that for any prior, the independence of components is insufficient for identifiability.
2.2 EXISTING SOLUTIONS FOR UNIDENTIFIABILITY PROBLEM
To obtain identifiability, it is necessary to restrict the mixing model. There are two possible ways to involve restrictions: 1) restrict the sources 2) restrict the mixing function f.
In the first direction, one idea is to involve temporal structure to the sources s, i.e. assuming the sources are time series s(t) with time index (Harmeling et al., 2003; Sprekeler et al., 2014; Hy- varinen & Morioka, 2016; 2017). This solution guarantees identifiability of the mixing model with few restrictions on the mixing function, but is limited in the setting of time series. A more general way is to involve an auxiliary variable u and assume that conditioned on them, the prior of sourees s is factorial member of exponential family (Hyvarinen et al., 2019; Khemakhem et al., 2020);
p(s/u)= = II
(s)s) Z uu)
exp
Σ (Ts(s))ss(()))
k
(2)
where Tij, Xe,t. Z, are the so-called sufficient statistics, coefficients and normalizing constant, respectively. Q; is the so-called base measure, and is simply set to be 1 in many cases. l: is the order of the distribution, and distributions with higher k are more flexible. The auxiliary variable u is additionally observed, which can be time index, class label, etc. This solution seems general, but it requires nk 1 distinet values of u to guarantee identifiability, which is impossible in most datasets. For example, if we assume that in MNIST (LeCun et al., 1998), the prior of s is a factorial multivarite Gaussian (k= 2), and attempt to identify the sources using the 10 labels of the dataset based on the theory above, then we can identify 4 sources at most. However, the intrinsic dimensions obviously far more than 4 (Pope et al., 202 ), and hence we cannot identify the true sources.
In the second direction, there are very few works to obtain identifiability by restricting the mixing function f in nonlinear ICA theory. One solution is to assume f be post-nonlinear, i.e. {x,}ş_ are nonlinear functions of the linear combination of sources, then the mixing model is identifiable (Taleb & Jutten, 1999). This condition is too restrictive (Hyvarinen et al., 2019) in practice.
THE NONLINEAR ICA FRAMEWORK
In this section, we propose our general framework for nonlinear ICA (see Fig. 1).
3

---

Published as a conference paper at ICLR 2022
3.1 DEFINITION OF GENERATIVE MODEL
We assume that the sources are independent conditioned on auxiliary variable w.
n
p(s u) = II (s(s))
(3)
where p; is the i-th marginal distribution of the joint distribution p. Note that so far we do not restrict the density function of each source (u(s:lu) to any particular functional form. Hence our restrictions on the prior are looser than existing works (Hyvarinen et al., 2019; Khemakhem et al., 2020)
Next we introduce some coherent and basic restrictions on the mixing function f. First, assume it is homeomorphism ', mapping sources s to a vector of observed data variables n-dimennional Riemannian manifold M embedded in I
Geoenaiiv Model
Estimating Model Voluree Prosering
Volume Preserving
Conaitiiaal Insepenience
Factarial Garocion
Observed Variables
f: RD M
(4) Figure I: Structure and mild assumptions of our proposed framework. x, S, u and z are the ob- served data variables, the sources, the additionally observed variable and the estimated latent vari- ables, respectively. f and g are the mixing func- tion and estimating function, respectively, and both of them are preseerir.....gg. Conditioned on u, the sources s are assumed to be independent, while the estimated latent variables is required to follow a factorial multivariate Gaussian.
S
This restriction also natural, as it is the real world data often lies low dimensional man- ifold in high dimensional space (Rifai et al., 2011; Pope et al., 2021). We further assume it is volume-preserving (Dinh et al., 2014), i.e. the volume of each infinitesimal area on RR equals to that of the area on M obtained by this trans- formation. The volume-preserving property is:
det Jr(s) =
(5)
where Jr is the Jacobian matrix of the mixing function f from the view of differential manifold?.
Besides ooleeeipreeerving transformations, there exists a broader class of non-volume-preseee mixing functions guarantee identifiability. We introduce such class of functions in Appendix B.
3.2 DEFINITION OF ESTIMATING MODEL.
Given the generative model above, next we seek an estimating model identify the sources from x. First, we set the estimating function g as a homeomorphism mapping the manifold M to R":
M
-
X Z,
where z are estimated latent variables. Note the estimating function g is set to be volume- oolume-preserving:
det Jg- 1.
(7)
Note that the Dolume-preservatte here is a natural regularization to the estimating function for identifying the true sources. This is essentially different from the volume- volume-preservation preservation of mixing functions, which is restriction on the generative process for identifiability.
To identify s by z, naturally should also be conditional independent. Besides, we set the distribu- tion of z conditioned on u as a factorial multivariate Gaussian:
(z;
14 (4)))
q(z u) = II (,(u)
exp
(8)
20
Actually, the model above gives a constraint to the priors p(s[u). Specifically, q(z/u) must be the push-forward of p()|u) through g o f. As result, the restriction on above actually an implicit
A homeomorphism a continuous function between topological spaces with a continuous inverse function. 2 Here Je is a n Jacobian matrix of f relative the charts (i.e. neighborhood with coordinate system) of IR" and M, rather than d Jacobian matrix relative that of R² and Rdd So is Jg-1.
4

---

Published as a conference paper at ICLR 2022
Scaling
Rolution
L
Scaling
1
(a)
(b)
Figure 2: Sketch of the discussion about condition (iii). The blue and the green ellipses represent ((2) u (1) and v(z)e(()), respectively, and their lengths of major and minor axes correspond to of and 02 The red solid arrows represent the estimated latent variables (Z1, 22), while the red dashed arrows represent the obtained new latent variables (21, zí2. The black solid arrows represent two operations: scaling let one Gaussian (represented by blue ellipse) isotropic, and rotation around the center with arbitrary angle. (a) If condition (iii) is not satisfied, then the latent variables obtained by the two operations are still independent conditioned on both (1)) and 22) (b) If condition (iii) satisfied, then the obtained latent variables are no longer independent conditioned on (?))
To see this, note that gelume-preservingg tramsformation does not change the volume of a distribution's support. Hence, a multivariate uniform distribution cannot be transformed into multivariate Gaussian by volume-preserving function, because uniform distribution has limited support while the support of Gaussian is the whole space R". Thus the priors p(s[u) cannot be multivariate uniform distribution in our theory, and at least its support should cover R"
THEORETICAL ANALYSIS
It is obvious that a generative model defined by Eq. 3-Eq. 5 is still unidentifiable using an estimating model defined by Eq. 6-Eq. 8. For example, if f point-wise nonlinear transformation mapping p(s]u) to an isotropic multivariate Gaussian, and is rotation around the center with arbitrary angle, then the conditions Eq. 3-Eq. 8 are satisfied, but the obtained z; still the mixing of s.
Therefore, we further seek some mild restrictions to guarantee the identifiability of the generative model. The goal to prove that under some mild restrictions, the estimated latent variables are the recoveries of the sources up to point-wise linear transformation, i.e. g f is point-wise linear function. Our main result is the following thcorems:
Theorem (nonlinear identifiability) Assume data points are sampled from model defined by to Eq. 3-Eq, 8, and there exist two distinct observations of u, denoted by (1) and (2),, such that:
(i) Both and g have all second order derivatives.
(ii) ()((()) ()(2) Vi [n].
(iil)
(u()))
(1)
} are distinct.
Then g is composition of point-wise nonlinear transformation and permulation.
Remarks. The theorem (proof is given in Appendix A) guarantees that the sources of the gener- ative model above are identifiable up nonlinear point-wise transformation using the estimating model above, and hence the generative model is identifiable. In the following, we discuss the mean- ing of the conditions (ii) and (iii) above, and omit condition (i) since it is very mild and trivial.
Condition (ii) means that the two multivariate Gaussians (1)))(() and q(z)u(2)) are concentric, which guarantees that samples from the two Gaussians have large enough "overlap". Otherwise, if the centers of the two Gaussians have large distance, samples from these two Gaussians are likely to be absolutely disjoint when using Monte Carlo sampling. In this case, for the estimating function g there equivalently one Gaussian because can deal with samples from the two Gaussians separately, and hence identifiability is not guaranteed. In our proof (see Appendix A), however, it seems that this condition is not theoretically necessary, as the supports of the two Gaussians are the whole space and hence they theoretically have overlap everywhere. How to prove identifiability theorem without this condition is very important, and we leave this problem to future works.
Condition (iii) means that there do not exist two estimated latent variables such that their vari- ances of the two Gaussians are proportional. If this condition is not fulfilled, then we can easily construct another latent variables which mixing of the estimated latent variables, and hence
5

---

Published as a conference paper at ICLR 2022
an (2)
ca(u
let is scal- ing matrix and R is an arbitrary rotation matrix of order 2, hence SR is volume- pellee-preserving trans- formation. Then (z), 22) are independent conditioned on both ( and u²)), and hence they are also set of estimated latent variables, but obviously they are the mixing of (z1,z2). Therefore, condition (iii) is natural and necessary restriction for identifiability.
the identifiability is impossibly guaranteed. As shown in Fig. 2, assume
(i(
(z(u)))
(1("))))
(2),2) (2) where S diag (oat V Vor(w
Although (ii) and (iii) are conditions on the distributions of the estimated latent variables z, they are also implicit restrictions on the prior of sources s. To see this, note that g f(s) and o f is function constrained by Eq. 4-Eq. 6, and hence (ii) and (iii) implicitly restrict the prior through g f.
If we further restrict the prior p(slu) to multivariate Gaussian, then can reduce nonlinear indeterminacy to linear indeterminacy:
Theorem (linear identifiability) Assume the hypotheses of Theorem I hold, and p(s)u(() and (())u(2)) are multivariate Gaussians. Then g f is composition of point-wise linear transfor- mation and permutation.
In this case, the sources will have similar restrictions with the estimated latent variables. Specif- ically, let μx (u) and o "²(()) (u) be the mean and variance of p(slu), then have: (ii') 14 (z(())) u (2), Vi [n]]; and (iii') { (f((
(2())) I are distinct.
(f(mm)) (u()))
The proofs of the theorems above (see Appendix A) give us some key insights for nonlinear ICA: nonlinear ICA framework using volume-press preservnng mixing functions only has three kinds of inde- terminacy: point-wise nonlinear transformation, permulation and rotation of the latent variables. Therefore, resolving the indeterminacy of rotation by condition (iii) will lead identifiability. and the indeterminacy of point-wise wiie nonlinear transformation can be reduced to point-wise linear trans- formation by the Gaussianity of p(s)u).
METHODOLOGY
As our theory requires an invertible estimating function, we set it as volume-preserving Flow- based model (Dinh et al., 2014). The estimating function is denoted by g(:; 0). where 0 refers to model parameters. Given the data variables x, g(x; 0) denotes the estimated latent variables z.
Given a dataset D = (())).))),, ....)).... (I.M), ")),, (zx(2.1), u2(2),, ............... (..2..,,, u (2)))} with only two labels (1) and (2), we can construct a loss by our theory. As in our theory, the conditional distribution of the estimated latent variables should be factorial multivariate Gaussian,
have to push {g((((.)) | and (g(2() }) to q(z) ()) and q(z) (2)) (defined by Eq. 14) respectively. Thus, minimize the negative log-likelihoon following (Sorrenson et al., 2020):
Σ
(g, 0) ( - (og(o:(u))
((0) =E (x,
(9)
20
To optimize the loss above, the number of sources n is required, which is usually unknown. Fortu- nately, we can integrate dimension reduction into the loss, and hence the underlying dimensionality can be estimated by learning. Specifically, the loss above can be further simplified as follows, which has the ability of dimension reduction (see Appendix for detailed derivation and discussion):
2
n
(())
log o; (x
(10)
k=1
where (1, (k), 0)
V ()) [g. 0) ( (a((x; 0)] We utilize the simplified
version throughout our experiments.
EXPERIMENTS
Although our Theorem I does not explicitly restrict the functional form of p(s)u), and it implicitly requires p(s) u) to be identical factorial multivariate Gaussian, up poeuee-preserving transfor-

---

Published as a conference paper at ICLR 2022
mations. However, it is difficult to verify whether this implicit restriction is satisfied given dataset. Therefore, our experiments are mainly based on Theorem 2, in which the prior also assumed to be factorial multivariate Gaussian. Hence, the main goal of this section is to show the performance of our framework to achieve linear identifiability in this case.
6.1 PROTOCOL.
Datasets. We run experiments both on an artificial dataset and a synthetic image dataset called "Triangles", as well MNIST (LeCun et al., 1998) and CelebA (Liu et al., 2015). The generation processes of the artificial dataset and the synthetic images are described in Appendix D.
Model specification. To implement the volume preseeeving estimating function g( 0), choose oolum--preserving flow called GIN (Sorrenson et al., 2020), which is
version of artificial data, the network is constructed by fully
RealNVP (Dinh et al., 2017). For experiments connected coupling blocks. For image datasets: Triangles and MNIST, the networks are constructed by both convolutional coupling blocks and fully connected coupling blocks. The parameters of each network are updated by minimizing the loss function in Eq. 10 using an Adam optimizer (Kingma & Ba, 2014). Details of the networks and their optimization refer (Sorrenson et al., 2020).
Performance metric. To quantitatively compare the performance of different methods, we compute the mean correlation coefficient (MCC) (Khemakhem et al., 2020) between the sources and the es- timated latent variables. The computation of MCC consists of three steps: i) calculate all pairs of correlation coefficients between the sources and estimated latent variables; ii) solve a linear sum as- signment problem to assign one estimated latent variable to each source for highest total correlation coefficients; iii) take the average correlation coefficients of the obtained pairs. high MCC means that we successfully identify the true sources, up to point-wise transformations.
6.2 RESULTS ON ARTIFICIAL DATASET AND TRIANGLES
On artificial dataset and Triangles, the labels of Gaussians in the prior of sources are used as the ob- servations the auxiliary variables u required in our theory. We mainly aim at verifying our theory, and comparing our framework with the state-of-the-art nonlinear ICA method, namely iVAE (Khe- makhem et al., 2020), which based on variational autoencoders (VAE) (Kingma Welling, 2013).
6.2.1 ABLATION STUDY
To verify what conditions in our theory are essential for identifying the true sources, perform an abla- tion study on artificial data. The chosen conditions include i) the concentrie property of two Gaussians
Table I: MCC score 1 artificial data when part of the conditions are not satisfied. Conditions
Mean
Fully satisfied
1.000
((s)u(1)) and ((s)u(")) denoted by p (2(
(1 * ((( ( )
1 1.000
uf (u(())) ii) the variances of the two Gaussains
1.000=0.000
should be not proportional, denoted by ((
Non-volume Non-volume-preserviie f (1) (2)
1.000=10.000
()( iii) the the volume-preserrvng property of oi(n
0.999
No overlap
0.942 0.064
mixing function f; iv) the existence of two distinct observations auxiliary variable (1) and (2);; v) the existence overlap of samples from
p(s(u()) and ()) For verifying experiment of each condition, we synthesize an artificial dataset in which the condition is not satisfied.
The quantitative results of our framework when the conditions above are not satisfied are shown in Table 1, and the qualitative results are shown in Appendix E for the sake of brevity. Note that in the case of μî (x()) * pi( (u (2)), samples from p(s)ut ru (!) and (2)) have non-zero overlap.
Over these experimental results, we made the following observations:
1) As long samples from (())u( and (2)) have non-zero overlap, GIN will almost perfectly identify the true sources, even if the p(s u ()) and p(s (2) are not concentric. However, when the overlap vanishes. the performance GIN has a sharp decline. These results indicate that the condition (ii) in our Theorem 1 necessary, but non-zero overlap is crucial for identifiability.

---

Published as a conference paper at ICLR 2022
(a) Ground truth (b) Data points (c) Estimation by iVAE (d) Estimation by GIN
Figure 3: Qualitative comparison of iVAE and GIN on artificial dataset with two classes. (a) ground truth (observations of the sources) consists of samples from two Gaussians, which are visualized by two different colors. (b) 2-dim projection of the 10-dim data points. (c) and (d) estimations of the ground truth by iVAE and GIN, respectively. The plotted estimated latent variables are chosen by assignment of correlation coefficients between sources and all estimated latent variables.
(a) Traversals by iVAE
(b) Traversals by GIN
Figure 4: Qualitative comparison of iVAE and GIN on Triangles images. Each row is traversal by manipulating one estimated latent variable. The last image of each row represents the heat map (the map of changed pixels) of that row, generating by taking the difference of the 3rd and 7th images. Obviously, the four rows correspond to rotation, width, height and gray level, respectively.
2) When (f cy(u
(1((3))
(1(t))
u (2), the sources are unidentifiable according to our theo-
retical analysis, but the experimental results show that in these cases the true sources still be well identified. This is not contradictory to the necessity of condition (iii) in Theorem I and two distinct classes, but indicates that there exist some biases in learning process to reduce the indeterminacy of rotation of latent variables. We conjecture that one bias is mini-batch sampling. As in mini-bater sampling, the empirical distributions of different mini-batches can be very different, while according our theory, two different priors are almost sufficient to guarantee identifiability.
3) When the mixing function f (not the estimating function g) is neeevolluee-reservingg the recov- ery of sources can be somewhat unsuccessful. To see this, note that in Fig. 7(d), the recovery of ground truth has some visual distortions. This demonstrates that involving some restrictions to f (like oolume-preservaeee
or the condition introduced in Appendix B) is necessary.
6.2.2 QUALITATIVE RESULTS
Results on artificial data. As shown in Fig. 3, both iVAE and GIN can successfully identify the true sources from their nonlinear mixing data points. While GIN almost perfectly recovers the ground truth, yet the recovery by iVAE has some distortions everywhere.
Results on Triangles. For qualitative comparison of iVAE and GIN, we plot traversals of the es- timated latent variables by them, which are obtained by three steps: i) deal with the assignment problem of correlation coefficients between 4 sources and all estimated latent variables, and choose the assigned latent variables for traversals, denoted by {z, { ii) estimate the variances of the cho- sen latent variables [o:)__1: iii) manipulate the value of the i-th latent variable between (-2o,, 2a,) while keep other estimated latent variables unchanged, and then generate the i-th row of images.
In Fig. 4, the qualitative performances of iVAE and GIN are distinct. Each plot estimated latent variable of ivVAEE is a mixing of the true sources, e.g. the 4-th row of Fig. 4(a) is a mixing of rotation and color. Compared with this, the plot estimated latent variables by GIN obviously correspond to the true sources: rotation, width, height and gray level, respectively. This shows that our framework much better than iVAE qualitatively, and is able to identify the true sources from images.
These results demonstrates that our framework is much more powerful than iVAE for identifying true sources both artificial data and synthetic images with a few classes.
8

---

Published as a conference paper at ICLR 2022
7
7
7
7
7
7
7
7
7
/
(a) Rotation
(b) Bend of the vertical line
7
7
7
7
7
7
7 7
7 7
7 7
7 7 7 7
7 7
7
/
(c) Thickness
(d) Bend of the horizontal bar
Figure 5: Traversals of estimated latent variables w/ top 4 standard deviations by GIN on MNIST.
6.2.3 QUANTITATIVE RESULTS
The quantitative results are shown in Table 2, in which '-mc' means the model is trained and tested on a dataset with m classes. According to the theory by (Hyvarinen et al., 2019; Khemakhem et al., 2020), iVAE requires 9 classes Triangles and 5 classes set iVAE-9c, iVAE-5c and iVAE-2c for fair comparisions with GIN. As for GIN, we set GIN-9c, GIN-2c and GIN-Ic verify whether classes predicted by our theory are sufficient and necessary. As shown in Table 2, for identifying true sources, GIN excels iVAE w.r.t. MCC on both artificial data and synthetic images. This might be due to the coluee-preserving essence of GIN, which provides strong but natural restriction for estimating func- tions according to our theory. Also, besides the given two classes, increasing the number of classes cannot markedly improve the performance of GIN, while reducing number of classes to 1 leads to sharp de- cline of MCC. This means for GIN, two classes of data sufficient and necessary for identifiability. These results are consistent with our theory.
artificial data, hence
Table 2: Mean and STD of MCC score on synthetic dataset and Triangles of trials.
Methods Artificial Data Triangles
iVAE-9c
0.664
iVAFE-Sc
0.985+00.001
iVAE-2c
0.983=0.001
0.659=0.055
GIN-9c
1.000
0.857+0.033
GIN-2c
1.000=0.000
0.863 0.787=0.033
GIN-1c
0.999=0.001
6.3 RESULTS REAL WORLD DATASETS
To verify whether our framework is able to identify sources from real world datasets, we conduct experiments on MNIST and CelebA, and the setting and results on CelebA are reported in Ap- pendix F. On MNIST, we pick images with digits '1' and '7' because they probably have non-zero overlap, and use their labels as observations of the auxiliary variable. As there is no ground truth in MNIST, we cannot compute the MCC score, and hence the qualitative results are mainly reported. Moreover, the estimated latent variables with high standard deviations are chosen to plot, as such variables are probably more meaningful (Sorrenson et al., 2020). We hope the estimated latent vari- ables will correspond to some interpretable attributes in the dataset, which are viewed as true sources in disentanglement literature (Bengio et al., 2013; Locatello et al., 2019).
As shown in Fig. 5, the estimated latent variables with top 4 standard deviations are highly inter- pretable, corresponding to rotation, bend, thickness and bend of the horizontal bar. These results are comparable with fully utilizing the 10 classes of MNIST (Sorrenson et al., 2020). Therefore, here two classes are probably sufficient for identifying true sources, which is consistent with our theory.
7 CONCLUSION
We have explored new direction in nonlinear ICA: restrict the mixing function to be volume- preserving, and meanwhile relax the restrictions on the sources. With mild conditions, we establish two novel identifiability theorems, which guarantee the sources can be identified up point-wise non-linearity and a point-wise linearity, respectively. The proofs give insights for nonlinear ICA: if there exists some natural conditions on mixing functions, the main indeterminacy is simply the rotation of latent variables. This provides new paths and new techniques for identifiability guarantee.
For the applicability of our theory, we show in experiments that even most of the conditions are not satisfied, the true sources can still be successfully identified. This indicates that there exists stronger identifiability theorem with less conditions, which an appealing for further exploration. We also empirically show that a volume sieuee--eserring flow-based model using two classes significantly excels the ofate-of-the-art ICA method (iVAE), and able to learn interpretable attributes from real world datasets. These results show the advantages and applicability of our proposed franework.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by China Key Research and Development Program 22000A44700 and Shanghai Municipal Science and Technology Major Project (2021SHZDZZX0002)
REFERENCES
Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Styleflow: Attribute-conditioned exploration stylegan-generated images using conditional continuous normalizing flows. ACM Transactions Graphics (TOG), 4063):1-21,,021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:1798 1777798888888 1828, 2013.
Lawrence Cayton. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1, 2005.
Pierre Comon. Independent component analysis, a new concept? Signal processing. 36(3):287-314, 1994.
T. Cover and A.E. Gamal. An information-thhoreei proof of hadamard's inequality. IEEE Transac- tions on Information Theory, 29(6):930-93 1983.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.85// 2014.
Laurent Dinh, Jaseha Sohldickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations. 2017.
Stefan Harmeling, Andreas Ziehe, Motoaki Kawanabe, and Klaus-Robert Müller. Kernel-based nonlinear blind source separation. Neural Computation, 15(5):1089-1124 2003.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, volume 29, pp. 3765- 3773, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460 469.469..7 PMLR, 2017.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turne. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference on Artificial Intel- ligence and Statistics, pp. 2207-2211 PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1402,690,, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 131..61 2013.
arXiv preprint
Xiangyu Kong, Changhua Hu, and Zhansheng Duan. Principal component analysis networks and algorithms. Springer, 2017.
10

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
1998.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Confer- ence on Machine Learning, pp. 1348-6359) PMLR, 2020.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerehner. dsprites: Disentanglement testing sprites dataset. http://hww...oommeeeeemm////pppp datasstt, 2017.
Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. In Proceedings of the 23rd International Conference on Neural Information Processing Systems- Volume 2, pp. 1786-118 1794. 2010.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations foundaiie. and learning algorithms. The MIT Press, 2017.
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Repre- sentations, 2021.
Salah Rifai, Yann N Dauphin, Pascal Vincent. Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. In Advances in neural information processing systems, volume 24, pp. 2294- 2302, 2011.
Peter Sorrenson, Carsten Rother, and Ullrich Kothe. Disentanglement by nonlinear ica with general incompressible-llow networks (gin). In International Conference Learning Representations, 2020.
Henning Sprekeler, Tiziano Zito, and Laurenz Wiskott. An extension of slow feature analysis for nonlinear blind source separation. The Journal of Machine Learning Research, 15(1)9921-447, 2014.
Anisse Taleb and Christian Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing, 27100)2287--2200, 1999.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.00017, 2019.

---

Published as a conference paper at ICLR 2022
A PROOF OF OUR IDENTIFIABILITY THEOREM
We first prove some lemmas on Fuclidean space, and then prove our theorems on Riemannian man- ifold using these lemmas.
A.1 LEMMAS ON EUCLIDEAN SPACE
Let s ~ p(s/u), s E R", and there exists a homeomorphism as follows:
f R" + IR"
(11)
S Z.
Suppose f volume-preserving:
|det JF(s)] =
(12)
This volume-pree preeerving property directly leads to key corollary: denote the density of condi- tioned on u by q( u), then p(s = q (f(s))u). To see this, note that the change of variable rule gives p()(u) q (f(s) u) |det Je (s)], and substituting Eq. 12 leads the conclusion. This is the starting point of our proof.
Further suppose components of are independent, and g(z) u) is factorial multivariate Gaussian:
p(s|u) = II P: (s:lu), i=1
(13)
n
g
II
z exp 1-00,(u)) (1)z))
(14)
Lemma Consider the model defined by Eq. 11 11-Eq. 14, if have all second order derivatives, then there exist (u), (1,2(1))7= such that
(u) S (s) + (,)
0f, 0f: (s) =0.
(15)
0s,Osk
0s,
Osk
Proof Note that p(s/u) q (f(s) u), according Eq. 13-Eq. 14, we have:
Il (s(lla II
exp (u) (w))(())))))))
(16)
z
1=1
1=1
Take the logarithm, we have:
Σ (logp((s.lu)) = Σ (0 (u),((s) (u)) log 27)
(17)
Because f have all second order derivatives, fi has Taylor expansion: fi(so + As) = f((80) + (( As+ As H,, (s()) As). Lets = so + As in Eq. 17, and suppose f(( = without loss of generality, then take the Taylor expansion of f. we have:
n
(logp;(s:/ur (s:lu))
n (u)As (s)) (1) (VS((s)) O(AsT As) + C.).
(18)
where n, 01, (i and C. is constant.
Note that the left hand side of the equation above does not contain cross terms (As, Ask J while the right hand side does. Therefore, the coefficients of cross terms on the right hand side should be zeros:
Σ (u) S/
0s:
0h (s)
ÖsSOst (s) 28 (u) Os;
(s)
0. s
(19)
Oss
Let (u) (,,(1) (u) and (u) 20% (u), then the conclusion is obtained.
12

---

Published as a conference paper at ICLR 2022
Remark:
Denote the mean of qi by M4s and the variance by of. then 8", = and 8 20 Without loss of generality, we can set μ4 O, Vi, and hence 0,, = 0, Vi. Therefore, Eq. 15 can be further simplified as
Os: Ösj
0s: (8) ( v,
Σ (1.(
(8)
(20)
Osk
Lemma Suppose all conditions in Lemma I are satisfied, and there exists two distinct values of u, denoted by (1) and u22), such that the following conditions are also satisfied:
pe(ult)) (i(u(2)), Vi E [m);
ii {
(i(u()) (n((1))) ) ( (on
are distinct.
Then V E R", the Jacobian Jr(s) = (2) (s)
(55)
is a generalized permutation matrix.
nxn
Proof Without loss of generality, let (u:(u(()) 0,4i E [n]. according Lemma 1 and condition
i), we have
-i=1 (1)(((())) Oss
05: (s) 8i: (s) 0
(s)
V # k.
(21)
Σ -==1 (z
8h
(8)
(8)
= 0
Uhe
@ffs (8) )) and A(u)
Let 8(,,8) diag Ei= (1)
0s
En= (1)(())
851
diag (..).......... (n,2(2)), ()), then the equation above can be written as
J+(s)] ((u Jr(s) = E(u(1), Je(s) A( JF(s) = E(u(2),s)
(22)
Due to (u) (u) O, E(u, 8) and A Auu) are always positive definite, hence the equations above have square roots on both the two hand sides as follows:
(( 2) = (2) )(( ,)
U(u()))((t 8)1/2
(23)
where (u) is orthogonal matrices, i.e. UU)))((n) =
Therefore, we have
Jr(s) = (( (22 8)1/2
E((()) s)122
(24)
This leads to
A(u "aa)
Elu (3) s) 130) E(u (0,)s) s)/2
1/2
(25)
Both the two hand sides of the equation above can viewed as the singular value decomposi- tion (SVD) of () s)),s))//"(('' 8)) Note that due to condition ii), the diagonal entries of
(2122) (1))-111 are distinct. Based on this, according to the uniqueness of SVD (Kong et al., 2017), (l) is the composition of a permutation matrix and a signature matrix. Substi- tute such decomposition of ()) into Eq. 24, we finally conclude that Jr(s) is a generalized permutation matrix.
Remarks: Since Jr(s) is a generalized permutation matrix, in each row and each column it has exactly one nonzero entry. Without loss of generality, we can assume af: (s) a O, while 0f 05, (s) = 0, Vj # i. Therefore, f((s) is exactly the function of and is independent of {s, J and hence it can be denoted by f(();). Therefore, under some mild conditions in Lemma 2, z is identical to up nonlinear posnt-wise transformation (and permutation).
13

---

Published as a conference paper at ICLR 2022
A.2 THEOREMS ON RIEMANNIAN MANIFOLD
Suppose data points are distributed on a Riemannian manifold M which is homeomorphic to IR.. The homeomorphic be denoted by
RR + M
(26)
8 X,
which is further assumed to be volume-preserving:
|det Jr(s)]
(27)
The density of s conditioned on u is denoted by p(s)u), which is assumed to be factorial, i.e. {s: 1 independent:
n
p(s u) I (s/
(28)
==1
Suppose the data variable x can be encoded into vector of latent variables z with another homeo- morphism, denoted by
g M R" +
(29)
where the homeomorphism also
det -g) 1,
(30)
and conditioned on u, follows a factorial multivariate Gaussian, denoted by
7, p4 (u))²
c(2 u = II
[=1 Z: exp
20 2-
(31)
Theorem (Nonlinear Identifiability) Assume data points are sampled from a generative model defined according to Eq. 26-Eq. 31, and there exist two distinct observations of u, denoted by u (1) and (2), such that the following holds:
i) Both f and g have all second order derivatives.
ii) ()((()) (2( Vi [n].
Ca (2)
iii)
are distinct.
Then g f is composition of point-wise nonlinear transformation and a permutation.
Proof Note that gof: z is homeomorphism which volume-preserving |det (s) and has all second order derivatives. According to Lemma 2, we can conclude that g o is composition point-wise nonlinear transformation and a permutation.
Theorem (Linear Identifiability) Assume the hypotheses of Theorem hold, and p(s)u(1) (1) and (2)) are multivariate Gaussians. Then gof is a composite of point-wise linear transformation and a permutation.
Proof Let h o f. According to Theorem 1, h is composite of point-wise nonlinear transfor- mation and permutation, and hence without loss of generality, we write
(s) hss (s.), vi E [n].
(32)
Since p(s)ui is multivariate Gaussian, let p(s(u) II z exp (.",,(()s (u)s)) Sub- stitute this expression and the equation above into Eq. 17, have
0 (u)s: + (u)s? = O (s)) + 0% (s) + C. vi E [n),
(33)
where C; is constant. This obviously leads
(;;);) = a;s; + by, Vi E [n].
(34)
To conclude, g is composition of point-wise linear transformation and permutation.
14

---

Published as a conference paper at ICLR 2022
B A CLASS OF IDENTIFIABLE NVP MIXING FUNCTIONS
Even if the mixing function is non-volume-preservee (NVP), still leads to identifiability when its Jacobian has the following form:
det J+(s) = II hh (s.) > 0,
(35)
where hú is arbitrary function satisfying (s;)) > O. Obviously this class function is extension of pollue-ppreservee transformations. To see this, note that when [s;[s;)] 1. this class of function degenerate into volume- volume-preserviee preserving transformations.
Substitute the equation above into the change of variable rule p(slu) q (f(s) uu) |det Jr(s)), and take the logarithm, we have
(s) + log (s,(s))
Σ (0%, (
log 2!)
(17')
Using this equation, the proof of Lemma still holds, because the in the proof only use the fact that the left hand side of Eq. 17 has no cross terms, which is also the property of the equation above.
Therefore, the class of mixing functions defined above also leads identifiability, even if it is non- volume-preserving. This means that there exist some non-trival extensions our theory.
C SIMPLIFIED Loss FUNCTION
Consider when (4(u) and (u) are trainable, their optimal solutions of Eq. 9 have the closed forms as follows:
p (x)) (k) = =F (xx (x:(x;)))
(36)
(n (1)) =v ( [x; 0) - 16 (x()))))
where k: = 1,2. Note that the optimal solutions depend on the parameters 0, hence when given 0, we denote ut (x) and of (k) by (*(u((), 0) and (u( 0) respectively. Substitute these optimal solutions the loss function above, we finally obtain:
2
(
log o. (w
(37)
In experiments, we only sample mini-batch from the full dataset D for each iteration, hence in this case (u(u (k); = VB (); (x(())); 0)]] which is the biased standard deviation of [g: 0) 1) (x.()))
C.I DIMENSION REDUCTION
Here will demonstrate that the simplified loss above leads to dimension reduction. This is vital property for identifiability. On the one side, dimension reduction enable the learning algorithm estimate the unknown dimensionality of data manifold. On the other, if model to identify the true sources, then dimension reduction latent variables z necessary. as in flow-based models the dimensionality (d) far more than the number true sources (n).
flow-based
In flow- model, the simplified loss is
2
d
(())
=
log logo,(u (1) () 0)
Σ log II o ( (k) ()) A=1
(38)
k= 1=1
Hence minimizing the simplified loss is equivalent to minimizing I o (k); ()))). To minimizing this term, most latent variables should have almost zero standard deviations.
15

---

Published as a conference paper at ICLR 2022
Note that in implementation, the standard deviations of latent variables could not be exact zero, and not all standard deviations can be almost zero. First, in flow-based models, the data points are usually augmented by adding small noise, and hence the dimensionality of augmented data manifold exactly d. As a result, the dimensionality of the corresponding latent space is also d, hence exact zero standard deviation is not possible due to the reversibilityy of flow-based models. Moreover, as we assume the transformation g is volume-preserving distribution should be equal, and hence there must exist some latent variables with high standard deviations.
the entropy of data distribution and latent
Therefore, the simplified loss will assign almost zero standard deviations to most latent variables while preserving some high standard deviations. A natural solution is to assign high standard de- viations to meaningful latent variables, and assign almost zero standard deviations to redundant dimensions. This is exactly the dimension reduction ability of the simplified loss.
C.2 INDEPENDENCE ENHANCEMENT
Next we will prove that the simplified loss can enhance the independence of different components of z. This is desirable property, as p(z)u, should be factorial according
theory.
Our proof is based on Hadamard's inequality (Cover & Gamal, 1983): for any n x n matrix K, det K < ПЛ, K2 with equality iff K, det Cov(z, z) $ II, Var(z;). Then the simplified loss has a lower bound as follows:
Ví + J. Let K Cov(z. (z, z), then we have
2
(
log Var Var.. (2))
log det Covaa (z).
(39)
k=1
where the subscripts u (R) represent the conditions.
When the simplified loss is optimized, the gap between it and its lower bound will become zero, and hence the equality holds. This means Cov(z, Zz) = O, Vi * j. Therefore, minimizing the simplified loss can enhance the independence of different latent variables by reducing correlations.
In addition, note that our derivation above does not depend on the Gaussianity of q(z/u), and hence conclusion is general.
D DETAILS ABOUT THE ARTIFICIAL AND SYNTHESIZED DATASETS
Table 3: Means and variances of each class in Triangles. a and 3 represents uniform random variables [-1, 1] and [1, 2). respectively.
Factors
Value Ranges
Means
Variances
Rotation
π
0 + 0.01
0.03
Width
1.32]
18 + 2a
13
Height
1, 32
18 + 20
1,3
Gray Level
00, 255]
170 + 5a
218
To generate the the artificial data, following (Sorrenson et al., 2020), we synthesize the sources and observed data by two steps: First, two sources sampled from a 2-dim mixture of two Gaussians with their mean both (0.0, 0.0), and with variances (1.0, 0.5) and (0.5, 1.0), respectively. From each Gaussian, 000 points are sampled. Then an 8-dim standard Gaussian noise scaled by 0.01 is concatenated with them. ii) The observed data is generated from the 10-dim sources, by randomly initialized GIN (Sorrenson et al., 2020) with 8 fully connected coupling blocks.
To quantitatively test our method on image datasets, we have synthesized gray-scale image dataset called "Triangles". Specifically, we choose to generate a grey-scale 32 x32 image dataset of 2-D shapes similar with dSprites (Matthey al., 2017) as shown in Fig. 6, in which all shapes are right triangles generated from factors: rotation, width (length the horizontal edge), height (length of the vertical edge) and gray level. These factors viewed sources, and the observations of sources are sampled from a mixture of two Gaussians with random means and variances. The value ranges of four factors are [-1, r). 32). 32] and [0, 255), respectively.
16

---

Published as a conference paper at ICLR 2022
Figure 6: Samples from Triangles.
According to our requirement, in each class these factors follow a factorial multivariate Gaussian. To satisfied condition (iii) in Theorem 1, we set the variance of each factor as a proper random value, and so do the mean because found in experiments that the condition (ii) in Theorem is not necessary. The particular parameters of the four factors are shown in Table 3
The generating process of each image consists of three steps: i) for each class, randomly select four means and four variances according to Table 3, and obtain a 4-dim Gaussian; ii) sample one point from the 4-dim Gaussian, which corresponds right triangle; iii) judge what pixels locate inside the triangle according to the given rotation, width and height, and then assign them with the given gray level.
E QUALITATIVE RESULTS OF ABLATION STUDY
Here show the qualitative results of ablation study (see Fig. support discussion in an intuitive way. In these figures, reconstruction is the estimated latent variables with top two standard deviations, Sorrenson al. (2020) reported that estimated latent variables with higher standard deviations are probably more meaningful. Spectrum is the sorted standard deviations of latent variables. The spectrum of the estimated latent variables is in black, while the spectrum of true latent variables is in grey.
F QUALITATIVE RESULTS ON CELEBA
To further show the performance of our framework real-world datasets, we conduct experiments on CelebA (Liu et al., 2015) using GIN. Since our framework requires two distinct classes, we use all images in CelebA one class, and their mirror images as another class. Such setting ensures the two obtained classes are distinct and have non-zero overlap. Similar with experiments on MNIST, pick latent dimensions with high standard deviations estimated latent variables, and plot them to show their corresponding underlying factors.
As shown in Fig. 8, the estimated latent variables with top 4 standard deviations highly inter- pretable, corresponding to lighting on the left, lighting on the right, skin color, background bright- ness, hair (bang) and grin, respectively. These results further demonstrate that our framework is probably able to identify the true sources from real-world datasets, and hence support our theory.
17

---

Published as a conference paper at ICLR 2022
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
(a) Fully Satisfied
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
(b) pf ()) (2)
OBSERVED (PROJECTION)
TRUTH
SPECTRUM
(u)
(f
(w
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
(d) Non-volume-ppeserving f
OBSERVED DATA (PROJECCION)
GROUND TRUTH
RECONSTRUCTION
SPECTRUM
(c) (1) (2)
OBSERVED DATA (PROJECTION)
GROUND TRUTH
RECONSTRRUCTION
SPECTRUM
(I) No overlap
Figure 7: Qualitative results of ablation study.
18

---

Published as a conference paper at ICLR 2022
Figure 8: Traversals of estimated latent variables with top 6 standard deviations by GIN on CelebA Each row traversal by manipulating one estimated latent variable. The six rows represent lighting on the left, lighting on the right, skin color, background brightness, hair (bang) and grin, respectively.
19


Reviews:
[{"review_id": "vfhJAaN31w", "reviewer": "Reviewer_xbzu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think there are some merits in terms of the novelty of the paper. However, the setting for ICA appears to be different from the classical setup as it requires labels. In the classical ICA setup, there appears to be no labels in the dataset. Understandably you are trying to solve the same problem but you have labels to train the model. \n\nMy questions are:\n(1) How do you select $n$ in Equation (8)\n\n(2) How do you obtain \\mu (u) and \\sigma (u) in equation (8)\n\n(3) So does this approach still work if we don't have any labels for the data?\n\n(4) Since the transformations are volume preserving, does that mean you recover the source signals in the correct scale?", "summary_of_the_paper": "The authors propose to use volume preserving transformation to solve the disentanglement problem in latent variable models such as ICA. The authors have explained that in literature that ICA can be expressed as a factorial member of the exponential family, however, this arrives at a solution which is not identifiable. The authors have explained that there are two ways of achieving disentanglement, this could be done by either (1) restricting the sources or (2) restricting the function f.\n\n\nInterestingly that authors have proposed to use a volume-preserving transformation typically found in normalizing flows to be able obtain identifiability. There is a volume preserving transformation from the sources to the mixed signal which represents the generative process. Then there is a volume preserving transformation from the mixed signal to the latent space (which is not the source signal). Then a factorial multivariaate Gaussian is used to identify the conditional independence of each source signal.\n\nIn the classical ICA problem setup, the dataset does not contain any labels. While in this setting the input dataset for ICA appears to contain labels (u?).", "summary_of_the_review": "The paper proposes a simple and elegant solution to the disentanglement problem encountered in ICA. So I recommend in accepting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "siMnfznb9p", "reviewer": "Reviewer_kNre", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The nonlinear ICA problem is of significant research interest, and it is important to provide an identifiability guarantee for nonlinear ICA. In this work, the authors impose some reasonable assumptions on both the sources and mixing functions and provide two theorems for nonlinear and linear identifiability respectively. In particular, the theoretical results lead to the insight that the main indeterminacy of a nonlinear ICA framework using volume-preserving transformations is the rotation of latent variables. The authors further perform experiments to show that their method outperforms the state-of-the-art nonlinear ICA method iVAE. \n\nThis paper is well-written, and the motivations and contributions are clearly presented. ", "summary_of_the_paper": "In this work, the authors propose a framework for nonlinear ICA, in which the mixing function is a volume-preserving transformation, and the conditions for the sources can be relaxed compared to some prior works. The authors prove the identifiability of the proposed framework and implement the framework by volume-preserving Flow-based models. Numerical experiments on both synthetic and real data are performed to corroborate the theoretical results.", "summary_of_the_review": "Overall, my impression is that this work provides some solid contributions to the interesting nonlinear ICA problem. One issue I hope the authors can address is including the source codes in a code appendix.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pVx27R0Yys-", "reviewer": "Reviewer_7dbX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will admit that I'm not familiar with the recent developments when it comes to ICA and to Flow-based modeling of densities, but this seems like a good paper. It is difficult for me to assess the importance of the original contribution from the authors.\n\nQ: Does the term \"point-wise linear\" refer to the kind of model described in https://arxiv.org/pdf/2001.06988.pdf ?", "summary_of_the_paper": "The authors reformulate ICA in a way that makes it approachable with Flow-based methods. This yields a practical approach that the authors can apply to MNIST and to another dataset of that level of simplicity.", "summary_of_the_review": "This is a good paper even though I do not have familiarity with the recent developments on that specific topic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F66q6OWe0QR", "reviewer": "Reviewer_eqBB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " The framework is mainly reused from Sorrenson et al.'s work. The authors simplify the loss derived by Sorrenson et al.; however, they do not make clear why this is necessary or desirable. Their main contribution are two novel identifiability theorems, which state that under a few more conditions in addition to the volume preserving mixing function the true sources can be identified. Empirically, not all of the conditions seem to be necessary. Even when a non-volume-preserving map is used, the true sources of synthetically generated data can be reconstructed in their ablation study in section 6.2.1. This ablation study raises the question whether the formulation of this framework for nonlinear ICA is to rigid, i.e. model used is too restricted.\n\nThe method outperforms iVAE, which is an important baseline in the field.\n\nAs a minor command, I want to add that there is a typo in the first sentence of section 6.2: it should be \"variables\", not \"variabls\".", "summary_of_the_paper": "The paper proposes a framework for nonlinear ICA with that restriction that the mixing function is a volume preserving transformation. The authors prove that given this restriction as well as a few other conditions the sources are identifiable. The model set up is very similar to the one used in Sorrenson et al.'s work \"Disentanglement by nonlinear ica with general incompressible-flow networks (gin)\": A real NVP model is made volume preserving by a slight modification of the flow map and this flow is used to learn the mixing function. The sources are assumed conditional independent given an auxiliary variable $u$, which are the labels of classes in their experiments. The mixing function is learned by maximizing the Gaussian likelihood of the reconstruction sources, which is also conditionally independent given $u$.\n\nThe authors demonstrate empirically that some of the conditions of their identifiability proof might not be necessary because the sources can still be reconstructed empirically although the condition is not satisfied. Hence, identifiability might even hold true for an even larger class of problems. They also compare their method to iVAE and show that their framework can identify the true sources of generated data much better than this baseline. Furthermore, they apply the framework to MNIST and can identify a few important source variables explaining most of the variance of the dataset.", "summary_of_the_review": "I appreciate the two novel identifiability theorems. However, the ablation study  in the experiments section lets the conditions appear to restrictive. Furthermore, it is not clear what distinguishes this work from Sorrenson et al.'s article apart form the identifiability theorems. Therefore, I tend towards rejecting this article but I am open to enter a discussion with the authors and the other reviewers.\n\nEdit:\nGiven the helpful clarification for the authors to distinguish their work from Sorrenson et al.'s article I will raise my score by one.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1lnw5Uq1tdv", "reviewer": "Reviewer_kWud", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\tThe authors provide theoretical support for the use of the pre-existing GIN model for ICA. They propose novel assumptions and prove this leads to identifiability.\n-\tThe paper is very clearly written.\n\nWeaknesses:\n-\tThe authors claim that a volume-preserving mixing function is a natural restriction and is easily satisfied. I would like to see a stronger argument why this is true, as it seems easy to think of non-volume-preserving mixing functions. Such an argument should include why the triangle dataset and MNIST would be generated by volume-preserving mixing functions. \n-\tThe experiments find that none of the assumptions in the main theorem are necessary for identifiability to hold. More discussion about what necessary conditions would look like would improve the paper.\n-\tThe experiments are not very strong. Only quantitively analyzes the simple triangle dataset, so does not include a non-trivial dataset with known sources, so that identifiability can be quantified. Also, the experiments show large overlap with the experiments of [Sorrenson et al 2020]. An additional experiment with more complicated images with known sources would improve the paper.\n-\tIt is unclear why the model does not fully succeed in identifying the true sources in the triangle dataset. Is one of the assumptions not satisfied? Are there learning difficulties?\n\nFurther comments:\n-\tIt seems that there is a constraint that q(z|u) must be equal to the push-forward of p(s|u) through $g \\circ f$, in other words, that Figure 1 can be interpreted as a commuting diagram. However, this is never explicitly stated in the definition of the estimating model. Could the authors please clarify this?\n-\tI believe the appendix should be separately provides as supplementary material.\n\nTypos:\n-\tIn (18) in the appendix, z_0 should be s_0?\n-\tAbove (23) in the appendix, should positive-defined be positive definite? Or positive semi-definite?\n", "summary_of_the_paper": "[Sorrenson et al 2020] proposed to do nonlinear ICA via volume preserving mixing functions, when a variable u is observed that makes the latent variables conditionally independent. Via maximum likelihood optimization, the ICA model uses a volume-preserving encoder and then maximises a latent distribution, which is a factorized Gaussian conditional on the observed variable u.\n\nThe submitted paper does a theoretical identifiability analysis on that procedure. It assumes that the data is generated by an independent distribution, given observed u, then mixed by a volume-preserving mixing function. Furthermore, it assumes that both the mixer and the encoders are twice differentiable, there exist two observed u variables, u^1 and u^2 such that that the two resulting distributions of encoded latent variables have the same mean and that the set of ratios of the two standard deviations for each variable is distinct. If these assumptions hold, the paper proves that the original sources are identifiable up to a permutation and dimension-wise diffeomorphism.\n\nThe authors do three experiments: 2D mixture of Gaussians, triangle images and MNIST. For the first two, the authors quantitatively estimate the source identification and for all datasets, qualitative results on identification are given.", "summary_of_the_review": "The paper provides an interesting theoretical analysis of an existing method. I weakly recommend acceptance, as the assumptions are not sufficiently motivated, the assumptions are shown to not be necessary and the experimental evaluation is not very strong.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an identifiable nonlinear ICA model based on volume-preserving transformations. The overall approach is very similar to the GIN method published @ ICLR 2020. There is a weak consensus among the reviewers that this paper has some merit, although none pushed for acceptance. After reviewing the paper myself, I agree that the contributions here appear to be incremental, but the results do push this growing field of identifiable latent variable models forward.
Decision: Accept (Poster)