On the Identifiability of Nonlinear ICA: Sparsity and Beyond
Authors: Yujia Zheng
Ignavier Ng
Kun Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: identifiability
nonlinear ICA
unsupervised learning

Abstract: Nonlinear independent component analysis (ICA) aims to recover the underlying independent latent sources from their observable nonlinear mixtures. How to make the nonlinear ICA model identifiable up to certain trivial indeterminacies is a long-standing problem in unsupervised learning. Recent breakthroughs reformulate the standard independence assumption of sources as conditional independence given some auxiliary variables (e.g., class labels and/or domain/time indexes) as weak supervision or inductive bias. However, nonlinear ICA with unconditional priors cannot benefit from such developments. We explore an alternative path and consider only assumptions on the mixing process, such as Structural Sparsity. We show that under specific instantiations of such constraints, the independent latent sources can be identified from their nonlinear mixtures up to a permutation and a component-wise transformation, thus achieving nontrivial identifiability of nonlinear ICA without auxiliary variables. We provide estimation methods and validate the theoretical results experimentally. The results on image data suggest that our conditions may hold in a number of practical data generating processes.

On the Identifiability of Nonlinear ICA: Sparsity and Beyond
Yujia Zhengl.? Ignavier Ng', Kun Zhang!. 1 Carnegie Mellon University
Mohamed bin Zayed University of Artificial Intelligence fyujiazh, ignavierng, kunz1]&cmu edu
Abstract
Nonlinear independent component analysis (ICA) aims recover the underlying independent latent sources from their observable nonlinear mixtures. How to make the nonlinear ICA model identifiable up to certain trivial indeterminacies a long-standing problem in unsupervised learning. Recent breakthroughs reformulate the standard independence assumption of sources as conditional independence given some auxiliary variables (e.g., class labels and/or domain/time indexes) as weak supervision or inductive bias. However, nonlinear ICA with unconditional priors cannot benefit from such developments. We explore alternative path and consider only assumptions on the mixing process, such as Structural Sparsity. We show that under specific instantiations of such constraints, the independent latent sources can be identified from their nonlinear mixtures up to a permutation and component-we wise transformation, thus achieving nontrivial identifiability of nonlinear ICA without auxiliary variables. We provide estimation methods and validate the theoretical results experimentally. The results image data suggest that our conditions may hold in a number of practical data generating processes.
Introduction
Nonlinear independent component analysis (ICA) is fundamental in unsupervised learning. generalizes linear ICA (Comon, 1994) to identify latent sources from observations, which are assumed be a nonlinear mixture of the sources. For an observed vector x, nonlinear ICA expresses it as f(s), where f is unknown invertible mixing function, and is latent random vector representing the (marginally) independent sources. The goal is to recover function from the observed mixture x up to certain indeterminacies. While nonlinear ICA is of general interest in a variety of tasks, such as disentanglement (Lachapelle et al., 2022) and unsupervised learning (Oja, 2002), its identifiability has been long-standing problem for decades. The key difficulty is that, without additional assumptions, there exist infinite ways to transform the observations into independent components while still mixed w.r.t. the sources (Hyvärinen and Pajunen, 1999).
well as sources
To deal with this challenge, existing works introduce the auxiliary variable u (e.g.. class label, domain index, time index) and assume that sources are conditionally independent given u (Hyvärinen and Morioka, 2016, 2017: Hyvärinen et al., 2019; Khemakhem et al., 2020; Lachapelle et al., 2022). Most of them require the auxiliary variable to be observable, while clustering- based methods (Willetts and Paige, 2021) and those for time series (Hälvä and Hyvärinen, 2020; Hälvä et al., 2021; Yao al., 2021) are exceptions. These works impose mild restrictions the mixing process but require many distinct values of u for identifiability, which might restrict their practical applicability. Motivated by this, Yang et al. (2022) reduce the number of required distinct values by strengthening the functional assumptions on the mixing process. As described, all these results are based on conditional independence of the sources given auxiliary variable u, instead of the standard marginal
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

independence assumption, to achieve identifiability in specific tasks, thanks to the weak supervision provided by the auxiliary variables. The identifiability of the original setting without auxiliary variables, which is crucial for unsupervised learning, stays unsolved.
Instead of introducing auxiliary variables, there exists the possibilityy to rely further restrictions on the mixing functions to achieve identifiability, but limited results are available in the literature over the past two decades. With the assumption of conformal transformation, Hyvärinen and Pajunen (1999) proved identifiability of the nonlinear ICA solution up to rotation indeterminacy for only two sources. Another result shown for the mixture that consists of component wise nonlinearities added to linear mixture (Taleb and Jutten, 1999).
In this work, aim show the identifiability of nonlinear ICA with unconditional priors. We prove that without any kind of auxiliary variables, the latent sources can be identified from nonlinear mixtures up component-wise invertible transformation and permutation under assumptions only on the mixing process. Specifically, we mainly focus on the structural sparsity condition w.r.t. the structure from sources to observed variables, i.c., the support of the Jacobian matrix of the mixing function. Under structural conditions on the mixing process, one could identify the true sources by sparsity regularization during estimation (Thm. 1). By removing rotation indeterminacy, structural sparsity benefits the identifiability of linear ICA with Gaussian sources as well, which was previously thought be unsolvable (Prop. 1, Thm. 2). Besides, we also develop another identifiability theory based on Independent Influences, which is an analogous concept of "sparsity" regarding the individual influencing process instead of the support (Prop. 2). We establish, to the best of our knowledge, one of the first identifiability results of the original nonlinear ICA model in a fully unsupervised setting.
Moreover, we also provide identifiability results for the undercomplete case, where the number of ob- served variables is larger than that of sources (Thm. 2, Thm. 3). This is of great practical interest and cannot be handled with previous conditions on nonlinear ICA, even with the help of auxiliary variables (Hyvärinen et al., 2019). We validate our theoretical claims experimentally, and the results on the im- age dataset suggest that our conditions appear to be reasonable for practical data generating processes.
2 Preliminaries
We consider the following data-generating process of ICA
(s()) II Ps. (s,), 1=1
(1)
x = x=((s),
(2)
where n denotes the number of latent sources, X = (x x,) denotes the observed random vector, s = (S,........ sn) ) is the latent random vector representing the marginally independent sources, Ps, is the marginal probability density function (PDF) of the i-th source is the joint PDF of random vector s, and f: x denotes nonlinear mixing function. For linear ICA, Eq. (2) is restricted as
x As,
where A = [A., ... A. denotes the mixing matrix. The goal of ICA is to learn an estimated unmixing function f- x + (or alternatively in the linear case, estimated unmixing matrix A -!),, such that S ($1. ......... ...)) consists of independent estimated sources.
Notations. Throughout this work, for any matrix M, we use M,,; to refer to its ith row, and M.,; to indicate its jth column. For any set of indices S c (1,...,, x (1....,n)) we analogously denote St. = (j (i. E S) and S., = (i (i.j) E S). We denote by the cardinality of a set. We also define the following technical notations.
Definition 1. Given a subset S C {1,...,. n.. the subspace RS is defined as
R3 = (= E R. i E S
- 2, =3-
In other words, IR'S refers to the subspace of I. indicated by an index set S. In the following, we define the support of matrix.
Definition 2. The support of matrix M E R is defined as
supp(M) {(i,]) Mi # 0}
(5)
2

---

With slight abuse of notation, we also reuse the notation supp to denote the support of a matrix-valued function, depending on the context. Definition 3. The support of function M G - R mmn is defined as
supp(M(()) = ((i.5) 30 E 03.
3 Identifiability with Structural Sparsity
3.1 Nonlinear ICA with Unconditional Priors: Structural Sparsity
In this section, we consider the identifiability of nonlinear ICA with unconditional priors, where the mixing function f in Eq. (2) is nonlinear. Different from recent breakthroughs in identifiable nonlinear ICA with side information (Hyvärinen et al., 2019; Lachapelle et al., 2022), in our setting, sources Eq. (1) do not need be mutually conditionally independent given an auxiliary variable. Instead, our setting is consistent with the original ICA problem based on general marginal independence assumption of sources. Besides, no particular form of the source distribution is assumed, which is different from most works that directly assume an exponential family (Hyvärinen al., 2019). The goal of identifiable nonlinear ICA is estimate the unmixing function f-1 so that the independent sources are identified up to a certain indeterminacy, which is usually a composition of component- wise invertible transformation and permutation (Hyvärinen and Pajunen, 1999). We formulate the structural sparsity condition below and present the identifiability result with its proof shown in Appx. A.1. It is worth noting that Lachapelle et al. (2022) leverage sparse mechanism between latents and auxiliary variables for disentanglement, by which part of the assumptions and proof technique are inspired. For brevity, we denote F and f as the support of the Jacobian Jr(s) and J (8), respectively. And T is matrix with the same support of ((s) in Jş(S) (s)))
Theorem 1. Let the observed data be sampled from nonlinear ICA model as defined in Eqs. (1) and (2). Suppose the following assumptions hold:
Mixing function f is invertible and smooth. Its inverse is also smooth.
it. For all E fl....,n. and j E Fs., there exist {s(0) Tt=l and T s.t.
span(Jr(s(0)) ). 1F = R7, and [Jr(sl(yT],, ER
F S [..
iv (Structural Sparsity) For all k {I,...,n], nn,, there exists Cs such that
n Fi. = (k). IEC
Then = f1 o f a composition of component-wise invertible transformation and a permutation.
Assumption ii in generic sense rules out a set of specific parameters to avoid ill-posed conditions such as the Jacobian is partially constant and is almost always to be satisfied asymptotically. Assumption ii corresponds to incorporating suitable sparsity regularization term into the estimating process with no restriction the minimal Lo norm among all functions that allow the model to perfectly the true distribution of sources. Thus, it indicates that we should estimate the mixing process by maximizing its sparsity. Our discussion will focus on the assumption of structural sparsity, which is the core of our theory.
the ground truth. It helps to find the estimated mixing function with
We start with the intuition of the proposed direction on exploiting sparsity on the mixing process for identifiability. As discussed in previous works (Hyvärinen and Pajunen, 1999), there exist infinitely many ways to preserve the independence among variables after mixing them up, such as Darmois construction (Darmois, 1951) and measure-presse prreuurirp automorphism (MPA) (Hyvärinen and Pajunen, 1999). Besides, sources with non non- Gaussian priors could be transferred to marginally independent Gaussians by trivial woint-wise transformations (Hyvärinen et al., 2019). Thus, the existing strategy of exploiting independence and -on-Gausiinnity for identification fails in the nonlinear case. However, the true structure is sparse enough, any alternative solution that produces any indeterminacy beyond component- wise transformations and permutations (c.g., mixtures or rotations) may correspond to
3

---

denser structure. In the light of that, instead of restricting the functional class (e.g., post nonlinear models (Taleb and Jutten, 1999) or conformal maps (Hyvärinen and Pajunen, 1999)) or introducing auxiliary variables for extra structural dependencies between latent sources (Hyvärinen et al., 2019; Lachapelle et al., 2022), we leverage the sparsity pattern, i.e., the support of the Jacobian of the mixing function, to identify the sources.
The structural sparsity assumption (Assumption in Thm. 1) implies that, for every latent source S;, there exists a set of observed variable(s) such that s, is the only latent source that participates in the generation of all observed variable(s) in the set. Graphically, for every latent source s, there exists a set of observed variable(s) such that the intersection of their/its parent(s) is s; (e.g., for in Fig. 1, there exist x1 and X2 such that the intersection of their parents is 81.). This encourages the connections to be sparse enough to make the disentanglement of each source based on structural conditions possible. Together with the sparsity regularization during estimation (Assumption iii) and other mild conditions, Assumption iv illustrates a graphical pattern of nonlinear ICA models that could be identified by exploiting structural sparsity. It indicates a structural criterion of the sparsity that is needed for identifiability. Beyond that, structural sparsity is also of practical interest for the interpretability of results (Zhang et al., 2009) and identification of other latent variable models (Rhodes and Lee, 2021). Mcanwhile, various versions of Occam's 's razor (e.g., faithfulness (Spirtes al., 2000), minimality principle (Zhang, 2013), and frugality (Forster et al., 2020)) are also fundamental to the identifiability of the underlying causal structure. Readers may refer additional discussion in Sec. 6.
S2
S3
S4
S5
X2
X3
X5
The struc- tural sparsity assump- tion of Thm. 1, where the matrix represents supp(Jr(s)).
Figure 1:
With structural sparsity. Thm. 1 shows the identifiability of nonlinear ICA while maintaining the standard mutually marginal independence assumption of sources. This is consistent with the original setting of ICA and plays an important role in a more general range of unsupervised learning tasks, compared to previous works relying on conditional independence given auxiliary variables. Moreover, one may modify Thm. I to consider the generating process based on auxiliary variables by imposing similar restrictions
the mixing process. Different from previous works (Hyvärinen et al., 2019; Yang et al., 2022), this extension removes common restrictions on the required auxiliary variables, such the number of distinct values (e.g., class labels), non-nronpeeee con-proportional variances, or sufficient variability. Therefore, although our main result (Thm. 1) focuses on nonlinear ICA with unconditional priors, it also provides potential insight into improving the flexibility of utilizing the auxiliary variable when it is available. As a trade-off, the additional assumptions on the sparsity might limit its usage.
Besides, under additional assumptions, we could reduce the indeterminacy in Thm. I and further identify sources up to a component-wise linear transformation. Most works in nonlinear ICA focus on the identifiability up to a component wise invertible transformation and permutation (Hyvärinen and Morioka, 2016; Khemakhem et al., 2020), which is also called pernutation-identiffbllli in the literature of disentanglement (Lachapelle et al., 2022). This indeterminacy is trivial compared to the fundamental nonuniqueness of nonlinear ICA and is analogous to the indeterminacy involving rescaling and permutation in linear ICA (Hyvärinen and Pajunen, 1999). Recently, Yang al. (2022) provided an identifiability result based on auxiliary variables that further reduces the indeterminacy of the component-we nonlinear transformation variables, we show conditions for the identifiability of nonlinear ICA with unconditional priors to component-wise linear transformation, with the proof in Appx. A.2. The reduced indeterminacy may give rise to a more informative disentanglement and open up the possibility for further improving the quality of recovery with only element-wiss wise linear operations, such as the Hadamard product.
linear one. Inspired by it but without auxiliary
Corollary 1. Let the observed data be sampled from a nonlinear ICA model as defined in Eqs. (1) and (2). Suppose the following assumptions hold:
i. The function h = f- f is composition of a component-wise invertible transformation and a permutation.
it The mixing function f is volume-presente
iii The source distribution Ps(s) is a factorial multivariate Gaussian.
Then h := -f-1 of is composition of a component-wiss linear transformation and permutation.

---

3.2 Removing Rotation Indeterminacy with Structural Sparsity
Rotation indeterminacy is one of the major obstacles the identifiability of ICA. Linear ICA exploits the maximization of the non- Gaussianity of the estimated sources (e.g.. Kurtois (Hyvärinen and Oja, 1997)). As direct result, the typical assumption is that at most one of the sources can have Gaussian distribution. In contrast, for the nonlinear case, a trivial point-wise function could transform sources to have any marginal distribution including Gaussian thus invalidating the effect of non- Gaussiantty. Therefore, removing rotation indeterminacy remains critical for the identifiabilin in the nonlinear case. For instance, "rotated- -aaiissann MPA produces nonuniqueness nonlinear ICA due to rotation indeterminacy (Hyvärinen and Pajunen, 1999). It first maps the distribution an isotropic Gaussian, then applies a rotation and maps it back without affecting its original distribution. To address for identifiability, Yang et al. (2022) assume that there do not exist two conditionally independent Gaussians such that their variances are proportional.
In the previous section, we have introduced the structural sparsity for the identifiability of nonlinear ICA. One may wonder whether it straightforwardly leads to the identifiability of linear ICA with Gaussian sources (Gaussian ICA), which was previously perceived to be impossible due to its rotation indeterminacy. The idea simple-any rotation of the true mixing matrix A will be less sparse if A satisfies the proposed structural sparsity assumption. W While the joint distribution of Gaussian sources stays invariant across rotations, the sparsity of the mixing matrix (i.e,, Lo norm) keeps changing. We first consider Assumption iv in Thm. I for Gaussian ICA as extension. Because the Jacobian of lin- ear mixing function is fixed, Assumption ii of Thm. does not directly hold in the linear Gaussian case. As result, cannot directly apply Thm. here and therefore propose some alternative conditions: Proposition 1. Let the observed data be sampled from linear ICA model defined in Eqs. (1) and (3) with Gaussian sources. Suppose the following assumptions hold:
Mixing matrix A is invertible.
if. There exists a matrix À s.t. for all j E supp(A)).. supp ))) E T
"upp(AA).."
ii |upp((A] S (u)p((A)]
iv. (Structural Sparsity) For all ie E (I,...,,]. n], there exists Cs such that
n supp(A..) [k}.
iEC
Then A ADP, where D is a diagonal matrix and P a column permutation matrix.
The proof is provided in Appx. A.3. Similar to Assumption ii of Thm. 1, Assumption ii of Prop. 1 excludes specific set of parameters that makes structure-based identification ill-posed.
The undercomplete case. For the full identifiability of nonlinear ICA, it is noteworthy that the invertibility of the mixing function is technically essential in our result (Thm. 1). This invertibility implics that the number of observed variables is equal to that of sources. Although that setting is rather common in the literature, especially for the theory of nonlinear cases (Hyvärinen and Pajunen, 1999; Hyvärinen et al., 2019), undercomplete ICA (i.c., there are more observed variables), may be of more practical interest and more challenging (Joho al., 2000). Therefore, we develop an alternative structural sparsity condition for the undercomplete case. As a trade-off, in the nonlinear case, the structural condition that removes the requirement of invertibility could only deal with the rotation indeterminacy. Thus we start from Gaussian ICA to introduce the condition.
Definition 4. Given matrix S E Rmxn, we define the function overlap Rm + {0, mp
(overlap(S))g = { if = 1 and it is not the only nonzero entry in Si.:
otherwise.
Theorem 2. Let the observed data be sampled from linear ICA model defined in Eqs. (1) and (3) with Gaussian sources. Differently, the number of observed variables (denoted as m) could be larger than that of the sources n, i.e., m n. Suppose the following assumptions hold:
The nonzero coefficients of the mixing matrix are randomly drawn from a distribution that is absolutely continuous with respect to Lebesgue measure.
5

---

ii. The estimated mixing matrix A has the minimal Lo norm during estimation.
ii. (Structural Sparsity) Given C S 21, where [C] 1, let Ac E "M represents
submalrix of A E Rmxx consisting of columns with indices C. Then, for all k E C, we have
U supp(Aw) - (A)) > |supp(An)]-
k'ec
Then A ADP with probability one, where D a diagonal matrix and a column permutation matrix.
The proof is provided in Appx. A.4. Intuitively, Assumption ii in Thm. 2 encourages the variability between the influences of each source. For each source, its influence on the observations should be as distinctive as possible, which is correlated with the sparsity of the mixing matrix. Fig. 2 is illus- tration of the assumption. In that case, k = and = {1, (.).. Let Ae
Aq
Aa
denotes the submatrix shown in the figure, where
U supp (F+) = 7. Blue A'EC
dotted square denotes (Ac) and we have rovorlv = Thus,
U supp(Ax)) - rank(overlapp) (c)) (the number of black dots), which k'ec
Figure 2: As- sumption ii of Thm. 2.
larger than (supp(A,) = For the nonlinear case, we prove that could help to distinguish spurious solutions due to the rotation indeterminacy, i.e., the Protated-Gaussiar " MPA (Defn. 2.5 in Gresele et al. (2021)). Gresele et al. (2021) prove that this class of MPAs could be ruled out with assumptions of conformal maps, non-Gaussianity and orthogonality of the Jacobian of the mixing function. Differently, we address it with the proposed structural sparsity condition (Assumption iii in Thm. 2). The corresponding theorem for the nonlinear case, with its proof given in Appx. A.5, is as follows:
Theorem 3. Given a nonlinear ICA model defined in Eqs. (1) and (2), where f the true mixing function. Consider f f G U G. where denotes an invertible Gaussianization! that maps the distribution to an standard isotropic (rotation cotation-invarion invariant) Gaussian, U denotes a rotation, and G-1 maps the distribution back to that before applying Ü G. If A'ssumptiinss i, ä and in of Thm. 2 are satisfied by replacing A with (s) and A with J Jy(s), then function h f-1 o f is composition of component-wise invertible transformation and a permutation with probability one.
Besides removing rotation indeterminacy, Thm. 3 also provides an insight for the full identifia- bility of nonlinear ICA, because all previous works assume that the nonlinear mixing function must be invertible (Hyvärinen and Morioka, 2016; Yang al., 2022) even with the help of auxiliary variables.
4 Identifiability with Independent Influences
In the previous section, we have presented theoretical results based on conditions of structural sparsity. Alternatively, we show in this section that sparsity conditions the support might be necessary in some specific scenarios, and that one can leverage the independence (in non non- -tat-sttittiaal sense) among the influences from different sources to the observations to achieve identifiability nonlinear ICA. This implies sparse interactions between the individual influencing processes and could be viewed as an analogous concept of "sparsity" w.r.t. influences. We show that without any kind of auxiliary vari- ables, the full identifiability of nonlinear ICA can also be provided given the condition of independent influences, thus complementing the results shown above from different view of "sparsity".
Nonlinear ICA with unconditional priors: Independent Influences. We now specify the setting of the required estimating process. We sample the estimated sources from multivariate Gaussian:
(s(S)
II
Z, exp (-0,,1&; -0,,87)
'One example is described in (Gresele et al., 2021), i.c., a composition of the element-wise CDFs of smooth factorised density and Gaussian, respectively.

---

1 where Z, > O is a constant. The sufficient statistics 0,,1 " and 8,,2 = 20 are assumed to be
linearly independent. We constraint the variances of to be distinct without loss of generality. Proposition 2. Let the observed data be sampled from a nonlinear ICA model as defined in Eqs. (1), (2), and (7). Suppose the following assumptions hold:
i. (Independent Inffuences): The inftuence of each source on the observed variables is inde- pendent of each other, i.e., each partial derivative /t dds is independent of the other sources and their influences in a non-statistical sense.
i.. The mixing function f and its inverse are twice differentiable.
ii. The Jacobian determinant of mixing function can be factorized as det((r(s)) = IT= (s;), where ys is a function that depends only on st. Note that volume-preserve transformation is special case when (1(8s) =
iv. During estimation, the columns of the Jacobian of the estimated unmixing function are regularized to be mutually orthogonal and with equal euclidean norm.
Then h = f-1 o f is composition of a component-wise invertible transformation and a permutaaion.
The proof is included in Appx. A.6. Note that some part of the proof technique is inspired by or based on Lemma in Yang et al. (2022). Assumption iv is a regularization during estimation and thus puts restriction on the true generating process. We achieve it by optimizing an objective function with a designed regularization term, of which the property is as follows (proof in Appx. A.7):
Proposition 3. The following inequality holds
1)
n
log
log [ct((] (x)) 2 0,
(8)
Dz,
1=1
with equality iff J, 1 (x) (x)) where (xx) is an orthogonal matrix and X(x) is a scalar:
Regarding the assumption of independent influences, to build intuition, one can consider a filmmaking process in safari park, of which the main characters are wild animals. The task of the boom operators here is to record the sound of animals. As there are many animals and microphones, the mixing (recording) process is highly dependent on the relative positions between them. Thus, the animal (source 8.), loosely speaking, influences the recording (mixing function f) through its position. The moving direction and speed of them can be loosely interpreted as the partial derivative w.r.t. the i-th source, i.e. 0r/Os: Assuming that the wild animals are not cooperative enough to fine-tune their positions and speeds for better recording and the safari park is not crowded, the influences of animals on the recording are generated independently by them in the sense that they are not affected by the others while moving. As result of that generating process, the column vectors of the Jacobian of the mixing function are uncorrelated with each other and the partial derivative w.r.t. to each source independent of other sources. Other practical scenarios include fields that adopt independent influences in process like orthogonal coordinate transformations (Gresele et al., 2021), such as dynamic control (Mistry et al., 2010) and structural geology (De Paor, 1983). is worth noting that this assumption of independent influences is different from the orthogonality condition proposed in (Gresele et al., 2021). For instance, if the sample mean is not zero, uncorrelatedness differs from orthogonality. And we argue that uncorrelatedness be more sensible as a condition on the influences. Besides, in (Gresele et al., 2021), orthogonality, together with conformal map and the others, proposed as conditions to rule out two specific types of spurious solutions (i.c., Darmois construction (Darmois, 1951) and Protated-Gaussinn MPAs (Locatello et al., 2019)) without identifiability result, although its empirical results are encouraging.
specific way to achieve independence might
Having said that, a violation of the assumption of independent influences could be ascribed to a deliberate global adjustment of sources. For example, cinema audio systems are carefully adjusted to achieve homogeneous sound effect on every audience. The position and orientation of each speaker are fine-tuned according to the others. In this case, it may be difficult to distinguish the influences from different speakers because of the fine-tuning. This may lead to a high degree of multicollinearity across the columns of Jacobian, thus violating the assumption of independent influences.
7

---

on Hatrax
Conrelation Marni
Comeeition Matri
0.3
Comeation Matrii
Rocivern Sources
0.6
0.6
0.6
622
0.4
6.1
0.2
Ground Sources
Sources
Sources
truhh
(a) SS.
(b) II.
(c) VP.
(d) Base.
Figure 3: Pearson correlation matrices between the ground-truth and the recovered sources.
Regarding Assumption ii in Prop. 2, we first note that mixing functions with factorial Jacobian determinants are much wider range as compared to component-wise transformations. To see this, straightforward example is the volume-preserving transformation, which has been widely adopted in generative models (Zhang et al., 2021). fact, all transformations with constant Jacobian determinant can be factorized w.r.t the sources, which, generally speaking, not component- wise. Besides, consistent with the empirical results in Yang et al. (2022), there exists other wowwume-preeerriigggggg transformations with factorial Jacobian determinant 2. Moreover, the volume- volume-preserving assumption has been demonstrated to be helpful to weaken the requirement auxiliary variables, specifically by reducing the number of required labels (Sorrenson et al., 2020; Yang al., 2022). Prop. 2 takes this further by imposing a weaker version, i.e., Assumption (iii), which, together with the other constraints, removes the need for any auxiliary variable.
5 Experiments
To validate the proposed theory of the identifiability of nonlinear ICA with unconditional priors, we conduct experiments based our main condition, i.e., structural sparsity (Thm. 1), well the condition of the independent influences (Prop. 2).
Setup. For the required regularization during estimation, consider regularized maximum- likelihood approach with the following objective: L(p-1;x) Ex p (x) - AR((f2 x)
where À is regularization parameter and R(f-1, x) is the regularization term. For Thm. L1 norm as an approximation of the Lo norm for efficiency (Donoho and Elad, 2003), therefore R(f-l,x == || (x)ll1; for Prop. 2. R(f- x) corresponds to LHS of Eq. (8). Following (Sorrenson al., 2020; Yang et al., 2022), we train a GIN (Sorrenson al., 2020) to maximize the the objective function xx).
Ablation study. We conduct an ablation study to verify the necessity of the proposed assumptions. Specifically, focus on the following models that correspond to different assumptions: (SS) The assumption of structural sparsity, as well as other assumptions in Thm. 1, are satisfied; (II) The assumption of independent influences, as well as other assumptions in Prop. 2, are satisfied; (VP) Compared to !1, only the assumption of independent influences is violated while the other assumptions (e.g., factorizable Jacobian determinant) are still satisfied; (Base) The vanilla bascline. Compared to VP, the (un)mixing function is not restricted to having factorizable Jacobian determinants. The data are generated according to the required assumptions. We also conduct comparison between the assumption of independent influences and orthogonality in (Gresele et al., 2021), which are presented in Appx. B together with experimental settings. Results for each model summarised in Fig. 4. For evaluation, between the true sources and the estimated ones (Hyvärinen and Morioka, 2016). One could observe that when the proposed assumptions MCC on average. This indicates that it is actually possible to identify sources from highly nonlinear mixtures up to trivial indeterminacies only based on restrictions on the mixing process. Moreover,
0.8
0.6
0.4
55
II
VP
Base
Model
Figure 4: Ablation study.
the mean correlation coefficient (MCC)
fully satisfied (SS //), model achieves the highest
A toy example: the Jacobian determinant mixing function w.r.t. where a, b o 0 are some constants.
G6192 a+b55
J22 b+o5y ) absi.

---

regarding Prop. 2, the importance of Assumption ii is supported by the higher performance of VP compared to that of Base. The visualization of the Pearson correlation matrices is shown Fig. 3.
Stability. To study the stability of the perfor- mance of identification w.r.t. different datasets varying the number of sources n. We test the model SS (Thm. 1) with different n. Visu- ally, we find that SS consistently outperforms Base (Fig. 5). Meanwhile, when the number of sources increases, one could observe that the MCC of Base decreases while that of SS stays stable. The stable empirical performance fur- ther validates our theoretical claims about the identifiability with structural sparsity.
Model
1.0
0.4
0.2
0.0
Number sources
Number sources
Figure 5: MCC w.r.t. different number of sources.
Image dataset. To study how reasonable the proposed theories are w.r.t. the practical generating process of observational data, we conduct experiments on the "Triangles" image dataset (Yang al., 2022). The process of generating this dataset mimics the process of drawing triangles by humans: i) First, we sample the elements needed for humans to draw monochrome triangle (i.e., rotation, width, height and grey level) from factorial multivariate Gaussian distribution. Different from (Yang al., 2022), we always sample from a single distribution order to guarantee that all priors are unconditionally independent; ii) Then, for each pixel, we decide whether it locates inside the triangle based the sampled elements and assign its gray level accordingly. Therefore, the process is similar to human drawing triangles. Even though each image is generated from these semantic elements, the true generating process and sources are still unknown (e.g.. a pixel could be (indirectly) influenced by multiple elements in complicated way). We apply GIN with sparsity regularization as the estimating method. The visualization of the identified sources (Fig. 6) indicates that our conditions may hold in practice.
Figure 6: Identification results on Triangles. Each row represents source identified by our model, with it varying from -2 to +2 SDs to illustrate its influence. The rightmost column given by the absolute pixel difference between -1 and +1 SDs. Visually, the four rows correspond to rotation, width, height and gray level, respectively.
heat map
IMA. Recently, Gresele et al. (2021) assume orthogonality, conformal map, and the others order to rule out Darmois MPAs. They formalize the
1.0
construction and orthogonality between columns of the Jacobian of the mixing function independent mechanism analysis (IMA) and show empirically that it improves the identification latent sources. To further explore their exciting results, generate datasets according to the generating process and regularization term de- scribed in (Gresele al., 2021). We sample the sources from the same distributions described in Appx. B and use GIN for training. From Fig. 7, one could observe that indeed IMA out- performs the baseline (Base) largely, which indicates that this condition is empirically helpful to the identifiability of nonlinear ICA. At the same time, MCC of I/ appears to be even higher than that of IMA, which might thanks to some additional constraints, such as the factorial volume. Meanwhile, it also supports the conjecture that IMA is a sensible condition for identification, and the identifiability based on may be achieved with some additional assumptions, such as conformal maps (Gresele et al., 2021).
0.8
0.6
0.0
Model
Figure 7: MCC w.r.t indepen- dent influences (II), independent mechanism analysis (IMA), and the baseline model (Base).
6 Discussion and Conclusion
Sparsity Assumptions. Sparsity assumptions have been widely used in various fields. For latent vari- able models, sparsity in the generating process plays an important role in the disentanglement or iden- tification of latent factors both empirically and theoretically (Bing et al., 2020; Rohe and Zeng, 2020; Moran et al., 1021; Rhodes and Lee, 2021; Lachapelle et al., 2022). In causality, various versions of

---

Occam's 's razor have been proposed to serve as fundamental assumptions for identifying the underlying causal structure (Spirtes al., 2000; Zhang, 2013; Raskutti and Uhler. 2018; Forster et al., 2020).
Formulated as a measure of the density of dependencies, sparsity assumptions are more likely to be held when the observations are actually influenced by the sources in a "simple" way. For example, in biology, analyses on ecological, gene-regulatory, metabolic, and other living systems find that active interactions may often be rather sparse (Busiello et al., 2017), even when these systems evolve with an unlimited number of complicated external stimuli. In physics, it is an important heuristic that a relatively small set of laws govern complicated observed phenomena. For instance, Einstein's 's theory of special relativity contains parsimonious relations between substances as an important heuristic to shave away the influence of ether compared to Lorentz's theory (Einstein, 1905; Nash, 1963).
However, sparsity is not an irrefutable principle and our assumption may fail in a number of situations. The most direct one could be scenario with heavily entangled relations between sources and observations. Let us consider the example of animal filmmaking in Sec. 4, where people are recording the sound of animals in safari park. If the filming location is restricted to narrow area of the safari park and multiple microphones are gathered together, the recording of each microphone will likely be influenced by almost all animals. In such a case, the dependencies between the recording of microphones and the animals are rather dense and our sparsity assumption is most likely not valid.
At the same time, even when the principle of simplicity holds, our formulation of sparsity, which is based on the sparse interactions between sources and observations, may still fail. One reason for this is the disparity between mechanism simplicity and structural sparsity. To illustrate this, one could consider the effect of sunlight on the shadow angles at the same location. In this case, the sun's rays and the shadow angles act the sources and the observations, respectively. Because rays of sunlight, loosely speaking, may be parallel to each other, the processes of them influencing the shadow angles may be almost identical. Thus, the influencing mechanism could be rather simple. On the other hand, each shadow angle is influenced by an unlimited number of the sun's 's rays, which indicates that the interactions between them may not be sparse, therefore violating our assumption. This sheds light on one of the limitations of our sparsity assumption, because the principle of simplicity could be formulated in several ways. Besides, these different formulations also suggest various possibilities for identifiability based on simplicity assumptions. Another proposed assumption, i.e., independent influ- ences, may be one of the alternative formulations, and more works remain to be explored in the future.
Conclusion. We provide identifiability results for nonlinear ICA with unconditional priors, which serve as one of the first steps to solve long-standing problem in unsupervised learning. In particular, we prove that the i.i.d. latent sources can be recovered up to a component-wise invertible transforma- tion and a permutation with only conditions on the nonlinear mixing process (e.g., structural sparsity). Therefore it stays closer to the original notion of ICA that is based on the marginal independence assumption of latent sources, while previous works rely on conditional independence on auxiliary variables as weak supervision or inductive bias. Besides, by removing rotation indeterminacy, structural sparsity benefits the identifiability of Gaussian ICA as well, which was also thought to be unsolvable before. Moreover, the results on the undercomplete case are of great practical interest and introduce insight for extending identifiable nonlinear ICA to general real-worle settings.
Our results on images illustrate the validity of the proposed conditions in practical data generating processes. In spite of this, it is possible that part of them is violated in several specific scenarios as discussed before. For example, the structural sparsity conditions do not apply to fully-connected structures, though the practical significance of such cases may be compromised by the lack of interpretability. As a complementary solution, we formulate the independent influences condition, which does not rely on the sparse structure of supports. While arguably natural in general, could still be violated due to a deliberate global adjustment of sources for homogeneity. We argue that this is inevitably a trade-off between introducing auxiliary variables and imposing restrictions on the mixing process to achieve the identifiability, whose practical use depends on the scenario and information available. Future work includes further generalizing and validating our theory.
Acknowledgements
We thank Lingjing Kong, Sébastien Lachapelle, and the anonymous reviewers for their constructive comments. This work was partially supported by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSF-Convergence Accelerator Track-D award #2134901, by grant from Apple Inc., and by a grant from KDDI Research Inc..
10

---

References
X. Bing, F. Bunea, Y. Ning, and M. Wegkamp. Adaptive estimation in structured factor models with applications overlapping clustering. The Annals of Statisties, 48(4):2055-2085 2020.
D. Busiello, Suweis, Hidalgo, and A. Maritan. Explorability and the origin of network sparsity in living systems. Scientific reports, 7(1):1-8, 2017.
Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
G. Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, page 231, 1951.
D. G. De Paor. Orthographic analysis of geological structures I. deformation theory. Journal of Structural Geology, 5(3-4):255-277 -27, 1983.
L. Dinh, J. Sohk-DDck,,ein, and S. Bengio. Density estimation using Real NVP. arXiv preprint arXiv:1605.08800 2016.
D. L. Donoho and M. Elad. Optimally sparse representation general (nonorthogonal) dictionaries via Lı minimization. Proceedings of the National Academy of Sciences, 100(5):2197-2202, 2003.
dictionnarii
A. Einstein. Does the inertia of a body depend upon its energy-content. Annalen der Physik, 18(13): 639-641, 1905.
H. Flanders. Liouville's theorem conformal mapping. Journal of Mathematics and Mechanics, 15 (1):157-161, 1966.
M. Forster, G. Raskutti, R. Stern, and N. Weinberger. The frugal inference of causal relations. The British Journal for the Philosophy of Science, 2020.
L. Gresele, J. von Kügelgen, V. Stimper, B. Schölkopf, and M. Besserve. Independent mechanism analysis, concept? Advances in Neural Information Processing Systems, 2021.
H. Hälvä and A. Hyvärinen. Hidden markov nonlinear ICA: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939-948. PMLR, 2020.
H. Hälvä, Le Corff, L. Lehéricy, So, Zhu, E. Gassiat, and A. Hyvärinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems, 34, 2021.
A. Hyvärimen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems, 29:3765-3773, 2016.
A. Hyvärinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In International Conference on Artificial Intelligence and Statisties, pages 460-469, PMLR, 2017.
A. Hyvärinen and E. Oja. fast fixed-point algorithm for independent component analysis. Neural computation, 9(7): 1483-149,, 1997.
A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999.
A. Hyvärinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference Artificial Intelligence and Statistics, pages 859-866 PMLR, 2019.
M. Joho, H. Mathis, and R. H. Lambert. Overdetermined blind source separation: Using more sensors than source signals in a noisy mixture. Proc. ICA, pages 81-86, 2000.
I. Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework, In International Conference on Artificial Intelligence and Statistics, pages 2207-2217, PMLR, 2020.

---

D. P. Kingma and P. Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Lachapelle, P. R. López, Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. Conference on Causal Learning and Reasoning, 2022.
Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14-4124. PMLR, 2019.
M. Mistry, J. Buchli, and S. Schaal. Inverse dynamics control of floating base systems using orthogonal decomposition. In 2010 IEEE international conference on robotics and automation, pages 3406-34000 1406-3412. IEEE, 2010.
G. E. Moran, D. Sridhar, Y. Wang. and D. M. Blei. Identifiable variational autoencoders via sparse decoding. arXiv preprint arXiv:2 /0. 10804, 2021.
L. Nash. The nature the natural sciences. 1963.
E. Oja. Unsupervised learning in neural computation. Theoretical computer science, 287(),177777,, 2002.
G. Raskutti and C. Uhler. Learning directed acyclic graph models based on sparsest permutations. Stat, 7(1)ze183 183, 2018.
T. Rhodes and D. Lee. Local disentanglement in variational auto-encoders using jacobian L_ regularization. Advances in Neural Information Processing Systems, 14:22708-22719, 2021.
K. Rohe and M. Zeng. Vintage factor analysis with varimax performs statistical inference. arXiv preprint arXiv:2004.05387, 2020.
P. Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ICA with general incompressible-flow networks (GIN). arXiv preprint arXiv:2001.04872, 2020.
P. Spirtes, C. N. Glymour, R. Scheines, and D. Heckerman. Causation, prediction, and search. MIT press, 2000.
A. Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing, 47(10):207-22220 1999.
L. N. Trefethen and D. Bau III. Numerical linear algebra, volume 50. Siam, 1997.
M. Willetts and B. Paige. I don't need u: Identifiable non-linear ICA without side information. arXiv preprint ar wxxiv22166.02.::5 2021.
X. Yang, Y. Wang. J. Sun, X. Zhang, S. Zhang, Z. Li, and J. Yan. Nonlinear ICA using volume- preserving transformations. International Conference on Learning Representations, 2022.
W. Yao, Y. Sun, A. Ho. C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:21 05.0228,, 2021.
J. Zhang, A comparison of three occam's razors for markovian causal models. The British journal for the philosophy of science, 64(2):423-444 2013.
K. Zhang, H. Peng. L. Chan, and A. Hyvärinen. ICA with sparse connections: Revisited. In International Conference Independent Component Analysis and Signal Separation, pages 195-202. Springer, 2009.
S. Zhang, C. Zhang, N. Kang, and Z. Li. iVPF: Numerical invertible volume preserving flow for efficient lossless compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 620 629, 2021.
12


Reviews:
[{"review_id": "skLa7x0sa0V", "reviewer": "Reviewer_ZHMG", "summary": "This paper considers the identifiability of the nonlinear ICA model with unconditional priors:\n$x = f(s)$\nwhere $x$ is the observed data and $s$ are the unobserved sources.\n\nThey prove that the sources $s$ are identifiable identifiable under two different settings:\n\n1. when a \"structural sparsity\" assumption holds;\n\n2. when an \"independent influence\" assumption holds.\n\nThey also provide experiments where they estimate the latent sources via regularized objective functions. The regularization terms are inspired by the two settings above.\n\nMore detail: \n\n1. Structural sparsity: this assumption is that for every latent source $s_i$, there is a set of observed variables such that $s_i$ is the only latent source that participates in the generation of all variables in the set. \n\nIn Theorem 1, the authors prove that under structural sparsity, the sources can be identified up to permutation and coordinate-wise transformation.  For this result, the authors also assume invertibility of $f$, that the observed Jacobian $J_f(s)$ spans the support of $J_f(s)$, and the estimated decoder is as sparse as the true decoder.\n\nIn Theorem 3, the authors again assume structural sparsity but relax the invertibility of $f$. They prove that if the estimated $\\widehat{f}$ is a 'rotated-Gaussian mixture preserving automorphism' (i.e. it has a particular functional form which rotates the true decoder $f$) then the sources are identifiable.\n\n\n2. Independent influence: this assumption is that the influence of each source on the observed variables is independent of each other i.e. $\\partial f / \\partial s_i$ is independent of the other sources and their influences in a non-statistical sense. \n\nIn Theorem 4, the authors prove that if independent influence holds (among other assumptions) then the latent sources are identifiable.  \n\n\n\n", "questions": "- The assumption of independent influence is that the influence of each source on the observed variables is independent of each other i.e. $\\partial f / \\partial s_i$ is independent of the other sources and their influences in a non-statistical sense.\n\nCan the authors be more precise about what \"in a non statistical sense\" means? From the paper, it seems that it means uncorrelatedness of the Jacobian columns? Can the authors clarify?", "limitations": "The authors do not elaborate on real life situations where their assumptions are likely to hold (except for the animal recording example in the independent influences section).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper is timely and relevant and seems like an important contribution to the study of identifiability\n\n- The writing is clear\n\nWeaknesses \n\n- Some definitions are unclear (see questions below).\n\n- The paper is missing a number of references relating to sparsity and identifiability in latent variable models. \n\nIn the linear case, Bing et al. (2020) and Rohe and Zeng (2020) prove that the assumption of sparsity in the mixing function (i.e. loadings matrix) gives identifiable solutions.\n\nIn the nonlinear case, Moran et al. (2021) prove that assuming sparsity in the decoder of a deep generative model gives identifiable solutions.\n\nSimilarly to the experiments in this paper, Rhodes and Lee (2021) propose a L1 regularization term on the Jacobian of the mixing function (without proving identifiability however). \n\n- Bing, Xin, et al. \"Adaptive estimation in structured factor models with applications to overlapping clustering.\" The Annals of Statistics 48.4 (2020): 2055-2081.\n\n- Rohe, Karl, and Muzhe Zeng. \"Vintage factor analysis with varimax performs statistical inference.\" arXiv preprint arXiv:2004.05387 (2020).\n\n- Moran, Gemma E., et al. \"Identifiable variational autoencoders via sparse decoding.\" arXiv preprint arXiv:2110.10804 (2021).\n\n- Rhodes, Travers, and Daniel Lee. \"Local Disentanglement in Variational Auto-Encoders Using Jacobian $ L_1 $ Regularization.\" Advances in Neural Information Processing Systems 34 (2021): 22708-22719.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PClA-HpXzo4", "reviewer": "Reviewer_TtGx", "summary": "This paper presents new identifiability result for nonlinear ICA that gets around the need for auxiliary information by leveraging a sparsity assumption on the mixing function. While sparsity has been explored in the nonlinear ICA community, the existing results have focused on the relationship between source distributions (Lachapelle et al., 2022), rather than on sparsity of the mixing function. On exception is Gresele et al [2021] that did show a different sparsity assumption rules out some of the standard counterexamples, but didn't provide a full identification proof. This paper does that and shows analogous results for the linear Gaussian case which well-known to not be identifiable without additional assumptions (usually non-Gaussian sources). The paper also gives results under an \"independent influences\" assumption which is more similar to Gresele et al [2021] and show identification results there too. Finally, the theory results are supported with some simple experiments.", "questions": "1. Do you know what was preventing MCC scores from getting above 0.9? These problems are relatively straightforward so I would have expected to see MCC score $\\rightarrow 1$; it would be good to understand what the limitations of the empirical results are?\n\nMinor\n* Condition (iii) of theorem 1 is hard to parse without a lot of back references. Consider rewording as follows, \"Denote by F the support of (all possible values of **the Jacobian of $f$**) $J_f (s)$. For all $i \\in {1, . . . , n}$,. there exist **|F_{i,:} | sources**...\". Similar amendments could be made to the other theorem statements too - the conditions in the theorems are already pretty dense, so it helps to remind the reader what all the variables refer to.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Strong identification (up to a component-wise transform & permutation) results in a hard problem. \n\nWeakness\n* I am always very uncomfortable with making assumptions on the mixing function $f: s \\rightarrow x$, especially in the image domain. While it is relatively intuitive that there should be some simple dependencies among the source variables (as people, we wouldn't be able to understand the world if this were not the case), it's not obvious that the mixing function should have any sparsity: consider how realistic images are rendered in modern rendering engines. The most realistic looking renderers use ray-tracing which explicitly bounce light around all the objects in the scene in order to correctly model how light reflects in a scene. That seems to imply a dense dependence between sources rather than any kind of sparsity. Of course, it may be the case that structured sparsity is a reasonable approximation of the mixing function, but I would have liked to see some more discussion on when we can expect this to hold. The discussion on page 7 with reference to the independent influences assumption was far better.\n* The empirical results have the expected relationships between approaches, but I was surprised that the MCC scores were only $\\approx 0.8$. \n\nI am conflicted on this paper - while ICA is not my research area, I know that this is a very well-studied area and identification in this IID setting is hard to achieve (as the paper points out - most papers rely on auxiliary variables). So if we take this work as a pure piece of theory, the structured sparsity condition is a sufficient condition for identification is a difficult domain, and this is very interesting paper. On the other hand - the practitioner in me can barely parse the structured sparsity assumption - so I'd have no idea whether or not it applied in my domain, so I think a lot more work is needed in expanding on when we expect this assumption to hold or alternatively, are there any testable implications of this assumption? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MERRuAam3oB", "reviewer": "Reviewer_3ycU", "summary": "The paper proves identifiability results for Independent Component Analysis. In contrast to other recent works which exploit assumptions on auxiliary variables, this work exploits sparsity assumptions on the mixing process.", "questions": "See above", "limitations": "See above", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Identifiably results for ICA is an important problem as it provides a rigorous foundation for other applied areas such as disentangled feature learning and parts of causality. As such, advances in this area are important and potentially impactful.\n\nThe paper is well written and clear. It explains the results in the context of the problem and prior work. I believe it makes a solid contribution and is worth accepting to the conference.\n\nOne thing I think could improve the paper would be to elaborate more on the assumtions being made and when they are violated. In particular:\n- what would happen if the ground truth data were to violate  the assumptions but the method were run anyway? \n- thinking about the equivalence classes of indeterminable solutions without this assumption, is it possible to relate these equivalence classes to the sparse solutions? E.g. does each equivalence class contain exactly one such sparse solution? Do only some contain one sparse solution (and if so, what is special about these equivalence classes)? \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept