Title: Generalizing Nonlinear ICA Beyond Structural Sparsity
Venue: NeurIPS 2023
Accepted: yes
Keywords: Latent variable models

nonlinear independent component analysis
Abstract: Nonlinear independent component analysis (ICA) aims to uncover the true latent sources from their observable nonlinear mixtures. Despite its significance, the identifiability of nonlinear ICA is known to be impossible without additional assumptions. Recent advances have proposed conditions on the connective structure from sources to observed variables, known as Structural Sparsity, to achieve identifiability in an unsupervised manner. However, the sparsity constraint may not hold universally for all sources in practice. Furthermore, the assumptions of bijectivity of the mixing process and independence among all sources, which arise from the setting of ICA, may also be violated in many real-world scenarios. To address these limitations and generalize nonlinear ICA, we propose a set of new identifiability results in the general settings of undercompleteness, partial sparsity and source dependence, and flexible grouping structures. Specifically, we prove identifiability when there are more observed variables than sources (undercomplete), and when certain sparsity and/or source independence assumptions are not met for some changing sources. Moreover, we show that even in cases with flexible grouping structures (e.g., part of the sources can be divided into irreducible independent groups with various sizes), appropriate identifiability results can also be established. Theoretical claims are supported empirically on both synthetic and real-world datasets.

Generalizing Nonlinear ICA Beyond Structural Sparsity

Abstract

Nonlinear independent component analysis (ICA) aims to uncover the true latent sources from their observable nonlinear mixtures. Despite its significance, the identifiability of nonlinear ICA is known to be impossible without additional assumptions. Recent advances have proposed conditions on the connective structure from sources to observed variables, known as Structural Sparsity , to achieve identifiability in an unsupervised manner. However, the sparsity constraint may not hold universally for all sources in practice. Furthermore, the assumptions of bijectivity of the mixing process and independence among all sources, which arise from the setting of ICA, may also be violated in many real-world scenarios. To address these limitations and generalize nonlinear ICA, we propose a set of new identifiability results in the general settings of undercompleteness, partial sparsity and source dependence, and flexible grouping structures. Specifically, we prove identifiability when there are more observed variables than sources (undercomplete), and when certain sparsity and/or source independence assumptions are not met for some changing sources. Moreover, we show that even in cases with flexible grouping structures (e.g., part of the sources can be divided into irreducible independent groups with various sizes), appropriate identifiability results can also be established. Theoretical claims are supported empirically on both synthetic and real-world datasets.

1 Introduction

The unveiling of the true generating process of observations is fundamental to scientific discovery. Nonlinear independent component analysis (ICA) provides a statistical framework that represents a set of observed variables x as a nonlinear mixture of independent latent sources s , i.e., x = f ( s ) . Unlike linear ICA (Comon, 1994), the mixing function f can be an unknown nonlinear function, thus generalizing the theory to more real-world tasks. However, the identifiability of nonlinear ICA has been a long-standing problem for decades. The main obstacle is that, without additional assumptions, there exist infinite spurious solutions returning independent variables that are mixtures of the true sources (Hyvärinen and Pajunen, 1999). In the context of machine learning, this makes the theoretical analysis of unsupervised learning of disentangled representations difficult (Locatello et al., 2019).

To overcome this challenge, recent work has introduced the auxiliary variable u , and assumed that all sources are conditionally independent given u . Most of these methods require auxiliary variables to be observable, such as class labels and domain indices (Hyvärinen and Morioka, 2016; Hyvärinen et al., 2019; Khemakhem et al., 2020a; Sorrenson et al., 2020; Lachapelle et al., 2022; Lachapelle and Lacoste-Julien, 2022), with the exceptions being those for time series (Hyvärinen and Morioka, 2017; Hälvä et al., 2021; Yao et al., 2021, 2022). While the use of the auxiliary variable u allows for the identifiability of nonlinear ICA with mild restrictions on the mixing process, it also necessitates a large number of distinct values of u , which can be difficult to obtain in tasks with insufficient side informa-

tion. Moreover, since these results assume that all sources are dependent on u , they cannot accommodate a subset of sources with invariant distributions (e.g., content may not change with different styles).

Another possible direction is to impose appropriate conditions on the mixing process, but limited results are available in the literature. For example, it has been shown that conformal maps are identifiable up to specific indeterminacies (Hyvärinen and Pajunen, 1999; Buchholz et al., 2022). Moreover, Taleb and Jutten (1999) identify the latent sources when the mixing process is a component-wise nonlinear function added to a linear mixture. These methods do not rely on conditional independence given the auxiliary variable and thus achieve the identifiability in a fully unsupervised setting. At the same time, the requirement of above-mentioned classes of the mixing function, such as conformal maps and post-nonlinear models, restricts the applicability of the results in another way. For instance, according to Liouville's theorem (Monge, 1850), conformal maps in Euclidean spaces of dimensions higher than two are Möbius transformations, which appear to be overly restrictive for most data-generating processes. As an alternative, Zheng et al. (2022) prove that, under the assumption of Structural Sparsity , the true sources can be identified up to trivial indeterminacies. Since the proposed condition is on the connective structure from sources to observed variables, i.e., the support of the Jacobian matrix of the mixing function, it does not require the mixing function to be of any specific algebraic form. Thus, Structural Sparsity may serve as one of the first general principles for the identifiability of nonlinear ICA in a fully unsupervised setting.

While being a potential solution to the identifiability of nonlinear ICA without side information, the assumption of Structural Sparsity has its limitations from a pragmatic viewpoint. The most obvious one arises from the fact that it may fail in a number of situations where the generating processes are heavily entangled. Although the principle of simplicity may be a general rule in nature, it is intuitively possible that Structural Sparsity does not apply to at least a subset of sources, such as one or a few speakers in a crowded room. Unfortunately, Zheng et al. (2022) require Structural Sparsity to hold for all sources in order to provide any identifiability guarantee. Therefore, it would be desirable in practice to provide weaker notions of identifiability, such as the ability to identify a subset of sources to a trivial degree of uncertainty, in cases of partial sparsity.

In addition to partial sparsity, identifiability with Structural Sparsity also fails with the undercompleteness (more observed variables than sources) and/or partial source dependence (potential dependence among some hidden sources). These limitations are not unique to the sparsity assumption, but rather a result of the traditional setting of ICA, where the numbers of the sources and observed variables must be equal and dependencies among sources are not allowed. However, both situations are quite common in practice. One may easily have millions of pixels (observed variables) but only dozens of hidden concepts (sources) in a picture, constituting an undercomplete case that cannot be handled by previous results. Meanwhile, dependencies among some variables are also prevalent in tasks such as computational biology (Cardoso, 1998; Theis, 2006). The alternative assumption of conditional independence given auxiliary variables may still be overly restrictive if applied universally to all sources. For the identifiability of nonlinear ICA to truly benefit scientific discovery in a wider range of scenarios, these methodological limitations should be properly addressed.

Aiming to generalize nonlinear ICA with Structural Sparsity , we first present a set of new identifiability results to address these fundamental challenges of undercompleteness, partial sparsity, and source dependence. We show that, under the assumption of Structural Sparsity and without auxiliary variables, latent sources can be identified from their nonlinear mixtures up to a component-wise invertible transformation and a permutation, even when there are more observed variables than sources (Thm. 3.1). Moreover, if the assumption of sparsity and/or source independence does not hold for some changing sources, we provide partial identifiability results, showing that the remaining sources can still be identified up to the same trivial indeterminacy (Thm. 4.1, Thm. 4.2). Furthermore, in the cases with flexible grouping structures (e.g., part of the sources can be grouped into irreducible independent subgroupings with various sizes, such as mixtures of signals with various dimensions), certain types of identifiability are also guaranteed with auxiliary variables (Thm. 4.3, Thm. 4.4). Therefore, we establish, to the best of our knowledge, one of the first general frameworks for uncovering latent variables with appropriate identifiability guarantees in a principled manner. The theoretical claims are validated empirically through our experiments and many previous works involving disentanglement.

2 Preliminaries

The data-generating process of nonlinear ICA is as follows:

where s = ( s 1 , . . . , s n ) ∈ S ⊆ R n is a latent vector representing the independent sources, and x = ( x 1 , . . . , x m ) ∈ X ⊆ R m denotes the observed random vector. The mixing function f is assumed to be smooth in the sense that its second-order derivatives exist. The primary objective of ICA is to establish identifiable models, i.e., the sources s are identifiable (recoverable) up to certain indeterminacies by learning an estimated mixing function ˆ f : ˆ S → X with assumptions identical to the generating process (Comon, 1994). Different from most ICA results where m = n and f : S → X must be linear, we allow m>n (i.e., undercompleteness) and f to be a general nonlinear function, therefore extending the previous setting. Thus, we relax the previous assumption on the invertibility of f , only necessitating it to be injective and its Jacobian to be of full column rank. Furthermore, we denote p s i as the marginal probability density function (PDF) of the i -th source s i and p s as the joint PDF of the random vector s . Moreover, we introduce some additional technical notations as follows:

Definition 2.1. Given a subset A ⊆ { 1 , . . . , n } , the subspace R n A is defined as

where z i is the i -th element of the vector z .

That is, R n A denotes the subspace of R n specified by an index set A . Furthermore, we define the support of a matrix as follows:

Definition 2.2. The support of a matrix M ∈ R m × n is defined as

̸

With a slight abuse of notation, we reuse supp( · ) to denote the support of a matrix-valued function:

Definition 2.3. The support of a function M : Θ → R m × n is defined as

̸

For brevity, we denote F and ˆ F as the support of the Jacobian J f ( s ) and J ˆ f (ˆ s ) , respectively. Additionally, T refers to a set of matrices with the same support of T ( s ) in J ˆ f (ˆ s ) = J f ( s ) T ( s ) , where T ( s ) is a matrix-valued function. Throughout this work, for any matrix M , we use M i, : to denote its i -th row, and M : ,j to denote its j -th column. For any set of indices B ⊂ { 1 , . . . , m } × { 1 , . . . , n } , analogously, we have B i, : := { j | ( i, j ) ∈ B} and B : ,j := { i | ( i, j ) ∈ B} .

3 Identifiability with undercompleteness

We first present the result on removing one of the major assumptions in ICA, i.e., the number of observed variables m must be equal to that of hidden sources n . We prove that, in the undercomplete case ( m>n ), sources can be identified up to a trivial indeterminacy under Structural Sparsity .

Theorem 3.1. Let the observed data be a large enough sample generated by an undercomplete nonlinear ICA model as defined in Eqs. (1) and (2) . Suppose the following assumptions hold:

Then s is identifiable up to an element-wise invertible transformation and a permutation.

The proof is included in Appx. A.1, of which part of the conditions and techniques are based on (Zheng et al., 2022). It is noteworthy that, same as previous work, we also need to add a sparsity

regularization on the learned Jacobian during the estimation so that | ˆ F| ≤ |F| , which is required for all sparsity-based identifications throughout the paper and we only emphasize here for brevity.

Assumption i avoids some pathological conditions (e.g., samples are from very limited subpopulations that only span a degenerate subspace) and is typically satisfied asymptotically. The first part implies that there are at least |F i, : | observed samples spanning the support space, which is almost always satisfied asymptotically. The second part is also relatively mild. Note that T refers to a set of matrices with the same support of T ( s ) in J ˆ f (ˆ s ) = J f ( s ) T ( s ) and J ˆ f (ˆ s ) i, : ∈ R n ˆ F i, : . Since we only necessitate the existence of one matrix T ∈ T in the entire space, even in rare cases where these two matrices do not share the same non-zero coordinates due to non-generic canceling between specific values of elements, there is almost always an existence of a matrix T ∈ T fulfilling the assumption.

Assumption ii, i.e., Structural Sparsity , originates from (Zheng et al., 2022). Intriguingly, compared to the original bijective setting considered by (Zheng et al., 2022), this assumption is much more likely to be satisfied in the undercomplete case. The key reason is that it only necessitates the existence of a subset of observed variables whose intersection uniquely identifies the target source variable. For instance, regarding s 1 in Fig. 1, there exist x 1 and x 4 s.t. the intersection of their parents is only s 1 . In principle, the size of this set can be quite small (e.g., one or two). Hence, it is very likely to be satisfied when there is a sufficient number of observed

variables (e.g., millions of pixels for images), which has also been verified empirically in our experiments (e.g., Fig. 4 in Sec. 5). Additionally, in some tasks, we might even construct or select observations in a data-centric manner to satisfy this assumption. Without the previous constraint of bijectivity, structural sparsity can truly be applied in a much broader range of practical scenarios.

By proving the identifiability in the undercomplete case, we remove the previous assumption of bijectivity on the mixing function f and thus generalizing the theory to more application scenarios. It is worth noting that, while some work has provided results without assuming bijectivity (Khemakhem et al., 2020a), they rely on extra information from many distinct values of auxiliary variables. Differently, we do not need any auxiliary variables and follow a fully unsupervised setting; Zheng et al. (2022); Kivva et al. (2022) explore the undercompleteness without any auxiliary variable. However, Zheng et al. (2022) only remove the rotational indeterminacy in the nonlinear case and Kivva et al. (2022) assume Gaussian mixture priors, while we provide the full identifiability result without distributional assumptions. At the same time, as elaborated above, the assumption of Structural Sparsity has been significantly weakened in the undercomplete case considered by our theorem. Moreover, identifiability with undercompleteness is also essential if assumptions are partially violated w.r.t. a subset of sources, of which the intuition is verified by theoretical results introduced in the following sections.

4 Identifiability with partial sparsity and source dependence

Under the condition of Structural Sparsity , we show the identifiability of undercomplete ICA with general nonlinear functions (Thm. 3.1). While this removes the restriction of bijectivity between sources and observed variables, it remains uncertain as to whether Structural Sparsity holds for all sources in a universal way. At the same time, even in the scenarios that Structural Sparsity may not be universally satisfied for all sources, it is still valuable to consider its potential to hold true for a subset of sources. This type of partial sparsity may often be the case in practical scenarios, as illustrated by our experiments (e.g., Fig. 5 in Sec. 5). However, the corresponding partial identifiability, i.e., the theoretical guarantee for the identification of the subset of sources satisfying Structural Sparsity , is not achieved by (Zheng et al., 2022). In fact, as long as one or a few sources do not meet the assumption of Structural Sparsity , the previous work is unable to provide any identifiability guarantees.

Furthermore, in addition to the universal sparsity, the statistical independence between sources is another fundamental assumption. This assumption arises from the original setting of ICA and has been adopted in most related works. However, in many real-world scenarios, requiring all sources to be mutually independent might be impractical, and there are likely to be a subset of sources that

are dependent in some way. For example, the frequency and duration of smoking, as well as the type of tobacco products used, are all interrelated factors that contribute to the development of lung cancer. Without any identifiability guarantees in the case where there exist any dependent sources, it is quite restrictive for nonlinear ICA, and even its undercomplete extension, to successfully deal with real problems in practice. Therefore, similar to the partial sparsity case, it is highly desirable that alternative theoretical results, i.e., identifiability for independent sources, can be guaranteed even with the existence of dependent sources.

To deal with these remaining challenges in the considered undercomplete case, we further relax other assumptions with additional information. To start with, we provide an identifiability result when Structural Sparsity holds true for only a subset of sources, thus alleviating the obstacle of partial sparsity. Moreover, we relax the mutual independence assumption and allow for some changing sources to be dependent on each other. To this end, we partition the sources into two parts s = [ s I , s D ] , where variables in s I are mutually independent, but those in s D do not need to be. Let s I and s D correspond to variables in s with indices { 1 , . . . , n I } and { n I +1 , . . . , n } , respectively. That is, s I = ( s 1 , . . . , s n I ) ∈ S I ⊆ R n I and s D = ( s n I +1 , . . . , s n ) ∈ S D ⊆ R n D . We denote the i -th scalar element in a vector, say s , as s i . For sources in s D , they do not need to be mutually independent as long as they are dependent on a variable u , i.e.,

It is noteworthy that we allow arbitrary relations between sources in s D . These sources might be grouped into several subspaces or actually be mutually independent, but we do not need to obtain this information as prior knowledge. This is essential since the exact dependence structures, or even the number of dependent variables, are usually unknown in practice. By keeping this type of uncertainty for both partial sparsity and/or partial source dependence, one can be more confident in applying the theoretical advancements of nonlinear ICA in various tasks. We prove the identifiability for this arguably more flexible scenario as the following theorems:

Theorem 4.1. Let the observed data be a large enough sample generated by an undercomplete nonlinear ICA model defined in Eqs. (2) and (3) . Suppose the following assumptions hold:

There exist n D +1 distinct values of u , i.e., u j with j ∈ { 0 , 1 , . . . , n D } , s.t. the n D vectors w ( s D , u , i ) with i ∈ { n I +1 , ..., n } are linearly independent, where vector w ( s D , u , i ) is defined as follows:

There exist u 1 , u 2 ∈ u , s.t., for any set A s ⊆ S with non-zero probability measure and cannot be expressed as B s I × s D for any B s I ⊂ S I , we have

̸

Then s D is identifiable up to an subspace-wise invertible transformation.

Thm. 4.1 ensures the subspace-wise identifiability of s D , i.e., the estimated subspace ˆ s D contains all and only information from s D . This implies that we can disentangle and extract the invariant part of the latents, beneficial for tasks like domain adaptation where recovering each individual source might not be necessary as long as the subspace that is invariant across domains can be disentangled. Furthermore, we also prove the component-wise identifiability as follows:

Theorem 4.2. In addition to assumptions in Thm. 4.1, suppose the following assumptions hold:

For each i ∈ { 1 , . . . , n I } , there exist { s ( ℓ ) } |F i, : n I | ℓ =1 and a matrix T ∈ T s.t. span { J f ( s ( ℓ ) ) i, : n I } |F i, : n I | ℓ =1 = R n I F i, : n I and [ J f ( s ( ℓ ) )T ] i, : n I ∈ R n I ˆ F i, : n I .

(Structural Sparsity) For all k ∈ { 1 , . . . , n I } , there exists C k s.t. ⋂ i ∈C k F i, : n I = { k } .

Then s I is identifiable up to an element-wise invertible transformation and a permutation.

The proofs are presented in Appx. A.2 and Appx. A.3. We tackle the challenge of partial sparsity and dependence by necessitating Structural Sparsity and independence only on a subset of sources in s I and prove that these sources can be identified up to trivial indeterminacies. For the remaining sources in s D , they only need to be dependent on an auxiliary variable u without necessitating conditional independence among sources or distributional assumption. This extends previous models that assume all sources to be conditionally independent given u (Hyvärinen et al., 2019; Lachapelle et al., 2022) or require the conditional distribution of the sources to be of a specific form (Khemakhem et al., 2020b).

The assumption on p ( s D | u ) in Thm. 4.1 indicates that the auxiliary variable u should have a sufficiently diverse impact on sources without independence assumption (i.e., s D ). It follows a similar spirit to the standard assumption of variability (Hyvärinen et al., 2019) but we further relax it. Specifically, we only need n D +1 values of u for the identifiability of sources in s I . This is intuitively reasonable since the fewer changes (smaller n D ) a system has, the easier (fewer required values, i.e., n D +1 ) that a larger part of it (larger n I , i.e., n -n D ) is identifiable. In contrast, most previous works require all sources to be dependent on an auxiliary variable u with 2 n +1 distinct values of u : no identifiability for any subset of sources can be provided if there exists any degree of violations, either on the number of sources dependent on u or the number of values of u . This limits the application of these results to ideal scenarios where all sources are influenced by the same auxiliary variable with sufficient changes without any type of compromise. In practice, however, it is often the case that only a subset of sources benefit from the additional information provided by auxiliary variables, different auxiliary variables may affect different sources, or auxiliary variables do not contain sufficient information. Assumption ii in Thm. 4.1 is originally from (Kong et al., 2022) and also necessitate the presence of change. Intuitively, the chance of having a subset A s on which all domain distributions have an equal probability measure is very slim, which has been verified empirically in (Kong et al., 2022). For both theorems, we consider the more challenging undercomplete case, for which the related identifiability results are lacking in the literature. Additionally, unlike previous works assuming specific distributions of sources such as exponential families, we do not have similar distributional assumptions on the sources.

4.1 Results with flexible grouping structures

If we further have access to the dependence structure among variables in s D , additional identifiability results for these sources may also be established. For example, consider the setting that s D = ( s n I +1 , . . . , s n ) can be decomposed to d irreducible independent subspaces { s c 1 , . . . , s c d } , of which each is a multi-dimensional vector consisting multiple sources. We denote the j -th consecutive d -dimensional vector ( j -th subspace) in s as s c j = ( s ( j -1) d +1 , . . . , s jd ) = ( s c j ( l ) , . . . , s c j ( h ) ) , where s c j ( l ) and s c j ( h ) are the first and the last sources in s c j , respectively. Then we have

This is similar to Independent Subspace Analysis (ISA) (Hyvärinen and Hoyer, 2000; Theis, 2006) but we allow only a subset of sources as a composition of (conditionally) independent subspaces instead of all, which formalizes the tasks of blind source separation or uncovering latent variable models with mixtures of both high-dimensional and one-dimensional signals. The considered general setting essentially covers ICA and ISA as special cases: if n I = n , it is consistent with the ICA problem; if n I = 0 , all sources can be decomposed into irreducible independent subspaces, and thus it becomes an ISA problem. The identifiability result under this setting is shown in the following theorem with its proof provided in Appx. A.4:

Theorem 4.3. Let the observed data be a large enough sample generated from an undercomplete nonlinear ICA model as defined in Eqs. (2) and (4) . Suppose the following assumptions hold:

There exist 2 n D + 1 values of u , i.e., u i with i ∈ { 0 , 1 , . . . , 2 n D } , s.t. the 2 n D vectors w ( s D , u i ) -w ( s D , u 0 ) with i ∈ { 1 , . . . , 2 n D } are linearly independent, where vector w ( s D , u i ) is defined as follows:

where where

There exist u 1 , u 2 ∈ u , s.t., for any set A s ⊆ S with nonzero probability measure and cannot be expressed as B s I ×S D for any B s I ⊂ S I , we have

̸

(Structural Sparsity) For all k ∈ { 1 , . . . , n I } , there exists C k s.t. ⋂ i ∈C k F i, : n I = { k } .

Then s I is identifiable up to an element-wise invertible transformation and a permutation, and s D is identifiable up to a subspace-wise invertible transformation and a subspace-wise permutation.

All assumptions align with the same principles as those elaborated in the theorems proposed above and have been adapted to cater to the flexible grouping structure. Specifically, in addition to the identifiability of sources in s I , we prove that we can also identify sources in s D up to an indeterminacy that, for each c i ∈ { c 1 , . . . , c d } , there exists an invertible transformation h c i s.t. h c i ( s c i ) = ˆ s c i , which is analogous to the previous element-wise indeterminacy. Consequently, even when dealing with mixtures of high and one-dimensional sources, like in the case of multi-modal data, we can still recover the hidden generating process to some extent. Based on the aforementioned theoretical results, which consider undercompleteness, partial sparsity, and partial source dependence, Thm. 4.3 further generalizes the identifiability of nonlinear ICA by relaxing the dimensionality constraint of the latent generating factors.

In this vein, it is natural to consider another dependence structure, i.e., sources in s D are not marginally but conditionally independent given an auxiliary variable u . This is similar to the assumption made in most previous works on identifiable nonlinear ICA with surrogate information, which assume that all sources are conditionally independent of each other given the auxiliary variable. However, our setting is more flexible in the sense that we do not assume all sources to be influenced by the auxiliary variable. Specifically, sources in s I are mutually independent as in the original ICA setting, while only sources in s D have access to the side information from the conditional independence given u , i.e.,

The identifiability result for all sources ( s I and s D ) is as follows with proof in Appx. A.5:

Theorem 4.4. Let the observed data be a large enough sample generated from an undercomplete nonlinear ICA model as defined in Eqs. (2) and (5) , suppose the following assumptions hold:

For each i ∈ { 1 , . . . , n I } , there exist { s ( ℓ ) } |F i, : n I | ℓ =1 and a matrix T ∈ T s.t. span { J f ( s ( ℓ ) ) i, : n I } |F i, : n I | ℓ =1 = R n I F i, : n I and [ J f ( s ( ℓ ) )T ] i, : n I ∈ R n I ˆ F i, : n I .

There exist 2 n D + 1 values of u , i.e., u i with i ∈ { 0 , 1 , . . . , 2 n D } , s.t. the 2 n D vectors w ( s D , u i ) -w ( s D , u 0 ) with i ∈ { 1 , . . . , 2 n D } are linearly independent, where vector w ( s D , u ) is defined as follows:

There exist u 1 , u 2 ∈ u , s.t., for any set A s ⊆ S with nonzero probability measure and cannot be expressed as B s I ×S D for any B s I ⊂ S I , we have

̸

iv. (Structural Sparsity) For all k ∈ { 1 , . . . , n I } , there exists C k s.t. ⋂ i ∈C k F i, : n I = { k } .

Then s is identifiable up to an element-wise invertible transformation and a permutation.

With different assumptions on different sets of sources, one could view this theorem as an expansion of both previous theoretical findings that impose distributional constraints on sources with auxiliary variables (e.g., (Hyvärinen et al., 2019)) and those that constrain the mixing function with Structural Sparsity (Zheng et al., 2022). This is particularly helpful in the context of self-supervised learning (Von Kügelgen et al., 2021) or transfer learning (Kong et al., 2022), where latent representations are modeled as a changing part and an invariant part. In (Kong et al., 2022), the component-wise identifiability for variables changing across domains (i.e., s D with multiple values of u in our setting) are provided but not those in the invariant part (i.e., s I in our setting). With the help of Thm. 4.4, we can show identifiability up to an element-wise invertible transformation and a permutation for each source, regardless of whether it changes across domains or not, which may help some related tasks where full identifiability is necessary. Furthermore, for causal reasoning or disentanglement with observational time-series data, our theorem benefits the identifiability of temporal processes involving instantaneous relations. Previous works in that area can only deal with time-delayed/changing influences, as they rely on the global conditional independence of all sources given the changing time index as the auxiliary variable (Hyvärinen and Morioka, 2016, 2017; Yao et al., 2022). Our theorem, on the other hand, provides the added ability to identify unconditional sources, thanks to the partially satisfied sparsity assumption, and thus aids the uncovering of latent processes with instantaneous relations.

5 Experiments

In order to validate the proposed identifibaility results, we conduct experiments using both simulated data and real-world images. It is noteworthy that there has been extensive research that has empirically verified that deep latent variable models are likely to be identifiable in complex scenarios, particularly in the disentanglement task (Kumar et al., 2017; Klys et al., 2018; Locatello et al., 2018; Rubenstein et al., 2018; Chen et al., 2018; Burgess et al., 2018; Duan et al., 2020; Falck et al., 2021; Carbonneau et al., 2022). While we are not sure of the exact inductive biases or side information that are available during the real-world application, which has been proved to be necessary (Hyvärinen and Pajunen, 1999; Locatello et al., 2019), the empirical success of these methods sheds light on the possibility of identification in the general settings considered in this work.

Setup. For settings with the auxiliary variable u , u is always available during estimation and we consider the dataset as D = {( x (1) , u (1) ) , . . . , ( x ( N ) , u ( N ) )} , where N is the sample size and u ( i ) is the value of u (or class label) corresponding to the data point x ( i ) . Given the estimated model ˆ f parameterized by θ , similar to (Sorrenson et al., 2020), we consider a regularized maximum-likelihood approach for the required sparsity regularization during estimation with the objective function as: L ( θ ) = E ( x , u ) ∈D [ log p ˆ f -1 ( x | u ) -λ R ] , where λ ∈ [0 , 1] is a regularization parameter and R is the regularization term on the Jacobian of the estimated mixing function, i.e., J ˆ f . Based on our experimental results (Fig. 8 in Appx. B.2), we adopt the minimax concave penalty (MCP) (Zhang, 2010) as the regularization term. For settings without the auxiliary variable, we remove the access of u and follow the same objective function in (Zheng et al., 2022). We train a General Incompressible-flow Network (GIN) (Sorrenson et al., 2020), which is a flow-based generative model, to maximize the objective function L ( · ) . Following (Sorrenson et al., 2020), where necessary, we concatenate the latent sources with independent Gaussian noises to meet the dimensionality requirements. All results are from 20 trials with random seeds. Additional details of the experimental setup are included in Appx. B.

Ablation study. We perform an ablation study to verify the necessity of the proposed assumptions. Specifically, we focus on the following models corresponding to different assumptions: (UCSS) The assumption of Structural Sparsity, as well as other assumptions in the undercomplete case (Thm. 3.1), are satisfied; (Mixed) The assumption of Structural Sparsity with undercompleteness and the required dependence structure among sources influenced by the auxiliary variable, as well as other assumptions in the partial sparsity and dependence case (Thm. 4.4), are satisfied; (Base) The vanilla baseline in the undercomplete case, where the assumption of Structural Sparsity is not satisfied compared to UCSS . The datasets are generated according to the required assumptions, the details of which are included in Appx. B. All experiments are conducted in the undercomplete case, where the number of observed variables is twice the number of sources. For datasets that contain both sources in s I and s D ( Mixed case), we set half as s I and the other half as s D , and the minimum required

3

4

5

6

Number of sources

0.2

0.4

0.6

0.8

1.0

MCC

Model = UCSS

3

4

5

6

Number of sources

Model = Base

1

2

3

4

5

m

/

n

0%

20%

40%

60%

80%

100%

Percentage

n

3

4

5

6

4

6

8

10

Number of sources

0.2

0.4

0.6

0.8

1.0

MCC

Model = Mixed

4

6

8

10

Number of sources

Model = Base

3

4

5

6

Number of sources

20%

40%

60%

80%

100%

Percentage

numbers of required distinct values have been assigned to the auxiliary variable u . Following previous works (Hyvärinen and Morioka, 2016; Lachapelle et al., 2022), we use the mean correlation coefficient (MCC) between the true sources and the estimated ones as the evaluation metric.

Results for each model are summarised in Fig. 2 and Fig. 3. It can be observed that when the proposed assumptions are met ( UCSS and Mixed ), our models achieve higher MCCs than Base . This indicates that it is indeed possible to identify sources from nonlinear mixtures up to trivial indeterminacy in the general settings with undercompleteness, partial sparsity, and partial source dependence. Additionally, we conduct experiments with different numbers of sources n to evaluate the stability of the identification. Our results show that both models consistently outperform Base across all values of n , further supporting the theoretical claims.

Undercomplete Structural Sparsity. As previously discussed, the assumption of Structural Sparsity (Assumption ii in Thm. 3.1) is far more plausible in an undercomplete setting considered in our theory (i.e., the number of observed variables m is larger than the number of sources n ) than in the more restrictive bijective scenario required in (Zheng et al., 2022) (i.e., the numbers are equal, m = n ). Consequently, extending the identifiability with structural sparsity from a bijective to an undercomplete setting significantly broadens its applicability in real-world contexts. In order to validate the necessity of the proposed generalization empirically, we construct several experiments studying the Structural Sparsity assumption in the undercomplete case. We consider different numbers of sources n with different degrees of undercompleteness ( m/n , where m is the number of observed variables). For each setting, we generate 50 random matrices where each entry is independently determined with an equal probability to be either zero or non-zero. The results of the percentages of matrices satisfying the assumption of Structural Sparsity are presented in Fig. 4. We could observe that there exists a significant gap on the percentages between the cases where m/n = 1 , i.e., the bijective setting, and the undercomplete settings where m/n > 1 . Thus, it is clear that the assumption is much more likely to hold true when we have more observed variables than sources. Furthermore, when the degree of undercompleteness increases, the percentage of cases satisfying structural converges to 1. This further suggests that the assumption will almost always hold with a sufficient degree of undercompleness, which is rather common in practice. For instance, a photo can easily have millions of pixels (observed variables) but only a dozen of hidden concepts (sources).

Partial Structural Sparsity. Moreover, as previously noted, it is not uncommon for Structural Sparsity to be violated for a subset of sources. For instance, certain sources (such as high-decibel sound sources) may exert influence over all observed variables (microphones). Nonetheless, the

000000000

O

oooooooo0

000000000

000000000

prior study (Zheng et al., 2022) necessitates that the sparsity assumption holds true for all sources, providing no identifiability assurance in cases of any degree of violation. To confront this practical obstacle, we propose Thm. 4.1 and Thm. 4.2 to demonstrate that the remaining sources (i.e., n I sources) can still be identified even when the sparsity assumption does not universally hold for some sources. These results can also be motivated empirically. For instance, from Fig. 4, one may find that Structural Sparsity does not likely to hold for all sources when m/n = 1 , which is the bijective setting considered in (Zheng et al., 2022). However, as discussed above, if a subset of sources satisfies the assumption, at least the identifiability for these sources could be guaranteed by our proposed theorems (Thm. 4.1 and Thm. 4.2) under certain conditions. To illustrate this, we conduct experiments in the bijective setting ( m/n = 1 ) and report the percentage of sources satisfying Structural Sparsity in Fig. 4. We consider datasets with different number of sources and generate 50 random matrices for each of these. Each entry is independently determined with an equal probability to be either zero or non-zero. Combining results from both Fig. 4 and Fig. 5, we observe that, even in scenarios where Structural Sparsity is rarely satisfied for all sources ( m/n = 1 in Fig. 4), it is almost always satisfied for a significant fraction of sources (Fig. 5). Consequently, our generalization also proves helpful even within the confines of the earlier bijective setting.

Image datasets. To study how reasonable the proposed theories are w.r.t. the practical generating process of observational data in complex scenarios, we conduct experiments on "Triangles" (Yang et al., 2022) and EMNIST (Cohen et al., 2017) datasets. The "Triangles" dataset consists of 60 , 000 synthetic 28 × 28 images of triangles, which are generated from 4 factors: rotation, height, width, and brightness. By fixing the number of pixels (observed variables) and generating factors (sources), we can guarantee that the images are generated according to an undercomplete process, although the exact generating process is still unknown (e.g., a pixel could be (indirectly) influenced by multiple factors in a complicated way). For the real-world dataset, EMNIST contains 240 , 000 28 × 28 images of handwritten digits and is a larger version of the classical MNIST dataset. Although we do not know the exact number of sources, it is highly possible that it is smaller than the number of pixels ( 784 ). We present the identified sources with the top four standard deviations (SDs) from both datasets in Fig. 6 and Fig. 7. In both figures, each row represents a source identified by our model, with it varying from -4 to +4 SDs to illustrate its influence. The rightmost column is a heat map given by the absolute pixel difference between -1 and +1 SDs. By observing the identified sources with the top four standard deviations, one could find that it is possible to identify semantically meaningful attributes from practical image datasets, which further suggests the potential of our theory in real-world scenarios. Additional results are available in Appx. B.2.

6 Conclusion

We establish a set of new identifiability results of nonlinear ICA in general settings with undercompleteness, partial sparsity and source dependence, and flexible grouping structures, thereby extending the identifiabilty theory to a wide range of real-world scenarios. Specifically, we prove the identifiability when there are more observed variables than underlying sources, and when sparsity and/or independence are not met for a subset of sources. Moreover, by leveraging various dependence structures among sources, further identifiability guarantees can also be obtained. Theoretical results have been validated through a combination of extensive previous studies and our own experiments, which involve both synthetic and real-world datasets. Future work includes adopting the theoretical framework for related tasks, such as disentanglement, transfer learning, and causal discovery. Furthermore, the proposed identifiability guarantees on generalized latent variable models bolster our confidence in uncovering hidden truths across diverse real-world settings in scientific discovery. We have only explored the visual disentanglement task, and the lack of other applications is a limitation of this work.

References

Ardizzone, T. Bungert, F. Draxler, U. Köthe, J. Kruse, R. Schmier, and P. Sorrenson. Framework for Easily Invertible Architectures (FrEIA), 2018-2022. URL https://github.com/vislearn/ FrEIA .

Breheny and J. Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The Annals of Applied Statistics , 5(1):232-253, 2011.

Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. arXiv preprint arXiv:2208.06406 , 2022.

P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Understanding disentangling in beta-V AE. Workshop on Learning Disentangled Representations at the 31st Conference on Neural Information Processing Systems , 2018.

M.-A. Carbonneau, J. Zaidi, J. Boilard, and G. Gagnon. Measuring disentanglement: A review of metrics. IEEE Transactions on Neural Networks and Learning Systems , 2022.

J.-F. Cardoso. Multidimensional independent component analysis. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP'98 (Cat. No. 98CH36181) , volume 4, pages 1941-1944. IEEE, 1998.

T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems , 31, 2018.

Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN) , pages 2921-2926. IEEE, 2017.

Comon. Independent component analysis, a new concept? Signal processing , 36(3):287-314, 1994.

Duan, L. Matthey, A. Saraiva, N. Watters, C. Burgess, A. Lerchner, and I. Higgins. Unsupervised model selection for variational disentangled representation learning. In ICLR , 2020.

Falck, H. Zhang, M. Willetts, G. Nicholson, C. Yau, and C. C. Holmes. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Systems , 34:8676-8690, 2021.

Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association , 96(456):1348-1360, 2001.

Hälvä, S. Le Corff, L. Lehéricy, J. So, Y. Zhu, E. Gassiat, and A. Hyvärinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems , 34, 2021.

Hyvärinen and P. Hoyer. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation , 12(7):1705-1720, 2000.

Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems , 29:3765-3773, 2016.

Hyvärinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In International Conference on Artificial Intelligence and Statistics , pages 460-469. PMLR, 2017.

Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks , 12(3):429-439, 1999.

Hyvärinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics , pages 859-868. PMLR, 2019.

Khemakhem, D. Kingma, R. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020a.

Khemakhem, R. Monti, D. Kingma, and A. Hyvarinen. Ice-beem: Identifiable conditional energybased deep models based on nonlinear ica. Advances in Neural Information Processing Systems , 33:12768-12778, 2020b.

P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1 x 1 convolutions. Advances in neural information processing systems , 31, 2018.

Kivva, G. Rajendran, P. Ravikumar, and B. Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems , 35:1568715701, 2022.

Klys, J. Snell, and R. Zemel. Learning latent subspaces in variational autoencoders. Advances in neural information processing systems , 31, 2018.

Kong, S. Xie, W. Yao, Y. Zheng, G. Chen, P. Stojanov, V. Akinwande, and K. Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning , pages 11455-11472. PMLR, 2022.

Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848 , 2017.

Lachapelle and S. Lacoste-Julien. Partial disentanglement via mechanism sparsity. In UAI 2022 Workshop on Causal Representation Learning , 2022.

Lachapelle, P. R. López, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. Conference on Causal Learning and Reasoning , 2022.

Locatello, D. Vincent, I. Tolstikhin, G. Rätsch, S. Gelly, and B. Schölkopf. Competitive training of mixtures of independent deep generative models. arXiv preprint arXiv:1804.11130 , 2018.

Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning , pages 4114-4124. PMLR, 2019.

P.-L. Loh and M. J. Wainwright. Support recovery without incoherence: A case for nonconvex regularization. The Annals of Statistics , 45(6):2455-2482, 2017.

Monge. Applications de l'analyse á la géométrie, 1850.

Ravikumar, G. Raskutti, M. J. Wainwright, and B. Yu. Model selection in Gaussian graphical models: High-dimensional consistency of ℓ 1 -regularized MLE. In Advances in Neural Information Processing Systems , 2008.

Ravikumar, M. J. Wainwright, G. Raskutti, and B. Yu. High-dimensional covariance estimation by minimizing ℓ 1 -penalized log-determinant divergence. Electronic Journal of Statistics , 5:935-980, 2011.

Rubenstein, B. Schölkopf, and I. Tolstikhin. Learning disentangled representations with wasserstein auto-encoders. In 6th International Conference on Learning Representations (ICLR 2018) , 2018.

Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ICA with general incompressible-flow networks (GIN). arXiv preprint arXiv:2001.04872 , 2020.

Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on signal Processing , 47(10):2807-2820, 1999.

Theis. Towards a general independent subspace analysis. Advances in Neural Information Processing Systems , 19, 2006.

Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems , 34:16451-16467, 2021.

J. Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ 1 -constrained quadratic programming (Lasso). IEEE Transactions on Information Theory , 55(5): 2183-2202, 2009.

Yang, Y. Wang, J. Sun, X. Zhang, S. Zhang, Z. Li, and J. Yan. Nonlinear ICA using volumepreserving transformations. In International Conference on Learning Representations , 2022.

Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2110.05428 , 2021.

Yao, G. Chen, and K. Zhang. Temporally disentangled representation learning. In Advances in Neural Information Processing Systems , 2022.

C.-H. Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics , 38(2):894-942, 2010.

Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems , 2022.

Appendix

Table of Contents

A Proofs

A.1 Proof of Theorem 3.1

Theorem 3.1. Let the observed data be a large enough sample generated by an undercomplete nonlinear ICA model as defined in Eqs. (1) and (2) . Suppose the following assumptions hold:

(Structural Sparsity) For each k ∈ { 1 , . . . , n } , there exists C k s.t. ⋂ i ∈C k F i, : = { k } .

Then s is identifiable up to an element-wise invertible transformation and a permutation.

Proof. Let h : s → ˆ s denotes the transformation between the true sources and estimated sources. We can apply the chain rule repeatedly to get:

Since J ˆ f (ˆ s ) and J f ( s ) both possess full column rank, J h ( s ) should have a non-zero determinant. From this, we can deduce by incorporating the inverse of h :

Our objective here is to demonstrate that the function h is a composition of a permutation and a component-wise invertible transformation. Let D ( s ) denote a diagonal matrix and P denote a permutation matrix, our goal can be rewritten as demonstrating that J h ( s ) -1 = D ( s ) P . This leads us to demonstrate that:

where T ( s ) ∈ R n × n is a square matrix. Here, we define F as the support of J f ( s ) , ˆ F as the support of J ˆ f (ˆ s ) and T as a set of matrices with the same support of T ( s ) . Furthermore, T ∈ T is a matrix with the same support as T ( s ) . Based on Assumption i, we have:

Further, we can express:

Given that the set { J f ( s ( ℓ ) ) i, : } |F i, : | ℓ =1 forms a basis of R n F i, : , we can express any vector in this space as a linear combination of these basis vectors. In particular, for any j 0 ∈ F i, : , the one-hot vector e j 0 ∈ R n F i, : can be written as

where α ℓ denotes the respective coefficient.

With this in mind, we can find the transformation of e j 0 under T as

According to Assumption i, each term in the above summation belongs to the space R n ˆ F i, : . Therefore, T j 0 , : itself resides in R n ˆ F i, : , i.e., T j 0 , : ∈ R n ˆ F i, : . Thus

Then the connections between these supports can be established according to Defn. 2.3

It is noteworthy that a similar strategy to derive Eq. 14 has been applied in (Zheng et al., 2022) and part of the proof technique is inspired by that work. In contrast to the proof by Zheng et al. (2022), which assumes the invertibility of f , we only necessitate its injectivity. This distinction allows for the inclusion of undercomplete cases.

Since J f ( s ( ℓ ) ) and J ˆ f (ˆ s ( ℓ ) ) have full column rank n , T ( s ( ℓ ) ) must have a non-zero determinant. Otherwise, it would follow that the rank of T ( s ( ℓ ) ) is less than n , which would imply a contradiction that J ˆ f (ˆ s ( ℓ ) ) = J f ( s ( ℓ ) ) T ( s ( ℓ ) ) has a column rank less than n . Representing the determinant of the matrix T ( s ( ℓ ) ) as its Leibniz formula yields

̸

where S n is the set of n -permutations. Thus, there is at least one term in the sum that is non-zero, i.e.,

̸

̸

Then we can conclude that this σ is in the support of T ( s ) since s ( ℓ ) ∈ s . Therefore, it follows that

which is equivalent to

Together with Eq. (14), we have

Denote

Then we have

Because of the sparsity regularization on the estimated Jacobian, we further have

Combining this with Eq. (21), we derive

̸

Suppose T ( s ) = D ( s ) P , then

̸

̸

there must exists i 3 ∈ C j 1 such that

Notice that σ ( j 3 ) ∈ T j 1 , : ∩ T j 2 , : implies

Then by Eqs. (28) and (29), we have

̸

Additionally, consider j 3 ∈ { 1 , . . . , n } for which

Since j 1 = j 2 , we can assume j 3 = j 1 without loss of generality. A similar strategy has been used previously in (Lachapelle et al., 2022; Zheng et al., 2022). Based on Assumption ii, there exists C j 1 ∋ j 1 such that ⋂ i ∈C j 1 F i, : = { j 1 } . Because

Since j 1 ∈ F i 3 , : , it follows that ( i 3 , j 1 ) ∈ F . Therefore, according to Eq. (14), we have

̸

This further implies ( i 3 , j 3 ) ∈ F by Eq. (20) and (23), which contradicts Eq. (27). Therefore, we have proven by contradiction that T ( s ) = D ( s ) P . By replacing T ( s ) with D ( s ) P in Eq. (9), we obtain Eq. (8), which is the goal.

A.2 Proof of Theorem 4.1

Theorem 4.1. Let the observed data be a large enough sample generated by an undercomplete nonlinear ICA model defined in Eqs. (2) and (3) . Suppose the following assumptions hold:

There exist n D +1 distinct values of u , i.e., u j with j ∈ { 0 , 1 , . . . , n D } , s.t. the n D vectors w ( s D , u , i ) with i ∈ { n I +1 , ..., n } are linearly independent, where vector w ( s D , u , i ) is defined as follows:

There exist u 1 , u 2 ∈ u , s.t., for any set A s ⊆ S with non-zero probability measure and cannot be expressed as B s I × s D for any B s I ⊂ S I , we have

̸

Then s D is identifiable up to an subspace-wise invertible transformation.

Proof. Let h : s → ˆ s denotes the transformation between the true and estimated sources. By using chain rule repeatedly, we have

Since J ˆ f (ˆ s ) and J f ( s ) both possess full column rank, J h (ˆ s ) should have a non-zero determinant. Thus, J h ( s ) must be invertible and have a non-zero determinant. Otherwise, one of them would not be of full column rank, which leads to a contradiction.

Applying the change of variable rule, we have

Taking the logarithm on both sides yields

Note that according to the model defined in Eqs. (2) and (3), the joint densities can be factorized as

Then we define the difference across domains as follows

By taking the difference on both sides of Eq. (33) corresponding to u j and u 0 , we obtain

where, for i ∈ { 1 , . . . , n I } , log p ˆ s | u (ˆ s i | u j ) and log p s | u ( s i | u j ) have been canceled because sources in s I are not dependent on u j . That is, q ( s i , u j ) = 0 if i ∈ { 1 , . . . , n I } .

Taking the derivatives of both sides of Eq. (36) w.r.t. ˆ s k where k ∈ { 1 , . . . , n I } , we have

Clearly, LHS of Eq. (37) equals zero. By considering each j ∈ { 1 , . . . , n D } for u j , we have n D equations like Eq. (37), which constitute a linear system with a n D × n D coefficient matrix.

According to the assumption, the coefficient matrix of the linear system has full rank. Thus, the only solution of Eq. (37) is ∂s i ∂ ˆ s k = 0 for i ∈ { n I +1 , . . . , n } and k ∈ { 1 , . . . , n I } .

As h -1 ( · ) is smooth, its Jacobian can be written as:

Since ∂ s j ∂ ˆ s k = 0 for j ∈ { n I +1 , . . . , n } and k ∈ { 1 , . . . , n I } , entries in the submatrix C must all be zero. Thus, the submatrix D must be invertible, otherwise J h -1 (ˆ s ) will not be invertible, which is a contradiction. Besides, based on Assumption ii, one can show that all entries in the submatrix B are zero according to part of the proof of Theorem 4.2 in (Kong et al., 2022) (Steps 1, 2, and 3). Therefore, ˆ s D is an invertible transformation of s D .

A.3 Proof of Theorem 4.2

Theorem 4.2. In addition to assumptions in Thm. 4.1, suppose the following assumptions hold:

(Structural Sparsity) For all k ∈ { 1 , . . . , n I } , there exists C k s.t. ⋂ i ∈C k F i, : n I = { k } .

Then s I is identifiable up to an element-wise invertible transformation and a permutation.

Proof. Our goal here is to show that ˆ s I is a composition of a permutation and a component-wise invertible transformation of sources in s I . By using chain rule repeatedly, we have

As h -1 ( · ) is smooth, its Jacobian can be written as:

In the proof of Thm. 4.1, we have shown that all entries in the submatrix C are zero. Then we have

where Eq. ( ⋆ ) is directly from the result that all entries in C , i.e., those in J h -1 (ˆ s ) n I +1: , : n I , are zero. Moreover, in the proof of Thm. 4.1, we have also shown that all entries in the submatrix B are zero. Let D ( s ) represent a diagonal matrix and P represent a permutation matrix. Thus, our goal is equivalent to show that J h ( s ) : n I , : n I = P I D I ( s ) or J h -1 (ˆ s ) : n I , : n I = J h -1 (ˆ s ) : n I , : n I = J h ( s ) -1 : n I , : n I = D I ( s ) -1 P I -1 . Then we need to prove that

where T ( s ) ∈ R n I × n I is a square matrix. Note that we have denoted F as the support of J f ( s ) , ˆ F as the support of J ˆ f (ˆ s ) and T as the support of T ( s ) . Besides, we have also denoted T as a matrix with the same support of T . According to Assumption i, we have

Since { J f ( s ( ℓ ) ) i, : n I } |F i, : n I | ℓ =1 forms a basis of R n I F i, : n I , for any j 0 ∈ F i, : n I , we are able to rewrite the one-hot vector e j 0 ∈ R n I F i, : n I as

where α ℓ is the corresponding coefficient. Then

According to Assumption i, each term in the above summation belongs to the space R n I ˆ F i, : n I . Therefore, T j 0 , : itself resides in R n I ˆ F i, : n I , i.e., T j 0 , : ∈ R n I ˆ F i, : n I . Thus

Then the connections between these supports can be established according to Defn. 2.3

Since J f ( s ( ℓ ) ) : , : n I and J ˆ f (ˆ s ( ℓ ) ) : , : n I have full column rank n I , T ( s ( ℓ ) ) must have a non-zero determinant. Otherwise, it would follow that the rank of T ( s ( ℓ ) ) is less than n I , which would imply a contradiction that J ˆ f (ˆ s ( ℓ ) ) : , : n I = J f ( s ( ℓ ) ) : , : n I T ( s ( ℓ ) ) has a column rank less than n I .

The determinant of the matrix T ( s ( ℓ ) ) can be represented as its Leibniz formula as

̸

Additionally, we have

where S n I is the set of n I -permutations. Therefore, there is at least one term in the sum that is non-zero, i.e.,

̸

̸

Then we can conclude that this σ must present in the support of T ( s ) since s ( ℓ ) ∈ s . Therefore, it follows that

which is equivalent to

Together with Eq. (48), we have

Because of the sparsity regularization on the estimated Jacobian, we further have

Combined with Eq. (55), we have

̸

Suppose T ( s ) = D I ( s ) -1 P I -1 , then

̸

Additionally, consider j 3 ∈ { 1 , . . . , n I } for which

Since j 1 = j 2 , we can assume j 3 = j 1 without loss of generality. Based on Assumption ii, there exists C j 1 ∋ j 1 such that ⋂ i ∈C j 1 F i, : n I = { j 1 } . Because

Since j 1 ∈ F i 3 , : n I , it follows that ( i 3 , j 1 ) ∈ F : , : n I . Therefore, according to Eq. (48), we have

Notice that σ ( j 3 ) ∈ T j 1 , : ∩ T j 2 , : implies

Then by Eqs. (62) and (63), we have

̸

Denote

Then we have

̸

̸

there must exists i 3 ∈ C j 1 such that

This further implies ( i 3 , j 3 ) ∈ F : , : n I by Eqs. (54) and (57), which contradicts Eq. (61). Thus, we have proven by contradiction that T ( s ) = D I ( s ) -1 P I -1 . By replacing T ( s ) with D I ( s ) -1 P I -1 in Eq. (43), we obtain Eq. (42), which is the goal.

A.4 Proof of Theorem 4.3

Theorem 4.3. Let the observed data be a large enough sample generated from an undercomplete nonlinear ICA model as defined in Eqs. (2) and (4) . Suppose the following assumptions hold:

There exist 2 n D + 1 values of u , i.e., u i with i ∈ { 0 , 1 , . . . , 2 n D } , s.t. the 2 n D vectors w ( s D , u i ) -w ( s D , u 0 ) with i ∈ { 1 , . . . , 2 n D } are linearly independent, where vector w ( s D , u i ) is defined as follows:

where

There exist u 1 , u 2 ∈ u , s.t., for any set A s ⊆ S with nonzero probability measure and cannot be expressed as B s I ×S D for any B s I ⊂ S I , we have

̸

(Structural Sparsity) For all k ∈ { 1 , . . . , n I } , there exists C k s.t. ⋂ i ∈C k F i, : n I = { k } .

Then s I is identifiable up to an element-wise invertible transformation and a permutation, and s D is identifiable up to a subspace-wise invertible transformation and a subspace-wise permutation.

Proof. Let h : s → ˆ s denotes the transformation between the true and estimated sources. By using chain rule repeatedly, we have

Because J ˆ f (ˆ s ) and J f ( s ) have full column rank, J h ( s ) must be invertible and have a non-zero determinant. Otherwise, one of them would not be of full column rank, which leads to a contradiction.

Applying the change of variable rule, we have

Taking the logarithm on both sides yields

Note that according to the model defined in Eqs. (2) and (3), the joint densities can be factorized as

Together with Eq. (67), we have

Therefore, for u = u 0 , . . . , u 2 n D , we have 2 n D + 1 such equations. Subtracting each equation corresponding to u 1 , . . . , u 2 n D with the equation corresponding to u 0 results in 2 n D equations:

Then we take the derivatives of both sides of Eq. (70) w.r.t. ˆ s k and ˆ s v where k, v ∈ { 1 , . . . , n } and k = v . Besides, if both k > n I and v > n I , then they are not indices of the same subspace. It is clear that the RHS of Eq. (70) equals to zero. For the i -th term of the summation on the LHS, we have the following equation after taking the derivatives:

̸

where i l and i h corresponds to the minimum and maximum indices of elements in s i = ( s i l , . . . , s i h ) . By iterating i from c 1 to c d , we are also iterating l from n I +1 to n . Thus by considering all those equations as well as iterating j in u j from 0 to 2 n D , we have a linear system with a 2 n D × 2 n D coefficient matrix.

Then, according to Assumption ii, the coefficient matrix of the linear system has full rank. Thus, the only solution of Eq. (71) is ∂s l ∂ ˆ s k ∂s l ∂ ˆ s v = 0 and ∂ 2 s l ∂ ˆ s k ∂ ˆ s v = 0 .

̸

Note that k = v and, if both k > n I and v > n I , they are not indices of sources in the same subspace. Besides, ∂s l ∂ ˆ s k ∂s l ∂ ˆ s v = 0 indicates that it is impossible for both ∂s l ∂ ˆ s k and ∂s l ∂ ˆ s v to be non-zero. Furthermore, because h is invertible, they cannot be both zero, indicating that it is either ∂s l ∂ ˆ s k = 0 or ∂s l ∂ ˆ s v = 0 . Then, because s I has invariant distribution w.r.t. u , if ∂s l ∂ ˆ s k = 0 (i.e., ∂s l ∂ ˆ s v = 0 ), then ˆ k / ∈ { 1 , . . . , n I } . Otherwise, s l will also be invariant, which is a contradiction. Therefore, ˆ k can only be the index of an estimated source from one independent subspace, which, together with the invertibility, leads to the conclusion that s D is a composition of an invertible subspace-wise transformation and a subspace-wise permutation of ˆ s D . So it is the mapping from ˆ s D to s D since the subspace-wise transformation is invertible and the inverse of a block-wise permutation matrix is still a block-wise invertible matrix.

̸

Now we have shown the identifiability result for s D = ( s n I +1 , . . . , s n ) , then we need to show that for the remaining sources s I = ( s 1 , . . . , s n I ) .

By using chain rule repeatedly, we have

Since we have shown that, for every l ∈ { n I +1 , . . . , n } and k ∈ { 1 , . . . , n I } , ∂s l ∂ ˆ s k = 0 , all entries of J h -1 (ˆ s ) n I +1: n, : n I must be zero.

Then we have

where Eq. ( ⋆ ) is directly from the result that all entries in J h -1 ( s ) n I +1: , : n I are zero.

Based on the proof of Theorem 4.2 in Kong et al. (2022) and Assumption iii, particularly, steps 1, 2, and 3, one can show that ˆ s I does not depend on s D . Let D ( s ) represents a diagonal matrix and P

represent a permutation matrix. Thus, our goal is equivalent to show that J h ( s ) : n I , : n I = P I D I ( s ) or J h -1 (ˆ s ) : n I , : n I = J h -1 (ˆ s ) : n I , : n I = J h ( s ) -1 : n I , : n I = D I ( s ) -1 P I -1 . Then we need to prove that

where T ( s ) ∈ R n I × n I is a square matrix. Note that we have denoted F as the support of J f ( s ) , ˆ F as the support of J ˆ f (ˆ s ) and T as the support of T ( s ) . Besides, we have also denoted T as a matrix with the same support of T . According to Assumption i, we have

Since { J f ( s ( ℓ ) ) i, : n I } |F i, : n I | ℓ =1 forms a basis of R n I F i, : n I , for any j 0 ∈ F i, : n I , we are able to rewrite the one-hot vector e j 0 ∈ R n I F i, : n I as

where α ℓ is the corresponding coefficient. Then

where the final ' ∈ ' follows from Assumption i that each element in the summation belongs to R n I ˆ F i, : n I . Thus

Then the connections between these supports can be established according to Defn. 2.3

Since J f ( s ( ℓ ) ) : , : n I and J ˆ f (ˆ s ( ℓ ) ) : , : n I have full column rank n I , T ( s ( ℓ ) ) must have a non-zero determinant. Otherwise, it would follow that the rank of T ( s ( ℓ ) ) is less than n I , which would imply a contradiction that J ˆ f (ˆ s ( ℓ ) ) : , : n I = J f ( s ( ℓ ) ) : , : n I T ( s ( ℓ ) ) has a column rank less than n I .

The determinant of the matrix T ( s ( ℓ ) ) can be represented as its Leibniz formula as

̸

where S n I is the set of n I -permutations. Thus, there is at least one non-zero term in the sum, i.e.,

̸

̸

Then we can see that this σ must present in the support of T ( s ) since s ( ℓ ) ∈ s . It follows that

which is equivalent to

Additionally, we have

Together with Eq. (80), we have

Denote

Then we have

Because of the sparsity regularization on the estimated Jacobian, we further have

Together with Eq. (87), we have

Suppose T ( s ) = D I ( s ) -1 P I -1 , then

̸

Additionally, consider j 3 ∈ { 1 , . . . , n I } for which

Since j 1 = j 2 , we can assume j 3 = j 1 without loss of generality. Based on Assumption iv, there exists C j 1 ∋ j 1 such that ⋂ i ∈C j 1 F i, : n I = { j 1 } . Because

Since j 1 ∈ F i 3 , : n I , we have ( i 3 , j 1 ) ∈ F : , : n I . Therefore, according to Eq. (80), we have

Notice that σ ( j 3 ) ∈ T j 1 , : ∩ T j 2 , : implies

Then by Eqs. (94) and (95), we have

̸

̸

there must exists i 3 ∈ C j 1 such that

̸

where

̸

which implies ( i 3 , j 3 ) ∈ F : , : n I by Eqs. (86) and (89), therefore contradicting Eq. (93). Thus, we have proven by contradiction that T (ˆ s ) = D I ( s ) -1 P I -1 . By replacing T ( s ) with D I ( s ) -1 P I -1 in Eq. (75), we obtain Eq. (74), which is the goal.

Therefore, s I is identifiable up to a composition of a component-wise invertible transformation and a permutation, and s D is identifiable up to a composition of a subspace-wise invertible transformation and a subspace-wise permutation.

A.5 Proof of Theorem 4.4

Theorem 4.4. Let the observed data be a large enough sample generated from an undercomplete nonlinear ICA model as defined in Eqs. (2) and (5) , suppose the following assumptions hold:

For each i ∈ { 1 , . . . , n I } , there exist { s ( ℓ ) } |F i, : n I | ℓ =1 and a matrix T ∈ T s.t. span { J f ( s ( ℓ ) ) i, : n I } |F i, : n I | ℓ =1 = R n I F i, : n I and [ J f ( s ( ℓ ) )T ] i, : n I ∈ R n I ˆ F i, : n I .

There exist 2 n D + 1 values of u , i.e., u i with i ∈ { 0 , 1 , . . . , 2 n D } , s.t. the 2 n D vectors w ( s D , u i ) -w ( s D , u 0 ) with i ∈ { 1 , . . . , 2 n D } are linearly independent, where vector w ( s D , u ) is defined as follows:

There exist u 1 , u 2 ∈ u , s.t., for any set A s ⊆ S with nonzero probability measure and cannot be expressed as B s I ×S D for any B s I ⊂ S I , we have

̸

Then s is identifiable up to an element-wise invertible transformation and a permutation.

Proof. Let h : s → ˆ s denotes the transformation between the true and estimated sources. By using chain rule repeatedly, we have

Because J ˆ f (ˆ s ) and J f ( s ) have full column rank, J h ( s ) must be invertible and have a non-zero determinant. Otherwise, one of them would not be of full column rank, which leads to a contradiction.

Applying the change of variable rule, we have

Taking the logarithm on both sides yields

Note that according to the model defined in Eqs. (2) and (3), the joint densities can be factorized as

Together with Eq. (99), we have

̸

Then we have

Therefore, for u = u 0 , . . . , u 2 n D , we have 2 n D + 1 such equations. Subtracting each equation corresponding to u 1 , . . . , u 2 n D with the equation corresponding to u 0 results in 2 n D equations:

Then we take the derivatives of both sides of Eq. (101) w.r.t. ˆ s k and ˆ s v where k, v ∈ { 1 , . . . , n } and k = q . For brevity, we first define the following terms:

where j = 1 , . . . 2 n D . Note that for i ∈ { 1 , . . . , n I } , s i does not depend on u . Thus, we have η ′′ i ( z i , u j ) = η ′′ i ( z i , u j ′ ) and η ′ i ( z i , u j ) = η ′ i ( z i , u j ′ ) , ∀ j, j ′ . Hence only sources dependent on u (i.e., s D ) remain in Eq. (107).

By considering each j ∈ { 1 , . . . , 2 n D } for u j , we have 2 n D equations like Eq. (107), which constitute a linear system with a 2 n D × 2 n D coefficient matrix.

̸

According to Assumption ii, the coefficient matrix of the linear system has full rank. Thus, the only solution of Eq. (107) is h ′ i, ( k ) h ′ i, ( q ) = 0 and h ′′ i, ( k,v ) = 0 for i = n c + 1 , . . . , n and k, v ∈ { 1 , . . . , n } , k = v .

As h -1 ( · ) is smooth, its Jacobian can be written as:

̸

Because h ′ i, ( k ) h ′ i, ( v ) = 0 , k, v ∈ { 1 , . . . , n } , k = v , for each i = n I +1 , . . . , n , there is at most one index r ∈ { 1 , . . . , n } s.t. h ′ i, ( r ) = 0 . Therefore, there is at most one non-zero entry in each row indexed by i = n D +1 , . . . , n in the Jacobian matrix J h -1 (ˆ s ) . Further, the invertibility of h -1 ( · ) necessitates J h -1 to be full-rank which implies that there is exactly one non-zero component in each row of sub-matrices C and D .

̸

Suppose that non-zero component lies in the sub-matrix C , then s D is not dependent on u . Thus, the only non-zero component must lie in D . Because J h is of full-rank and C is a zero sub-matrix, D must have full rank. Hence, ˆ s D must be a composition of a component-wise invertible transformation and a permutation of s D . Moreover, according to part of the proof of Theorem 4.2 in Kong et al. (2022) (Steps 1, 2, and 3), the submatrix B is zero if Assumption iii holds.

Now we have shown that, for the matrix J h -1 , its submatrix D is a generalized permutation matrix and both submatrices B and C are zero-matrices). Because h -1 is smooth and invertible, the sub-matrices of the corresponding positions of J h have the same properties. Then, we need to show the identifiability for the remaining sources s I = ( s 1 , . . . , s n I ) , i.e., ˆ s I is a permutation with component-wise invertible transformation of s I .

Let D ( s ) represents a diagonal matrix and P represent a permutation matrix. Thus, our goal is equivalent to show that J h ( s ) : n I , : n I = P I D I ( s ) or J h -1 (ˆ s ) : n I , : n I = J h -1 (ˆ s ) : n I , : n I = J h ( s ) -1 : n I , : n I = D I ( s ) -1 P I -1 . By using chain rule repeatedly, we have

where Eq. ( ⋆ ) is directly from the result that all entries in C , i.e., those in J h -1 (ˆ s ) n I +1: , : n I , are zero. Therefore our goal is equivalent to show that

where T ( s ) ∈ R n I × n I is a square matrix. Note that we have denoted F as the support of J f ( s ) , ˆ F as the support of J ˆ f (ˆ s ) and T as the support of T ( s ) . Besides, we have also denoted T as a matrix with the same support of T . According to Assumption i, we have

Then we have

Additionally, we have

Since { J f ( s ( ℓ ) ) i, : n I } |F i, : n I | ℓ =1 forms a basis of R n I F i, : n I , for any j 0 ∈ F i, : n I , we are able to rewrite the one-hot vector e j 0 ∈ R n I F i, : n I as

where α ℓ is the corresponding coefficient. Then

where the final ' ∈ ' follows from Assumption i that each element in the summation belongs to R n I ˆ F i, : n I . Thus

Then the connections between these supports can be established according to Defn. 2.3

Since J f ( s ( ℓ ) ) : , : n I and J ˆ f (ˆ s ( ℓ ) ) : , : n I have full column rank n I , T ( s ( ℓ ) ) must have a non-zero determinant. Otherwise, it would follow that the rank of T ( s ( ℓ ) ) is less than n I , which would imply a contradiction that J ˆ f (ˆ s ( ℓ ) ) : , : n I = J f ( s ( ℓ ) ) : , : n I T ( s ( ℓ ) ) has a column rank less than n I .

The determinant of the matrix T ( s ( ℓ ) ) can be represented as its Leibniz formula as

̸

where S n I is the set of n I -permutations. Thus, there is at least one term in the sum that is non-zero, i.e.,

̸

̸

Then we can conclude that this σ is in the support of T ( s ) since s ( ℓ ) ∈ s . Therefore, it follows that

Combined with Eq. (118), we have

Because of the sparsity regularization on the estimated Jacobian, we further have

Together with Eq. (125), it follows that

̸

Suppose T ( s ) = D I ( s ) -1 P I -1 , then

which is equivalent to

Denote

Then we have

̸

̸

Additionally, consider j 3 ∈ { 1 , . . . , n I } for which

Since j 1 = j 2 , we can assume j 3 = j 1 without loss of generality. Based on Assumption iv, there exists C j 1 ∋ j 1 such that ⋂ i ∈C j 1 F i, : n I = { j 1 } . Because

Since j 1 ∈ F i 3 , : n I , we have ( i 3 , j 1 ) ∈ F : , : n I . Therefore, according to Eq. (118), we have

̸

̸

there must exists i 3 ∈ C j 1 such that

Note that σ ( j 3 ) ∈ T j 1 , : ∩ T j 2 , : implies

Then by Eqs. (132) and (133), we have

This implies ( i 3 , j 3 ) ∈ F : , : n I by Eqs. (124) and (127), which contradicts Eq. (131). Therefore, we have proven by contradiction that T ( s ) = D I ( s ) -1 P I -1 . By replacing T ( s ) with D I ( s ) -1 P I -1 in Eq. (113), we obtain Eq. (112), which is the goal.

B Experiments

In this section, we describe the experimental settings as well as some additional results.

B.1 Supplementary experimental settings

To produce observational data that meets the required assumptions for different models, we simulate the sources and mixing process as follows:

UCSS . To ensure that the true nonlinear mixing process adheres to the Structural Sparsity condition (Assumption ii in Thm. 3.1), as per previous work (Zheng et al., 2022), we generate observed variables in a structured way: Each observed variable is only a nonlinear mixture of its direct ancestors. For instance, if the observed variable x 1 has parents s 1 and s 2 , then x 1 = f 1 ( s 1 , s 2 ) . We use Generative Flow (GLOW) (Kingma and Dhariwal, 2018) with a projection layer as the nonlinear function f i . The difference between GLOW and GIN is that GLOW does not impose a constraint on the determinant of the Jacobian, thus being more suitable for the general nonlinear function since it has less inductive bias. The implementation of GLOW is a part of FrEIA 1 (Ardizzone et al., 2018-2022).

The ground-truth sources are sampled from a multivariate Gaussian, with zero means and variances sampled from a uniform distribution on [0 . 5 , 3] , which are of the same values as in previous works (Khemakhem et al., 2020a; Sorrenson et al., 2020; Zheng et al., 2022). It is worth noting that we sample sources from a single multivariate Gaussian so that all sources are marginally independent, unlike from most previous works assuming conditional independence given auxiliary variables.

Mixed . For the Mixed model, we partition sources into s I and s D . For sources in s I , we sample them in the same way as that for UCSS . For sources in s D , we sample them from 2 n D +1 multivariate Gaussian distributions as required by Assumption ii in Thm. 4.4. Similarly, these multivariate Gaussian distributions are of zero means and variances sampled from a uniform distribution on [0 . 5 , 3] . For sources in s I , we generate their influences on the observed variables in a structured way described above to satisfy the partial sparsity assumption (Assumption iv in Thm. 4.4); for sources in s D , we remove the constraint on the structure and permit each source to affect all observed variables.

Base . For the Base model, following (Sorrenson et al., 2020), we use GLOW (Kingma and Dhariwal, 2018) as the mixing function to generate the data. The sources are from a single multivariate Gaussian distribution with zero means and variances uniformly sampled from the interval [0 . 5 , 3] . No constraints on the structure have been imposed for the Base model.

In the evaluation of our model, we utilize the Mean Correlation Coefficient (MCC) as a metric for assessing the correspondence between the ground-truth and recovered latent sources. The MCC is calculated by first determining the pair-wise correlation coefficients between the true sources and the recovered sources after a nonlinear component-wise transformation learned by regression. Subsequently, an assignment problem is solved to match each recovered source with the corresponding ground-truth source that exhibits the highest correlation. MCC is a widely accepted metric in the literature for measuring the degree of identifiability, accounting for component-wise transformations (Hyvärinen and Morioka, 2016). Our results are all based on 20 trials, each with a different random seed.

For the synthetic datasets used in our experiments, the sample size is 2000. The parameters used for training include a learning rate of 0 . 01 and a batch size of 200 . Additionally, the number of coupling layers for both GIN and GLOW is set as 10. In regard to the "Triangles" dataset, it comprises 60 , 000 32 × 32 images of drawn triangles. The statistics of the dataset are described in (Yang et al., 2022). For the experiments conducted on this dataset, the learning rate is set at 3 × 10 -4 and the batch size is 100 . Concerning the EMNIST dataset, it includes 240 , 000 28 × 28 images of real-world handwritten digits. The learning rate and batch size used for these experiments are 3 × 10 -4 and 240 , respectively. The experiments are conducted directly using the official implementation of GIN 2 (Sorrenson et al., 2020) with an additional sparsity regularization term on the Jacobian of the estimated mixing function.

B.2 Supplementary experimental results

In this section, we delve deeper into the applicability of our results by offering additional empirical studies that further illuminate the implications of the proposed theory. Specifically, we focus on the following aspects: (1) the effect of different regularization terms on the performance of identification; (2) the applicability of the theory as illustrated by additional real-world examples.

Regularization. For the regularization term, directly utilizing the ℓ 0 penalty may be computationally infeasible as it results in a discrete optimization problem. To overcome this issue, we adopt the ℓ 1 regularizer, which has been extensively studied in the literature for highdimensional support recovery, particularly for variable selection (Wainwright, 2009) and Gaussian graphical model selection (Ravikumar et al., 2008). The usage of ℓ 1 regularizer induces sparsity in the solution; however, it may also introduce bias which can negatively affect the performance (Fan and Li, 2001; Breheny and Huang, 2011). This is because the ℓ 1 norm penalty also

3

4

5

6

Number of sources

0.78

0.80

0.82

0.84

0.86

MCC

1

MCP

SCAD

penalizes both small and large entries, unlike the ℓ 0 norm which remains constant for nonzero entries. To remedy this bias issue, we explore alternative penalties, such as the smoothly clipped absolute deviation (SCAD) penalty (Fan and Li, 2001) and minimax concave penalty (MCP) (Zhang, 2010), which can be interpreted as hybrids of ℓ 0 and ℓ 1 penalties. Additionally, we note that the support recovery of the ℓ 1 penalty is based on the incoherence conditions in various cases (Wainwright, 2009; Ravikumar et al., 2008, 2011), which may be restrictive in practice, whereas the SCAD and MCP penalties do not rely on such conditions (Loh and Wainwright, 2017). Based on our experimental results (Fig. 8), we adopt the MCP penalty as the regularization term.

EMNIST. To further demonstrate the generalizability of the proposed identifiability results, we present identification results for all digits on the EMNIST dataset. As previously mentioned, we show the recovered attributes with the top4 singular values. From Fig. 9, it is clear that these attributes are highly interpretable and appear to be the underlying concepts that influence the process of writing digits by hand. This indicates the potential applicability of our assumptions in real-world scenarios.

(b) Angle

(c) Upper width

(d) Height

O

0000000

00000

O

173456789

123956789

2

2

2

2

2

2

2

2

2

2

2

3456789

3

3

3

3

3

3

3

625ε

3

3

3

3

3

3

3

34567

456789

y5

4

4

4

y

u5é789

456789

5é7

4

hh

4

456789

5

5

5

5é7

5

5

S

5

68L26

2

2

2

J2

7

7

7

789

7

7

7

L

89

8

8

8

8

8

8

8

8

9

9

9

9

9

9

9

6

0000000(

00000000

?

123956789

2

2

2

2

2

2

2

2

2

2

N

2

2

2

2

2

3456789

3

3

33

333

3

3

3

3

33333

3

3

3956789

y5é76

4

44

4

S6789

4Ｓ6789

456789

4

4

44

4

456789

5

5

5

5

5

S

S

S

5

56:7

67

2

9

2

6

2

2

9

7

7

7

789

7

7

7

7

7

8

.8

8

8

8

8

(C)

89

8

8

8

68

9

9

9

9

9

9

9

9

9

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper utilizes the structural sparsity assumption on the support of the Jacobian matrix of the mixing function to extend the identifiability of nonlinear ICA in more settings including under-completeness, partial sparsity and source dependence, flexible grouping structures. It is a technically solid paper supported by theorems, proofs and experiments.\n\nSTRENGTHS:\n1. Overall, the manuscript is well written with clear organization, comprehensive literature review, technically solid theorems, detailed proofs and promising experiment results. \n\n2. This work addressed some limitations of theorems about identifiability with Structural Sparsity in Zheng et al. 2022 and extended nonlinear ICA with Structural Sparsity to more general settings. The proposed theorems could be more practically useful in real-world datasets.\n\n3. The notations, theorems and proofs are clear in general.\n\nWEAKNESSES:\n1. This work is interesting, and it would be great if code is provided to replicate the results. Please consider making the code publicly available. \n\n2. The meanings of some notations are not clear. See Questions. \n\n3. Ablation study. The author(s) only evaluated MCCs w.r.t. the number of sources. In Figure 3, the MCCs for 8 or 10 sources are a bit low so I wonder if more samples can help to improve MCCs. It would be more informative and convincing if more experiment configurations are considered (e.g., number of samples, various grouping structures) to demonstrate the effectiveness of proposed Theorems.\n\n4. Ablation study. Though the result comparison seems obvious visually, the authors should consider performing statistical tests to compare results between proposed methods and baseline method. \n\n5. Minor: \"exits\" should be \"exists\" at line 236.\n\nQUESTIONS:\n1. Theorem 3.1: Is $|\\mathcal{F}_{i,:}|$ the $L_0$ or $L_1$ norm of $\\mathcal{F}$? Is $\\mathcal{C}_k$ a minimal set of sample indices to uniquely identify source $k$? How does the assumption ii show Structural Sparsity? The regularization constraint $|\\hat{\\mathcal{F}}| \\leq |\\mathcal{F}|$, which induces sparsity, should be included in Theorems. Also, I note that the author(s) tried to explain the assumptions in the following paragraphs, but I would suggest to describe the Theorems, at least Theorem 3.1, in plain words so that readers can better understand the Theorems. Or at least explain the notations (e.g., $\\mathcal{C}_k$) which are not explained in Section 2 Preliminaries.\n\n2. Theorem 3.1: Zheng et al. 2022 also proposed a Theorem on the undercomplete case. Could you kindly clarify the novelty between your proposed Theorem and that proposed in Zheng et al. 2022?\n\n3. Theorem 4.1: The author(s) claimed that we do not need to know the dependence structures or the number of dependent sources, but it is not intuitive to me how Theorems 4.1 and 4.2 uncover the dependence structures and the number of dependent sources. Could you please clarify?\n\n4. Theorem 4.2: What are $u_1$ and $u_2$? Are they two different sets of auxiliary labels?\n\n5. Lines 281 - 283: The claim on multi-modal data is unclear. Could you please clarify how to identify linkage across multiple modalities?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article serves as an extension to the work of Zheng et al. 2022, which posited the identifiability of nonlinear ICA based on specific structural sparsity assumptions related to the mapping of sources and mixtures. This current article expands on that by addressing the undercomplete case—where the number of mixtures exceeds the number of sources—and furthermore, it relaxes both the sparsity and source independence assumptions to yield more general identifiability results.\tIn the end authors provide some numerical examples to illustrate the applications of their identifiability theorems.\n\nSTRENGTHS:\nThe article tackles the foundational issue of the nonlinear inverse problem, making significant assertions regarding fundamental identifiability theorems. These are premised on assumptions of partial independence and structural sparsity.The problem under investigation is a fundamental problem and the article offers some important results for this problem.\n\nWEAKNESSES:\nThe most significant shortcoming of the article lies in its presentation, particularly in its explanation of the core assumptions underpinning the theorems, as well as the motivation for the conditions applied in these assumptions. Absent a solid grasp of these theorems, it becomes challenging to properly evaluate the paper's contributions and ascertain its potential impact.\n\nIn terms of specific issues:\n\n* Concerning Theorem 3.1: This theorem seems to aim at generalizing Theorem 1 from Zheng et al., 2022 for a complete (m=n) case to an undercomplete (m>n) case. The identifiability results for ICA setups typically do not depend on a specific estimator choice or estimation algorithm. However, the statement of Theorem 3.1 seems rather unclear in this context. According to the article's notation, \\hat{f} refers to a specific estimate of the mixing function. Both the set of support matrices \\mathcal{T} and the support \\hat{\\mathcal{F}} depend on this particular estimate. The assumption (i) used in Theorem 3.1 is based on \\mathcal{T} and \\hat{\\mathcal{F}}, hence, this condition appears to be linked to a specific choice of the estimator for mixing. It would be beneficial if the authors could clarify whether the assumption (i) must hold on a particular \\hat{f}, a specific set of functions, etc. This clarification will likely impact the proof in the supplementary material and the explanation given between lines 129-136.\n\n\n* Line 97: Traditional or linear ICA does not necessarily require m=n.\n\n* Line 114: Should \\mathcal{S} be \\mathcal{A}?\n\n* Line 111 vs Line 536: The symbol \\mathcal{T} has two differing definitions - the set of matrices sharing the same support as T(s) and the support of T(s) itself.\n\n* Theorem 4.1: The vectors 'w' - whose independence implies identifiability - have not been sufficiently motivated or explained.\n\n* Similar comments can be made for other identifiability theorems.\n\nQUESTIONS:\n* Line 190: Could you clarify what is meant by \"changing\" sources?\n\n* Line 195-196: 6, the phrase \"For sources s_D, they do not need to be mutually independent as long as they are dependent on the variable u\" is somewhat confusing. Subsequent equation (3) implies that S_D and S_I are conditionally independent when conditioned on u, and that the components of s_I are independent. What is the necessity for this latent variable u? Perhaps the authors could shed some light on this.\n\n* How does Theorem 4.1’s contribution compare with those from Khemakhem et al. (2020a) and Sorenson et al. (2020)?\n\n* Lines 215-217: Could you elucidate what condition (i) in Theorem 4.2 represents? What do s_d and B_{s_I} signify? Perhaps the discussions on lines 229-247 should precede Theorem 4.2 to better contextualize its contents and results, potentially with more lucid explanations.\n\n* Regarding Figure 4 and 5, could you expound on how these examples pertain to the nonlinear ICA setup (what are the sources, which appear to be images, and what are the nonlinear mixings)? How are the interpretations in the captions of these figures derived? Could you also elucidate how these examples relate to the identifiability theorems presented in the article and the conditions stipulated in these theorems?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends identifiability theory of nonlinear ICA (NICA), and deep latent variable models in general, by utilizing structural sparsity. In particular, previous works have shown that NICA can be identified if there is some observed auxiliary data or latent dependencies that essentially capture the inductive biases in the data generative process. The approach of structural sparsity (Zheng, '22) instead takes an alternative approach, namely constraining the nonlinear mixing function and its Jacobian. In this paper the authors extend that work as follows i.e. assuming structural sparsity and some additional assumptions:\n\n1. the authors show identifiability for a situation in which the mixing function is injective rather than bijective i.e. undercomplete case / i.e. smaller latent dimension than observed dimension. This is important for e.g learning low dimensional, semantically meaningful, interpretable latent features\n\n2. the authors show identifiability for a situation in which the structural sparsity principle applies only to some of the independent components\n\n3. further identifiability is shown for situation where not all the latent components are independent, rather components form independent subspaces. In fact some components may be dependent, conditionally independent or have some grouping structures.\n\nSTRENGTHS:\nFirst, this paper is in general of good quality in that it is well organized and, in general, clearly written.\n\nMain strengths:\na.) most significantly, authors remove several strong limitations of previous works and extend identifiability of structural sparsity to undercomplete and case where not all latent components are structurally sparse (in those situations the remaining sources are shown to be identifiable). As a result these ideas are now more applicable to realistic data and scenarios.\n\nb.) These results have been reached, mostly, without too strong additional assumptions. For instance, it is shown that the necessary assumptions are more likely to hold in this new undercomplete case which is encouraging!\n\nc.) authors bridge gap between structural sparsity and the previous works that assume auxiliary variables. in particular, this work allows unconditionally independent components to follow structural sparsity and components which are conditionally independent given auxiliary variables. Whilst arguably to be expected, it is important to show this result (but see below for potential related weakness)\n\nWEAKNESSES:\nIn general the weakness of this paper is that provides only few theoretical advances (albeit important; as mentioned above) but provides little beyond that. In particular, this is the results of:\n\n1. Contribution of the paper is not as significant as the authors describe, or at least there is limited coverage of relevant works\n2. Potential problems in some of the identifiability theorems\n3. Novelty is limited to identifiabiltiy theorems -- no new algorithms \n4. Experiments are lacking\n\nI will expand on each of these points below:\n\nMore detail for 1.):  In particular the authors state that \"Therefore, we establish, to the best of our knowledge, one of the first general frameworks for uncovering latent variables with appropriate identifiability guarantees in a principled manner\". I think this is too vague and general and fails to acknowledge the generality of some other works -- your work can be novel whilst admitting the generality of some other works too. First, Kivva '22 show a very general framework for identifiability by making, arguably, less strong assumptions on the mixing functions -- currently the work of Kivva '22 is only mentioned later on in section 3 (and even there in a problematic manner as ill point out below). Due to the generality of the results in Kivva, I would expect their result to be discussed in the introduction / early on in the text and tell the reader why yours is better or at least different. For example Kivva make different type of assumptions on the mixing function (piecewise affine) etc, while you on the sparsity. Second the work of Halva '21 (disentangling identifiable features) provides another very general framework and unlike what you claim, it is not limited to time-series but to any dependencies of arbitrary order, and also does not require condition independence on some auxiliary variables but rather also assumes unconditional independence.\n\nMore detail for 2.): In Theorems 4.1 and 4.3  $S_d$ is \"identified up to an invertible transformation\". Surely if something is identified just up to invertible (vector-valued) function then we are not doing any better than nonlinear ICA i.e. we are essentially where one started and thus we have not identified anything. To me this is misleading and not a publishable identifiability result (if I have understood correctly -- please correct me if I'm wrong and I'll adjust my score accordingly). Authors do acknowledge this point but rather than talking about it they make a vague remark that \"Thm. 4.1 may be helpful for some tasks that do not necessitate the recovery of each individual source, such as domain adaptation.\" This does not suffice in my opinion. And similarly about Thm 4.3 they say: \"there exists an invertible transformation $h_{c_i}$ which is analogous to the previous element-wise indeterminacy. Consequently, even when dealing with mixtures of high and one-dimensional sources, like in the case of multi-modal data, we can still recover the hidden generating process to some extent.\" Again I think this is bit generous and hiding the fact that $\\mathbf{s}_{c_i}$ is fully unidentifiable in the sense of nonlinear ICA. At least this limitation must be admitted more clearly -- preferably its usefulness would be shown empirically.\n\nMore detail for 3.):\nThere is a simple regularization term of the jacobian added -- but this is heurestic (vs. mle methods) from previous work. Undoubtedly there could be work done towards what is the best way to estimate a model that assumes structural sparsity but such is not done here.\n\nMore detail for 4.) \nAn important question is whether structural sparsity is a valid assumption. I think this can indeed be the case for many types of generative processes.  But the question then is then do the experiments strengthen that intuition. I feel not. It is not clear to me why e.g. EMNIST experiment there would be structural sparsity. EMNIST is also a very simple data set. I would expect the experiments to show that the learned independent components are useful in practical applications (see the brain signal experiments in Halva '21 for instance or in Khemakhem (iVAE) '20). I'm not saying specifically this type of real data experiments need to be introduced, but something to further highlight the strenght of this method would be helpful. Another example is to evaluate the method more thoroughly on some benchmarks from the disentanglement learning literature. There are also some claims in the paper that would be good to justify experimentally for instance you claim on line 311-313 that \"This is particularly helpful in the context of self-supervised learning 311 (Von Kügelgen et al., 2021) or transfer learning (Kong et al., 2022), where latent representations are 312 modeled as a changing part and an invariant part.\" If this indeed the case, why not show that on data? Indeed 311 to 324 gives nice discussion and its a great shame this has not been shown experimentally as it would really take this paper to the next level.\n\nQUESTIONS:\nI will use this space for further suggestions and questions:\n\n- Could you please introduce structural sparsity bit more clearly and intuitively -- if one has not read the original Zheng'22 paper then it is difficult to follow.\n\n- In estimation, please clarify: is it required that we know which groups of latent variables are independent, and which are potentially dependent? How is this exactly established in practice?\n\n- please explain in more detail how your algorithm, in practice, allows dimension reduction without assuming observation noise, and how jacobian can be computed for non-bijective transformation\n\n- What is the level of nonlinearity in the mixing functions? I dont believe this is mentioned anywhere, e.g. number of layers or similar. \n\n- \". Since the proposed condition is on the connective structure from sources to observed variables, i.e., the support of the Jacobian matrix of the mixing function, it does not require the mixing function to be of any specific algebraic form.\". Please make this sentence bit more precise or explain better what does 'specific algebraic form' mean. Because structural sparsity does still limit the form of the function -- f can not longer be any arbitrary function.\n\n-  \"Most of these methods require auxiliary variables to be observable, such as class labels and domain indices (Hyvärinen and Morioka, 2016, 2017\". H&M 2017, really only require the previous data so it's not really a big limitation and it's arguable whether this really constitutes of having auxiliary variables...I would consider moving that reference to the next sentence since it's a time-series model : \"with the exceptions being those for time series...[move H&M'17 reference here]\"\n\n- \"The most obvious  one arises from the fact that it may fail in a number of situations where the generating processes are heavily disentangled.\" Please explain in more detail why this may be?\n\n- \"We first present the result on removing one of the major assumptions in ICA, i.e., the number of observed variables m must be equal to that of hidden sources n.\" This makes it sound like it hasn't been done previously in general, which of course it has been done many times previously in linear ICA (e.g eriksson and koivunen '03) and nonlinear ICA (e.g. khemakhem '20, halva '21 etc etc). so rather than saying removing majort assumption in ICA, make it specific to sparsity\n\n- as for the title: are you really \"generalizing beyond structural sparsity\"? as I feel structural sparsity is still the fundamental building block here. I would say you are generalizing structural sparsity in nonlinear ICA.\n\n- \"This is similar to Independent Subspace Analysis (ISA) (Theis, 2006)\"  Either explain why you cite Theis, or cite an earlier ISA work (e.g. Hyvarinen & Hoyer, 2000)?\n\nLIMITATIONS:\nAuthors should discuss limitations in more detail:\n\n- what are possible limitations of structural sparsity assumptions should be discussed more e.g. any scenarios where you expect it to be a poor assumption?\n\n- the point discussed in the \"Weaknesses\" section about the limitations in the identifiability theorems of 4.1 and 4.3 must be addressed and justified much more clearly or else those theorems should be removed\n\n- authors should note the heurestic approach of their estimation algorithm -- does GIN even have universal _function_ approximation capability? \n\n- related, discuss limitations of estimation algorithm in general. Is the algorithm guaranteed to find the correct sparsity for example? If not, then that should be pointed out as need for future works.\n\n- \"However, our setting is more flexible in the sense that we do not assume all sources to be influenced by the auxiliary variable. Specifically, sources in $s_I$ are mutually independent as in the original ICA setting, while only  sources in $s_D$ have access to the side information from the conditional independence given u,\". This is true and a nice result of theorem 4.4, but there is the limitation that should be discussed, namely now you are making restricting assumption on _both_ the mixing function as well as on the auxiliary variables -- in some sense this is worst of both worlds (but still a nice theoretical result with possible practical uses).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends a recent result from Zheng et al 2022, which introduces an assumption they call “structural sparsity” to induce identifiability in nonlinear ICA without relying on a common (but arguably unrealistic) assumption that the observed variables are conditionally independent given observed auxiliary information. Whereas Zheng et al 2022 gave identifiability results only in the setting where the structural sparsity assumption holds perfectly and there are an equal number of sources and observed variables, this paper relaxes these assumptions in several interesting ways and gives identifiability or partial identifiability results in these more general settings.\n\nThe first theoretical contribution shows identifiability under structural sparsity in the undercomplete setting, where there are more observed variables than sources. This lets them relax the usual assumption that the mixing function must be bijective, and instead only requires that the mixing function be injective.\n\nThe second theoretical contribution relaxes the structural sparsity assumption to the setting where you have partial structural sparsity (it holds for a subset of sources) or partial independence of sources and shows partial identifiability under these assumptions. Here the partial dependence of sources does not need to be known.\n\nThe third theoretical contribution assumes that the dependence between sources is known, and the fourth theoretical contribution assumes the sources with dependencies are conditionally independent given auxiliary variables (which is distinct from existing work because they don’t assume all sources are influenced by the auxiliary variable, just the dependent sources).\n\nThey use an estimation method using a sparsity regularizer (that encourages a sparse estimated mixing function) with a flow-based generative model. They perform experiments on two simple visual datasets (Triangles and EMNIST) and perform ablations where they generate data that satisfy two combinations of assumptions for their theory, compared to a base setting that does not satisfy their assumptions. Following existing work, they use MCC as their metric and their models achieve higher MCC when the assumptions are satisfied.\n\nSTRENGTHS:\n- Overall, this is a very interesting paper and makes novel contributions in what I think is an interesting setting: using sparsity to induce identifiability in nonlinear ICA.\n- They clearly motivate why relaxing each assumption makes the assumptions more realistic.\n- I agree that the “conditional independence given auxiliary information” assumption that is common in the literature is not a great assumption, and I’m happy to see recent work removing or reducing this assumption.\n- They don’t require distributional assumptions.\n- It is well-written and well-structured overall. It is very clear what the prior work accomplishes and what the contributions are.\n\nWEAKNESSES:\n- There could be more experiments in realistic settings. (However, given the strength of the theoretical contributions in this paper, I think the paper should be accepted as is.)\n\nMinor comments on the writing (did not affect score):\n- Line 84-85: You say “part of the sources can be grouped into irreducible independent subgroups…”, but “irreducible subgroup” is a term in algebra with a specific meaning. You could avoid this “collision” by saying “irreducible independent subgroupings” or something similar.\n- Line 156: You start a sentence with the word “Differently, …” which sounds strange. You could say “In contrast, …” instead.\n- Line 167: “While this removes the restriction of bijectivity between sources and observed variables, it remains uncertain as to whether Structural Sparsity holds in general, particularly for all sources in a universal way.”\n    - This sentence is confusing - you are saying it is uncertain whether Structural Sparsity holds in general, but Structural Sparsity is one of your assumptions. Are you saying it is uncertain whether Structural Sparsity is a reasonable assumption, based on whether it is likely to be satisfied on real-world data?\n- Line 177: It’s also weird to start this sentence with “Differently”.\n- Multiple lines: You start a handful of sentences with “Besides, …” and each time that is not really the word you mean. You should rethink how each of these sentences connects to the previous sentences and find the appropriate word for each case.\n\nQUESTIONS:\n- Are you aware of Lachapelle et al 2022b? See https://arxiv.org/pdf/2207.07732.pdf. Lachapelle et al 2022a uses mechanism sparsity to induce permutation identifiability, but Lachapelle et al 2022b extends this approach to the partial identifiability setting. It would be (1) worth mentioning in the Introduction section that Lachapelle et al 2022a introduced the idea to use sparsity to induce identifiability, which inspired the approach of Zheng et al. 2022 (as stated in the text of Zheng et al 2022, see Section 3.1 of that paper), and (2) to cite Lachapelle et al 2022b as prior work using sparsity for partial identifiability (though in a distinct setting from your results as it relies on conditional independence given observed auxiliary variables).\n\nLIMITATIONS:\n- Their experiments are only on visual disentanglement tasks and there are many other interesting disentanglement or other tasks that would be interesting to see in future work.\n- No concerns about negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces more flexible ways to perform nonlinear independent component analysis (nonlinear ICA).  Nonlinear ICA involves identifying the sources s from the observed x when both s and x are related by x = f(s) and f is a nonlinear function.  Previous work has developed a method for this problem under a strict structural sparsity assumption that the s's and x's are one-to-one and onto, and all the s's are independent of each other.  Current work provides theorems that relax the assumption in various ways including: (1) undercompletness--there can be more observed variables x than sources s; (2) partial sparsity--only a subset of all the s's may map to x's; (3) source dependence--all sources s do not have to be statistically dependent on each other; and (4) flexible grouping structures--the possibility that some of the sources can be partitioned into independent subgroups of sources.  There are experiments on synthetic and real world datasets that show the effectivness of their approach.\n\nSTRENGTHS:\nThis paper is original because it introduces novel approaches, as far as I know, that extend the situations where nonlinear ICA can be applied.\n\nThe paper exhibits good quality in various ways.  First, there are various theoretical results included in the paper that extend the cases where nonlinear ICA can be applied.  Second, there are also results in several experimental settings that back up the theory.\n\nThe paper is mostly clear in its explanations.\n\nIn terms of significance, extending the situations where nonlinear ICA can be applied is an accomplishment.\n\nWEAKNESSES:\nWhile in theory extending the cases where nonlinear ICA can be applied is a strength, because there wasn't any empirical qualitative evaluation of how this approach compares to other approaches in disentangling the sources, it is not clear how significant this work is.  It does not have to be a comparison of how well it disentangles sources; it could be comparing them on some other application, such as how well they extract features that are useful for classification, for example.  It does not even have to be comparing this paper's approach to previous approaches; it could be comparing the different extensions of nonlinear ICA presented in this paper.\n\nAlso, as pointed out by the authors, another limitation of this work is that the experimental results were only on visual datasets but not on other modalities.\n\nQUESTIONS:\nWhile the undercompleteness result appears to me to be unique to this paper, (Zheng et al 2022) also has an undercompletness result.  This paper is written so that it sounds like (Zheng et al 2022) has no undercompletness result.  It would be nice if this situation could be explained or clarified.\n\nIt was a bit confusing that on line 113, A is defined as a set of natural number tuples but on line 103 A is defined as a subset of natural numbers.\n\nI think on line 114 that A_{:,j} := { i | (i,j) \\in S } should really be A_{:,j} := { i | (i,j) \\in A }.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}