Title: On the Identifiability of Sparse ICA without Assuming Non-Gaussianity
Venue: NeurIPS 2023
Accepted: yes
Keywords: independent component analysis

second-order statistics

sparsity
Abstract: Independent component analysis (ICA) is a fundamental statistical tool used to reveal hidden generative processes from observed data. However, traditional ICA approaches struggle with the rotational invariance inherent in Gaussian distributions, often necessitating the assumption of non-Gaussianity in the underlying sources. This may limit their applicability in broader contexts. To accommodate Gaussian sources, we develop an identifiability theory that relies on second-order statistics without imposing further preconditions on the distribution of sources, by introducing novel assumptions on the connective structure from sources to observed variables. Different from recent work that focuses on potentially restrictive connective structures, our proposed assumption of structural variability is both considerably less restrictive and provably necessary. Furthermore, we propose two estimation methods based on second-order statistics and sparsity constraint. Experimental results are provided to validate our identifiability theory and estimation methods.

On the Identifiability of Sparse ICA without Assuming Non-Gaussianity

Abstract

Independent component analysis (ICA) is a fundamental statistical tool used to reveal hidden generative processes from observed data. However, traditional ICA approaches struggle with the rotational invariance inherent in Gaussian distributions, often necessitating the assumption of non-Gaussianity in the underlying sources. This may limit their applicability in broader contexts. To accommodate Gaussian sources, we develop an identifiability theory that relies on second-order statistics without imposing further preconditions on the distribution of sources, by introducing novel assumptions on the connective structure from sources to observed variables. Different from recent work that focuses on potentially restrictive connective structures, our proposed assumption of structural variability is both considerably less restrictive and provably necessary. Furthermore, we propose two estimation methods based on second-order statistics and sparsity constraint. Experimental results are provided to validate our identifiability theory and estimation methods.

1 Introduction

Independent component analysis (ICA) [12] has emerged as an essential statistical tool in the scientific community, with application in various disciplines including neuroscience [27], biology [41, 9], and Earth science [32]. It aims to uncover the hidden generative processes that govern the observed data and separate mixed signals into independent sources. However, it is known that the traditional approaches of ICA struggle with Gaussian sources, which may limit their applicability in a wide variety of contexts. For instance, several biological traits and measurements such as height, blood pressure, and measurement errors in genetics, as well as the thermal noises in electronic circuits, may often be normally distributed [29, 34]. Classical identifiability results in ICA rely on higher-order statistics (e.g., Kurtois) [24], and cannot provide desired theoretical guarantees when there is more than one Gaussian source. Identifiability based on second-order statistic is thus essential in these scenarios.

The primary hurdle in applying ICA to Gaussian sources lies in the rotational invariance of Gaussian distribution [25]. To address this issue, earlier studies [30, 5, 35] incorporated additional information by assuming that the sources are nonstationary. However, this extra information may not always be readily available, limiting the generalizability of these approaches. Thus, recent research [49, 1] has started to delve into the connective structure between the sources and the observed variables, as opposed to solely focusing on distributional assumptions (e.g., non-Gaussianity and nonstationarity). This shift of focus is motivated by a key observation: despite the rotational invariance in the distribution of Gaussian sources, the sparsity of mixing matrix undergoes noticeable changes, i.e., it may be denser after rotation [49]. Building on this insight, Zheng et al. [49] introduced two assumptions on the support of the mixing matrix to achieve identifiability of Gaussian sources, leading to a novel perspective for tackling this long-standing challenge in the field of ICA. On the other hand, Abrahamsen and Rigollet [1] assumed that the mixing matrix is generated from a sparse Bernoulli-Gaussian ensemble.

Although the rotational invariance of Gaussian distribution can be resolved by the structural assumptions on the mixing matrix proposed by Zheng et al. [49], they may be deemed overly restrictive. For instance, both of their structural assumptions cannot deal with the case where the set of observed variables influenced by one source is a subset of those affected by another source, which may not be uncommon in practice. This may limit the applicability of ICA in complex real-world scenarios, thus underscoring the need for a weaker and more flexible structural assumption that is capable of addressing the rotational invariance in a more universally applicable manner.

To enhance the applicability with Gaussian sources, we develop an identifiability theory of ICA from second-order statistics under more flexible structural constraints. We introduce a novel assumption, namely structural variability , that is considerably weaker than existing ones. Notably, this assumption is proved to be among the necessary conditions for identifying Gaussian sources by focusing on the connective structure (i.e., the support of the mixing matrix). Moreover, we propose two estimation methods grounded in sparsity regularization and continuous constrained optimization. The efficacy of our proposed methods has been validated through experiments, which also reaffirm the validity of our theoretical result. Lastly, as a matter of independent interest, we establish the connection between our identifiability result of ICA with causal discovery from second-order statistics; our finding further bridges the gap between these two fields and provides insights into the interpretation of our result.

2 Problem Setting

We consider the ICA setup given by x = ˜ As , where x ∈ R n denotes the observed random vector, s ∈ R n is the latent random vector representing the independent components, also called sources, and ˜ A = [˜ a 1 | · · · | ˜ a n ] ∈ R n × n denotes the unknown mixing matrix. We assume here that ˜ A has full rank and all sources are standardized. For the estimated mixing matrix ˆ A and the ground-truth one ˜ A , we write ˆ A ∼ ˜ A if they differ only in column permutations and sign changes of columns, and ˆ A ̸∼ ˜ A vice versa. The goal is then to estimate ˆ A such that ˆ A ∼ ˜ A ; in this case, one could identify a one-on-one mapping between the ground truth sources and the estimated ones, i.e., the unknown mixing process has been demixed during estimation. Since the support of a mixing matrix A essentially represents the connective structure between sources and observed variables, we have the following definition for the ease of reference.

̸

Definition 1 (Connective Structure). Given a mixing matrix A , we define its connective structure as a directed bipartite graph G A = ( V A , E A ) from sources s to observed variables x , where the nodes and edges are defined as V A := { s i } n i =1 ∪ { x i } n i =1 and E A := { ( s j , x i ) : a ij = 0 } , respectively.

̸

Notations. We use bold capital letters (e.g., A ), bold lowercase letters (e.g., a ), and italic letters (e.g., a ) to denote matrices, vectors, and scalar quantities, respectively. For any matrix A , we denote its j -th column by a j , i -th row by a i, : , and ( i, j ) -th entry by a i,j . Wealso denote by A J the submatrix of A by obtaining the columns indexed by set J . For any vector a , we denote its i -th entry by a i . We define the support set of matrix A as supp( A ) := { ( i, j ) : a i,j = 0 } , and its support matrix as ξ A which is of the same size as A , where ( ξ A ) i,j = × if a i,j = 0 and ( ξ A ) i,j = 0 otherewise. The notations of support set and support matrix are similarly defined for vector a . We denote by ∥ A ∥ 0 the number of nonzero entries in A , and we have ∥ A ∥ 0 = | supp( A ) | = ∥ ξ A ∥ 0 . Furthermore, we denote the n × n identity matrix and m × n zero matrix by I n and 0 m × n , respectively; to lighten the notation, we drop their subscripts when the context is clear. We also use [ n ] to denote { 1 , 2 , . . . , n } .

̸

3 Identifiability Result without Assuming Non-Gaussianity

By exploiting the non-Gaussianity of the sources, such as fourth-order cumulant, existing approaches are able to estimate the true mixing matrix ˜ A up to signed column permutation when there is at most one Gaussian source [23, 2]. However, these approaches typically fail in the presence of more than one Gaussian source, because higher-order statistics cannot be utilized for full identifiability. The primary challenge of achieving identifiability for Gaussian sources lies in the rotational invariance of the Gaussian distribution. More specifically, the second-order statistics (or, more specifically, population-level covariance matrix) ˜ Σ = ˜ A ˜ A ⊤ remains unchanged if one replaces ˜ A with ˜ AU for any orthogonal matrix U . Therefore, considering only second-order statistics, the true mixing matrix ˜ A is, generally speaking, only identifiable up to right orthogonal transformation without

further assumptions. In this section, we adopt a different perspective that departs from traditional distributional assumptions (i.e., non-Gaussianity and fourth-order cumulant), and instead introduces novel and precise assumptions on the connective structure, specifically the support of the mixing matrix. These assumptions enable identification of the true mixing matrix ˜ A up to signed column permutation using second-order statistics and sparsity constraint. Roughly speaking, with the population-level covariance matrix ˜ Σ , we consider the following formulation:

Note that we start with the assumption of AA ⊤ = ˜ Σ in the formulation above and our identifiability result (e.g., in Theorem 1); such an assumption can be obtained, e.g., from the equality of Gaussian likelihoods in the large sample limit. This also inspires our sparsity-regularized likelihood-based estimation method that will be described in Section 4.2, which is more inline with, e.g., model selection approaches based on sparsity-regularized likelihood [36, 18].

In Section 3.1, we first examine various types of constraints arising from second-order statistics. We then present the main identifiability result in Section 3.2. We show that our assumptions are strictly weaker than existing sparsity assumptions in Section 3.3, and establish the connection between our identifiability theory with causal discovery in Section 3.4. All proofs are given in Appendices C and D.

3.1 Semialgebraic Constraints Arising from Second-Order Statistics

We first discuss various notions related to constraints arising from covariance matrices of observed variables x , which serve as a fundamental basis for introducing our assumptions and identifiability result of ICA in Section 3.2. These notions have been studied in the field of algebraic statistics [16], factor analysis [15], graphical models [19, 40], and causality [14, 20].

We begin with the following definition on the set of covariance matrices entailed by ξ for different values of the free parameters in ξ .

Definition 2 (Covariance Set). The covariance set of support matrix ξ is defined as

The support ξ of mixing matrix A imposes certain constraints on the entries of the covariance matrix Σ = AA ⊤ , which, by Tarski-Seidenberg theorem (see [6]), correspond to semialgebraic constraints , i.e., equality and inequality constraints. The covariance set Σ ( ξ ) is then said to be a semialgebraic set , i.e., a set that can be described with a finite number of polynomial equations and inequalities [6]. Clearly, if a covariance matrix Σ belongs to the covariance set Σ ( ξ ) , then Σ satisfies the semialgebraic constraints imposed by ξ .

For an equality constraint, the set of values satisfying the constraint has zero Lebesgue measure over the parameter space involved. Given a support matrix ξ , we denote by H ( ξ ) the set of equality constraints it imposes on the corresponding covariance matrices. On the other hand, the set of values satisfying an inequality constraint has nonzero Lebesgue measure. To illustrate these constraints, we provide a three-variable example below. Note that the example only serves as illustrations of the constraints; our estimation methods (in Section 4) do not require deriving them in practice.

Example 1 (Semialgebraic Constraints). Consider support matrices

The equality constraints imposed by ξ 1 include

while the inequality constraints imposed by ξ 2 include

The detailed derivation can be found in Appendix D.1 and provides insights into how such constraints arise from the corresponding support matrices. In the example above, the covariance matrix Σ generated by any mixing matrix A with support ξ 1 must satisfy the corresponding equality constraint; similarly, the covariance matrix Σ generated by any mixing matrix A with support ξ 2 must satisfy the above inequality constraint. These constraints serve as footprints of the mixing matrix on the covariance matrix, and can be exploited for its identifiability, which we explain in the next section.

3.2 Identifiability Result from Second-Order Statistics

In this section, we present our identifiability result of ICA from second-order statistics. The core idea is to introduce precise and mild assumptions on the connective structure from sources to observed variables. These assumptions facilitate the identification of the mixing matrix through the application of a sparsity constraint, formulated in Problem (1). To begin, we describe our primary assumption concerning the connective structure as follows.

̸

Assumption 1 (Structural Variability). Every pair of the columns in the support matrix of A differ in more than one entry. That is, for every i, j ∈ [ n ] and i = j , we have

The assumption above implies that every pair of sources should influence more than one different observed variable. Notably, in the field of nonlinear ICA with auxiliary variable, Hyvärinen and Morioka [22], Hyvärinen et al. [26] have adopted the assumption of sufficient variability which requires that the auxiliary variable has a sufficiently diverse effect on the distributions of sources; specifically, the conditional distributions of the sources given the auxiliary variable must vary sufficiently. In contrast, our assumption of structural variability requires that every pair of sources influence sufficiently diverse sets of observed variables, facilitating the disentanglement of each source.

We provide several examples in Appendix E.1 to illustrate the broad applicability of the assumption above. Furthermore, the following proposition justifies such an assumption because it is a necessary condition for identifiability via second-order statistics and sparsity. The intuition is that if Assumption 1 is violated, there exists a rotation that maps matrix ˜ A to another matrix ˆ A which has equal or smaller number of nonzero entries and is not a column permutation of ˜ A .

Proposition 1. If the true mixing matrix ˜ A does not satisfy Assumption 1, then there exists a solution ˆ A to Problem (1) such that ˆ A ̸∼ ˜ A .

Remark 1 (Necessary Condition). Assumption 1 is a necessary condition for identifiability of ICA via second-order statistics and under sparsity constraint.

We also adopt the following assumption on the mixing matrix for the identifiability of ICA.

Assumption 2 (Permutations to Lower Triangular Matrix). The matrix A can be permuted by independent row and column permutations to be lower triangular. That is, there exist permutation matrices P 1 and P 2 such that P ⊤ 1 AP 2 is lower triangular.

As we show in the proof of identifiability result in Theorem 1, Assumption 2, loosely speaking, ensures that the resulting covariance matrix does not contain 'nontrivial' inequality constraints. In Example 1, support matrix ξ 1 satisfies Assumption 2 and leads to an equality constraint, while matrix ξ 2 fails to meet this assumption, resulting in an inequality constraint. The Lebesgue measure of the parameters leading to such inequality constraint is not zero, thus requiring additional assumptions to handle such cases. Therefore, we adopt Assumption 2 in this work and focus on equality constraints.

A key ingredient of our identifiability result based on sparsity is the dimension of the covariance set Σ ( ξ ) . It may be natural to expect that the dimension of Σ ( ξ ) , denoted as dim( Σ ( ξ )) , equals the number of parameters used to specify the mixing matrices, i.e., ∥ ξ ∥ 0 . This is not the case for general mixing matrices, but we show that such property holds under Assumption 2.

Proposition 2 (Dimension of Covariance Set). Let ξ be a support matrix that satisfies Assumption 2. Then, its covariance set has a dimension of ∥ ξ ∥ 0 , i.e., dim( Σ ( ξ )) = ∥ ξ ∥ 0 .

Note that Assumption 2 allows independent row and column permutations, which thus may be rather mild especially for sparse mixing matrix. Below we provide an example of the connective structure that satisfies this assumption. We also introduce an efficient approach to verify whether a mixing matrix satisfies Assumption 2 in Appendix E.2.

Example 2. If the connective structure G A of mixing matrix A is a polytree, then matrix A satisfies Assumption 2.

Finally, the following assumption is needed to ensure that the equality constraints arising from the covariance matrix are entailed by the true mixing matrix, rather than accidental parameter cancellations. This establishes a correspondence between equality constraints in the covariance matrix and those imposed by the support of the mixing matrix. Similar assumption has been employed in various tasks such as causal discovery [39, 20], as discussed in Section 3.4.

Assumption 3 (Faithfulness). For a mixing matrix A and the resulting covariance matrix Σ , Σ satisfies an equality constraint κ only if κ ∈ H ( ξ A ) .

The following proposition justifies Assumption 3 and demonstrates that it is a generic property in the sense that it holds almost everywhere in the space of possible mixing matrices. In other words, it is only violated for a set of mixing matrices with zero Lebesgue measure.

Proposition 3 (Generic Property). Suppose that the nonzero coefficients of matrix A are randomly drawn from a distribution that is absolutely continuous with respect to Lebesgue measure. Then, matrix A satisfies Assumption 3 with probability one.

With the aforementioned assumptions in place, we are ready to present our main identifiability result of ICA: by making use of second-order statistics and sparsity constraint, we show that the true mixing matrix can be identified up to signed column permutation. The key intuition is as follows: although in general there is rotational invariance in second-order statistics, the sparsity of the estimated mixing matrix, under our assumptions above, is guaranteed to be denser than the true mixing matrix after any rotation (except permutation). Note that the proof leverages the technical tools developed by Ghassami et al. [20] which involve different types of rotations.

Theorem 1 (Identifiability with Sparsity). Suppose that the true mixing matrix ˜ A satisfies Assumptions 1, 2, and 3. Let ˆ A be a solution of the following problem:

Then, we have ˆ A ∼ ˜ A .

3.3 Related Sparsity Assumptions

As briefly discussed in Section 1, Zheng et al. [49] have also provided identifiability result of ICA without assuming non-Gaussianity. Their approach, similar to ours, relies on assumptions related to the support of the mixing matrix, which are provided below.

Assumption 4 (Zheng et al. [49]). For every set I ⊆ [ n ] where |I| > 1 and for all i ∈ I , we have

Assumption 5 (Zheng et al. [49]). For every i ∈ [ n ] , there exists set I ⊂ [ n ] such that

While Assumptions 4 and 5 shed light on resolving the long-standing challenge of the rotational indeterminacy of Gaussian sources, their general applicability remains unclear. One notable restriction is that they require each column of the support matrix to not be a subset of the other, formalized below. This may be overly restrictive in practical scenarios and is not the case for our Assumption 1. Assumption 6 (Column Subset). Each column in the support of A is not a subset of the other.

Since the true generating process of real-world data is inaccessible, it is challenging to quantitatively evaluate the applicability of these sparsity assumptions. In light of this challenge, we demonstrate the significance and advantage of our Assumption 1 by proving that it is strictly weaker than Assumptions 4 and 5. This also strengthens the validity of our result (i.e., Proposition 1) that Assumption 1 is a necessary condition for achieving identifiability with second-order statistics and sparsity constraint.

Theorem 2. For mixing matrix A , we have the following chain of chain of implications:

Assumption 4 = ⇒ Assumption 6 = ⇒ Assumption 1 .

Furthermore, there exists a matrix A satisfying Assumption 1 that does not satisfy Assumption 4.

Theorem 3. For mixing matrix A , we have the following chain of chain of implications:

Assumption 5 = ⇒ Assumption 6 = ⇒ Assumption 1 .

Furthermore, there exists a matrix A satisfying Assumption 1 that does not satisfy Assumption 5.

Remark 2. Assumption 1 is not only strictly weaker than both Assumption 4 and 5, but also accommodates cases where one column of mixing matrix A is a subset of of another (provided that their supports differ in more than one entry). On the other hand, the latter two assumptions cannot accommodate such cases, further demonstrating the flexibility and broader scope of Assumption 1.

3.4 Connection with Causal Discovery

ICA has emerged as a useful tool for causal discovery over the past two decades [38]. In particular, Shimizu et al. [38] demonstrated that the identifiability of ICA based on non-Gaussianity can be leveraged to discover the complete structure of a linear non-Gaussian structural equation model (SEM). In this section, we establish the connection and provide an analogy between ICA and causal discovery from second-order statistics. This connection further bridges the gap between these two fields, and provides insights into the interpretation of our identifiability result.

Let R n × n off be the set of matrices whose diagonal entries are zero, and diag( R n > 0 ) be the set of positive diagonal matrices. Consider the linear SEM x = ˜ B ⊤ x + e , where x denotes the random vector, ˜ B ∈ R n × n off denotes the weighted adjacency matrix representing a directed graph without self-loop, and e is the independent noise vector with covariance matrix ˜ Ω ∈ diag( R n > 0 ) . The graph is often assumed to be a directed acyclic graph (DAG); in this case, two DAGs are said to be Markov equivalent if they share the same skeleton and v-structures [43], resulting in the same set of conditional independencies. Also, the inverse covariance matrix of x is given by ˜ Θ = ( I -˜ B ) ˜ Ω -1 ( I -˜ B ) ⊤ . We refer readers to Spirtes et al. [39], Glymour et al. [21] for more details and a review of causal discovery.

Score-based method is a major class of causal discovery methods that optimizes a goodness-of-fit measure under a sparsity constraint [21], e.g., BIC [36]. In essence, score-based causal discovery from second-order statistics can often be formulated in the large sample limit as the following optimization problem (the commonly used acyclicity constraint is omitted here and will be clarified subsequently):

By substituting A := ( I -B ) Ω -1 2 into the above formulation, we obtain the ICA formulation with second-order statistics and sparsity constraint introduced in Problem (1). To establish a precise connection between formulations (4) and (1), we present the following theorem which indicates that these formulations can be translated into each other.

Theorem 4 (Equivalent Formulations). Suppose ˜ A = ( I -˜ B ) ˜ Ω -1 2 . Then, we have:

(a) Let ( ˆ B , ˆ Ω ) be a solution to Problem (4) . Then, ˆ A := ( I -ˆ B ) ˆ Ω -1 2 is a solution to Problem (1) .

(b) Let ˆ A be a solution to Problem (1) . Then, there exist matrices ˆ B ∈ R n × n off and ˆ Ω ∈ diag( R n > 0 ) such that ˆ A ∼ ( I -ˆ B ) ˆ Ω -1 2 , and ( ˆ B , ˆ Ω ) is a solution to Problem (4) .

Thus, the formulations of causal discovery and ICA via second-order statistics and sparsity constraint share inherent similarities. The key difference lies in their respective goals-the former aims to estimate the support of ˜ B up to a Markov equivalence class [39], while the latter aims to estimate ˜ A up to signed column permutation. The other difference is that ( I -˜ B ) ˜ Ω -1 ( I -˜ B ) ⊤ represents the inverse covariance matrix ˜ Θ of x in causal discovery, while ˜ A ˜ A ⊤ represents the covariance matrix ˜ Σ of x in ICA.

In addition to establishing the connection between formulations (4) and (1), we show that the assumptions we employ for identifiability of ICA, namely Assumptions 1, 2, and 3, are inherently related to causal discovery. Notably, Assumption 3 has been used in causal discovery [39, 20] to ensure that the conditional independencies in the distribution are entailed by the true directed graph. We now present a result that establishes the connection of Assumptions 1 and 2 with causal discovery.

Theorem 5. Suppose A ∼ ( I -B ) Ω -1 2 for matrices A ∈ R n × n , B ∈ R n × n off , and Ω ∈ diag( R n > 0 ) . Then, A satisfies Assumptions 1 and 2 if and only if B represents a DAG whose Markov equivalence class is a singleton.

In causal discovery, it is rather common to assume that the true directed graph is acyclic and accordingly incorporate an acyclicity constraint to formulation (4). As indicated in Theorem 5 (and Proposition 10 in Appendix D.8), this acyclicity assumption corresponds to Assumption 2 in the context of ICA. Therefore, Theorem 4 can be straightforwardly extended to show the equivalence between formulations (2) and (4) with an additional acyclicity constraint on matrix B . Furthermore, it is worth noting that the mapping from mixing matrix A satisfying Assumption 2 to a DAG is unique, which is straightforwardly implied by Shimizu et al. [38, Appendix A].

Proposition 4 (Shimizu et al. [38]). Suppose matrix A is non-singular and satisfies Assumption 2. Then, there exist unique matrices B ∈ R n × n off and Ω ∈ diag( R n > 0 ) such that A ∼ ( I -B ) Ω -1 2 . Furthermore, matrix B represents a DAG.

Moreover, as indicated in Theorem 5 (and Proposition 11 in Appendix D.8), Assumption 1 implies that the Markov equivalence class of B is a singleton; in this case, the true DAG can be completely identified. In particular, the Markov equivalence class of DAG is a singleton when all edges are either part of a v-structure or required to be oriented to avoid forming new v-structures or cycles [31, 4].

4 Estimation Methods with Second-Order Statistics

Building upon the identifiability result provided in Section 3, we propose two estimation methods that leverage second-order statistics and sparsity. These methods involve solving a continuous constrained optimization problem, which we discuss in detail in this section. First, in Section 4.1, we introduce a novel approach to formulate the search space in Problem (2) that enables the application of continuous optimization techniques. We then describe the proposed estimation methods in Section 4.2. All proofs are provided in Appendix D.

4.1 Characterization of Search Space

The key to achieving the identifiability result presented in Theorem (1) lies in the optimization problem (2), where the search space involves the matrices A that satisfy Assumption 2. Consequently, a crucial question arises: is there an efficient approach for exploring the space of matrices A that satisfy Assumption 2? Inspired by Zheng et al. [48], Wei et al. [45], Zhang et al. [47], we introduce the following function to characterize the search space:

Here, symbol ⊙ denotes the Hadamard product. We then provide the following lemma that establishes the relationship between function g ( A ) and a specific type of permutation, namely simultaneous equal row and column permutation.

Lemma 1. For any matrix A , g ( A ) = 0 if and only if it can be permuted via simultaneous equal row and column permutations to be lower triangular.

Intuitively speaking, if we interpret matrix A as a weighted adjacency matrix of a directed graph, say G , then tr((off( A ) ⊙ off( A )) k ) counts the number of lengthk weighted closed walks in G excluding the self-loops. Therefore, g ( A ) counts the total number of weighted closed walks in G without including self-loops. g ( A ) = 0 then implies that G does not contain any cycle longer than one (i.e., it may contain self-loops). It is known that a directed graph is acyclic if and only if its weighted adjacency matrix can be permuted via simultaneous equal row and column permutations to be strictly lower triangular. In our case, G may contain self-loops, and thus can be permuted to a lower triangular form. This distinction elucidates the difference between our characterization and that introduced by Zheng et al. [48], Zhang et al. [47], i.e., their characterization focuses on matrices that are strictly lower triangular, while ours focuses on lower triangular matrices.

The following proposition sheds light on the connection between g ( A ) and Assumption 2.

Proposition 5. The matrix A satisfies Assumption 2 if and only if there is a matrix ˆ A such that it is a column permutation of A and that g ( ˆ A ) = 0 .

The above proposition indicates that the search for matrices A satisfying Assumption 2 can be effectively conducted by considering the constraint g ( A ) = 0 . Accordingly, we establish an alternative formulation of the identifiability result presented in Section 3.2. In the following section, we will introduce efficient approaches for solving Problem (5) .

Theorem 6 (Alternative Formulation of Identifiability). Suppose that the true mixing matrix ˜ A satisfies Assumptions 1, 2, and 3. Let ˆ A be a solution of the following problem:

Then, we have ˆ A ∼ ˜ A .

Algorithm 1 Decomposition-Based Method

Require: initial penalty coefficient c 1 > 0 ; multiplicative factor β > 1 ; maximum number of iterations k max > 0 ; tolerance ϵ 1 , ϵ 2 > 0 ; initial solution A 0 ; empirical covariance matrix ¯ Σ

1: for k = 1 , 2 , . . . , k max do

3: if ∥ A k A ⊤ k -¯ Σ ∥ 2 F < ϵ 1 and g ( A k ) < ϵ 2 then break

2: Solve A k := arg min A ∈ R n × n ρ ( A ) + c k 2 ∥ AA ⊤ -¯ Σ ∥ 2 F + c k 2 g ( A ) 2 initialized at A k -1

4: Update penalty coefficient c k +1 := βc k

5: Output solution A k

Algorithm 2 Likelihood-Based Method

Require: initial penalty coefficient c 1 > 0 ; multiplicative factor β > 1 ; maximum number of iterations k max > 0 ; tolerance ϵ > 0 ; initial solution A 0 ; empirical covariance matrix ¯ Σ

1: for k = 1 , 2 , . . . , k max do

3: if g ( A k ) < ϵ then break

2: Solve A k := arg min A ∈ R n × n L ( A ; ¯ Σ ) + ρ ( A ) + c k 2 g ( A ) 2 initialized at A k -1

4: Update penalty coefficient c k +1 := βc k

5: Output solution A k

4.2 Estimation Methods

Based on the identifiabiltiy results of Theorems 1 and 6, we propose two estimation methods, called SparseICA , to perform ICA from second-order statistics that leverage sparsity regularization and continuous constrained optimization. To proceed, we define ¯ Σ as the empirical covariance matrix of observed variables x and T as the sample size.

Decomposition-based method. Given the formulation in Eq. (5), we consider the following constrained optimization problem

where ρ ( A ) is a suitable sparsity regularizer, often expressible as ρ ( A ) = ∑ i,j ρ ( a i,j ) . Formulation (5) indicates that one should apply the ℓ 0 regularizer ρ ( A ) = ∥ A ∥ 0 . Alternatively, other possible choices include the ℓ 1 regularizer ρ ( A ) = ∥ A ∥ 1 that supports continuous optimization. Further details regarding our specific choice of sparsity regularizer will be elaborated later in this section. On the other hand, we simply use the empirical covariance matrix ¯ Σ as an estimate of the true covariance matrix ˜ Σ , which is found to work well across different sample sizes in our experiments. One may also adopt a regularized estimator of the form ¯ Σ + η I with a proper choice of η , which may have notable advantage in certain cases [13, 28, 37].

Likelihood-based method. In addition to the decomposition-based method above, we introduce a likelihood-based estimation method formulated by the following constrained optimization problem:

is the negative Gaussian log-likelihood function and ρ ( A ) is a sparsity regularizer. The following result establishes the theoretical guarantee of this likelihood-based method.

Theorem 7 (Likelihood-Based Method). Suppose that the true mixing matrix ˜ A satisfies Assumptions 1, 2, and 3. Let ˆ A be a solution of Problem (7) with sparsity regularizer ρ ( A ) = 0 . 5 ∥ A ∥ 0 log T . Then, we have ˆ A ∼ ˜ A in the large sample limit.

Implementation. Based on Theorems 6 and 7, ideally one should adopt the ℓ 0 regularizer ρ ( A ) = λ ∥ A ∥ 0 and develop an exact discrete search procedure over the support space of matrix A . However, such approach may pose computational challenges in practice. Since the functions L ( A ; ¯ Σ ) and g ( A ) are differentiable, in this work we develop an estimation procedure that leverages efficient continuous optimization techniques. Therefore, some possible choices for ρ ( A ) are ℓ 1 , smoothly

clipped absolute deviation (SCAD) [17], and minimax concave penalty (MCP) [46] regularizers. The ℓ 1 regularizer has been shown to exhibit bias during estimation [17, 10], especially for large coefficients. Here, we adopt the MCP regularizer that is less susceptible to such issue, given by

where λ and α are hyperparameters.

To solve Eqs. (6) and (7), standard constrained optimization methods can be used, such as quadratic penalty method, augmented Lagrangian method, and barrier method [7, 8, 33]. In this work, we adopt the quadratic penalty method that converts each constrained problem into a sequence of unconstrained optimization problems where the constraint violations are increasingly penalized. We describe the full procedure of the decomposition-based and likelihood-based methods based on quadratic penalty method in Algorithms 1 and 2, respectively. The unconstrained problem in each iteration can be solved using different continuous optimization solvers, including first-order methods such as gradient descent and steepest descent, as well as second-order methods such as quasi-Newton methods. In our experiments presented in the subsequent section, we employ L-BFGS [11], a quasi-Newton method, to solve the unconstrained optimization problem.

It is worth noting that the formulations in Eqs. (6) and (7) involve solving nonconvex optimization problems; in practice, the optimization procedure may return stationary points that correspond to suboptimal local solutions. Therefore, we run the method for a number of times and choose the final mixing matrix via model selection. Further details regarding the optimization procedure and implementation are provided in Appendix F.

5 Experiments

To empirically validate our proposed identifiability results, we carry out experiments under various settings. We also conduct ablation studies to verify the necessity of the proposed assumptions and include FastICA [23] as a representative baseline. Specifically, we consider the following methods:

SparseICA : Decomposition-based (Eq. (6)) or likelihood-based (Eq. (7)) method on data where both Assumptions 1 and 2 hold;

Vanilla : Decomposition-based (Eq. (6)) or likelihood-based (Eq. (7)) method without the constraint g ( A ) = 0 , on data where neither Assumption 1 nor Assumption 2 holds;

FastICA-D : FastICA on data where both Assumptions 1 and 2 hold;

FastICA : FastICA on data where neither Assumption 1 nor Assumption 2 holds.

For all experiments, we simulate 10 sources, and generate the supports of the true mixing matrices ˜ A according to the assumptions required by each method above. The nonzero entries of ˜ A are sampled uniformly at random from [ -0 . 8 , -0 . 2] ∪ [0 . 2 , 0 . 8] . We use mean correlation coefficient (MCC) and Amari distance [3] as evaluation metrics, where all results are reported for 10 random trials.

Different sample sizes. We first consider entirely Gaussian sources and different sample sizes. The empirical results of MCC are shown in Figure 1, while those of Amari distance are given in Figure 4 in Appendix G. By comparing SparseICA with Vanilla and FastICA , it is evident that the identification performance is much better across different sample sizes when the required assumptions on the connective structure are satisfied, as validated by Wilcoxon signed-rank test at 5% significance level. Furthermore, the unsatisfactory results of FastICA-D indicate that our estimation methods are also essential for ensuring the quality of the identification, which further validates the proposed identifiability theory. Since FastICA-D performs similarly to FastICA , it suggests that the datagenerating process, while meeting our assumption, may not be inherently simpler to recover without considering specific procedure to handle Gaussian sources. In addition, as expected, the performance of SparseICA improves in terms of both MCC and Amari distance as the sample size increases.

Different ratios of Gaussian sources. We now conduct empirical study to investigate the performance in the presence of Gaussian and non-Gaussian sources. Here, the non-Gaussian sources follow exponential distributions. We consider different ratios of Gaussian sources, which are specifically 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , and 1 . For instance, ratio of 0 . 6 indicates that there are 6 Gaussian sources

Vanilla

FastICA-D

FastICA

30

100

300

1000

3000

10000

Sample Size

0.55

0.60

0.65

0.70

0.75

0.80

MCC

30

100

300

1000

3000

10000

Sample Size

0.4

0.5

0.6

0.7

0.8

MCC

(a) Likelihood-based method.

(b) Decomposition-based method.

Vanilla

FastICA-D

FastICA

0.0

0.2

0.4

0.6

0.8

1.0

Ratio of Gaussian Sources

0.6

0.7

0.8

0.9

1.0

MCC

(a) Likelihood-based method.

0.0

0.2

0.4

0.6

0.8

1.0

Ratio of Gaussian Sources

0.4

0.6

0.8

1.0

MCC

(b) Decomposition-based method.

and 4 non-Gaussian sources. The empirical results of MCC based on 1000 samples are depicted in Figure 1, while those of Amari distance are provided in Figure 4 in Appendix G. One observes that the identification performance SparseICA is rather stable across different ratios of Gaussian sources, which may not be surprising as it leverages only second-order statistics. On the other hand, the performance of FastICA-D and FastICA deteriorates as the ratio of Gaussian sources increases, because it relies on non-Gaussianity of the sources. It is also observed that, in the presence of Gaussian sources, FastICA-D and FastICA may perform well, provided that the ratio (or number) of Gaussian sources is not large. This suggests a potential future direction to integrate our method based on second-order statistics with existing methods that rely on non-Gaussianity, which may better handle both Gaussian and non-Gaussian sources.

6 Conclusion

We develop an identifiability theory of ICA from second-order statistics without relying on nonGaussianity. Specifically, we introduce novel and precise assumptions on the connective structure from sources and observed variables, and show that our proposed assumption of structural variability is strictly weaker than the previous ones. Importantly, we prove that this assumption is one of the necessary conditions for achieving identifiability in the investigated setting. We further propose two estimation methods based on second-order statistics that leverage sparsity regularization. Moreover, we establish a precise connection between our identifiability result of ICA and causal discovery from second-order statistics, which may open up avenues for exploring the interplay between ICA and causal discovery with linear Gaussian SEM. Our theoretical claims have also been empirically validated across different settings. The limitations include the lack of finite sample analysis and broader application of our theory in more real-world tasks, which are worth exploring in future work.

Acknowledgments

The authors would like to thank the anonymous reviewers for helpful comments and suggestions. This project is partially supported by NSF Grant 2229881, the National Institutes of Health (NIH) under Contract R01HL159805, a grant from Apple Inc., a grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.

References

N. Abrahamsen and P. Rigollet. Sparse gaussian ICA. arXiv preprint arXiv:1804.00408 , 2018.

L. Albera, A. Ferreol, P. Chevalier, and P. Comon. ICAR: a tool for blind source separation using fourth-order statistics only. IEEE Transactions on Signal Processing , 53(10):3633-3643, 2005.

S.-i. Amari, A. Cichocki, and H. Yang. A new learning algorithm for blind signal separation. In Advances in Neural Information Processing Systems , 1995.

S. A. Andersson, D. Madigan, and M. D. Perlman. A characterization of Markov equivalence classes for acyclic digraphs. The Annals of Statistics , 25(2):505-541, 1997.

A. Belouchrani, K. Abed-Meraim, J.-F. Cardoso, and E. Moulines. A blind source separation technique using second-order statistics. IEEE Transactions on signal processing , 45(2):434-444, 1997.

R. Benedetti and J.-J. Risler. Real algebraic and semi-algebraic sets . Actualités mathématiques. Hermann, Paris, 1990.

D. P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods . Academic Press, 1982.

D. P. Bertsekas. Nonlinear Programming . Athena Scientific, 2nd edition, 1999.

A. Biton, I. Bernard-Pierrot, Y. Lou, C. Krucker, E. Chapeaublanc, C. Rubio-Pérez, N. LópezBigas, A. Kamoun, Y. Neuzillet, P. Gestraud, L. Grieco, S. Rebouissou, A. de Reyniès, S. Benhamou, T. Lebret, J. Southgate, E. Barillot, Y . Allory, A. Zinovyev, and F. Radvanyi. Independent component analysis uncovers the landscape of the bladder tumor transcriptome and reveals insights into luminal and basal subtypes. Cell Reports , 9(4):1235-1245, Nov 2014.

P. Breheny and J. Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The Annals of Applied Statistics , 5(1):232-253, 2011.

R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing , 16(5):1190-1208, 1995.

P. Comon. Independent component analysis, a new concept? Signal processing , 36(3):287-314, 1994.

M. J. Daniels and R. E. Kass. Shrinkage estimators for covariance matrices. Biometrics , 57(4): 1173-1184, 2001.

M. Drton. Algebraic problems in structural equation modeling. Advanced Studies in Pure Mathematics , 77, 2018.

M. Drton, B. Sturmfels, and S. Sullivant. Algebraic factor analysis: Tetrads, pentads and beyond. Probability Theory and Related Fields , 138, 09 2005.

M. Drton, B. Sturmfels, and S. Sullivant. Lectures on Algebraic Statistics , volume 39 of Oberwolfach Seminars . Springer, 2009.

J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association , 96(456):1348-1360, 2001.

J. Friedman, T. Hastie, and R. Tibshirani. Sparse inverse covariance estimation with the graphical Lasso. Biostatistics , 9:432-41, 2008.

D. Geiger, D. Heckerman, H. King, and C. Meek. Stratified exponential families: Graphical models and model selection. The Annals of Statistics , 29(2):505-529, 2001.

A. Ghassami, A. Yang, N. Kiyavash, and K. Zhang. Characterizing distribution equivalence and structure learning for cyclic and acyclic directed graphs. In International Conference on Machine Learning , 2020.

C. Glymour, K. Zhang, and P. Spirtes. Review of causal discovery methods based on graphical models. Frontiers in Genetics , 10, 2019.

A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems , 2016.

A. Hyvärinen and E. Oja. A fast fixed-point algorithm for independent component analysis. Neural Computation , 9(7):1483-1492, 1997.

A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural networks , 13(4-5):411-430, 2000.

A. Hyvärinen, J. Karhunen, and E. Oja. Independent Component Analysis . John Wiley & Sons, Inc, 2001.

A. Hyvärinen, H. Sasaki, and R. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics , 2019.

T.-P. Jung, S. Makeig, M. McKeown, A. Bell, T.-W. Lee, and T. Sejnowski. Imaging brain dynamics using independent component analysis. Proceedings of the IEEE , 89(7):1107-1122, 2001.

O. Ledoit and M. Wolf. A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis , 88(2):365-411, 2004.

M. Lynch, B. Walsh, et al. Genetics and analysis of quantitative traits , volume 1. Sinauer Sunderland, MA, 1998.

K. Matsuoka, M. Ohoya, and M. Kawamoto. A neural net for blind separation of nonstationary signals. Neural Networks , 8:411-419, 1995.

C. Meek. Causal inference and causal explanation with background knowledge. In Conference on Uncertainty in Artificial Intelligence , 1995.

N. Moulin, F. Gresselin, B. Dardaillon, and Z. Thomas. River temperature analysis with a new way of using independant component analysis. Frontiers in Earth Science , 10, 12 2022.

J. Nocedal and S. J. Wright. Numerical optimization . Springer series in operations research and financial engineering. Springer, 2nd edition, 2006.

H. W. Ott. Noise Reduction Techniques in Electronic Systems . New York : Wiley, 1988.

D.-T. Pham and J.-F. Cardoso. Blind separation of instantaneous mixtures of nonstationary sources. IEEE Transactions on signal processing , 49(9):1837-1848, 2001.

G. Schwarz. Estimating the dimension of a model. The Annals of Statistics , 6(2):461-464, 1978.

J. Schäfer and K. Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical applications in genetics and molecular biology , 4, 02 2005.

S. Shimizu, P. O. Hoyer, A. Hyvärinen, and A. Kerminen. A linear non-Gaussian acyclic model for causal discovery. Journal of Machine Learning Research , 7(Oct):2003-2030, 2006.

P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search . MIT press, 2nd edition, 2001.

S. Sullivant. Algebraic geometry of gaussian Bayesian networks. Advances in Applied Mathematics , 40(4):482-513, 2008. ISSN 0196-8858.

A. E. Teschendorff, M. Journée, P. A. Absil, R. Sepulchre, and C. Caldas. Elucidating the altered transcriptional programs in breast cancer using independent component analysis. PLoS Comput. Biol. , 3(8):1539-1554, 2007.

L. N. Trefethen and D. Bau. Numerical linear algebra , volume 50. SIAM, 1997.

T. Verma and J. Pearl. Equivalence and synthesis of causal models. In Conference on Uncertainty in Artificial Intelligence , 1990.

P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. Jarrod Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. Carey, ˙ I. Polat, Y. Feng, E. W. Moore, J. Vand erPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and S. . . Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods , 17: 261-272, 2020.

D. Wei, T. Gao, and Y. Yu. DAGs with no fears: A closer look at continuous optimization for learning Bayesian networks. In Advances in Neural Information Processing Systems , 2020.

C.-H. Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics , 38(2):894-942, 2010.

Z. Zhang, I. Ng, D. Gong, Y. Liu, E. M. Abbasnejad, M. Gong, K. Zhang, and J. Q. Shi. Truncated matrix power iteration for differentiable DAG learning. In Advances in Neural Information Processing Systems , 2022.

X. Zheng, B. Aragam, P. Ravikumar, and E. P. Xing. DAGs with NO TEARS: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems , 2018.

Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems , 2022.

Appendices

Table of Contents

A Preliminaries on Support Rotation

Before presenting the proofs of the theoretical results, we review the definition of support rotation and its potential effects on a support matrix, given by Ghassami et al. [20]. Note that the definition and remark below are quoted from Ghassami et al. [20, Section 3.1], with only minor modifications.

Definition 3 (Support Rotation [20]). The support rotation, denoted as R ( i, j, k ) , is a transformation that modifies a support matrix ξ by applying a Givens rotation in the ( j, k ) plane, setting the element ξ i,j to zero. The outcome of is the support matrix of Q G ( j, k, tan -1 ( -q i,j /q i,k ) ) , where Q ∈ arg max Q ′ ∣ ∣ ∣ supp ( Q ′ G ( j, k, tan -1 ( -q ′ i,j /q ′ i,k )))∣ ∣ ∣ such that the support matrix of Q ′ is ξ .

Note that G ( j, k, tan -1 ( -q ′ ij /q ′ i,k )) is the Givens rotation in the ( j, k ) plane that makes the q ′ i,j entry zero.

Remark 3 (Ghassami et al. [20]). The effects of applying a support rotation R ( i, j, k ) can be categorized into the following four cases:

Reduction: If ξ i,j = ξ i,k = × and ξ l,j = ξ l,k for all l ∈ [ p ] \{ i } , then only ξ i,j becomes zero.

Reversible acute rotation: If ξ i,j = ξ i,k = × and there exists a row i ′ such that the j -th and k -th columns differ only in that row, then ξ i,j becomes zero and both ξ i ′ ,j and ξ i ′ ,k become × .

Irreversible acute rotation: If ξ i,j = ξ i,k = × and the j -th and k -th columns differ in at least two rows, then ξ i,j becomes zero and all entries on the j -th and k -th columns become × on the rows on which they differed.

Column swap: If ξ i,j = × and ξ i,k = 0 , then columns j and k are swapped.

B Proofs of Useful Lemmas

We provide several lemmas that will be useful for subsequent proofs.

B.1 Proof of Lemma 2

Lemma 2 (Nonzero Diagonal Entries). For any non-singular matrix A , there exists a permutation matrix P such that the diagonal entries of AP are nonzero.

Proof. We prove it by contradiction. Suppose that there always exists a zero diagonal entry in every column permutation.

We first represent the determinant of the matrix A as its Leibniz formula:

where S n is the set of n -permutations. Because for every permutation, there always exists a diagonal entry with value zero, we have

Thus, it follows that det( A ) = 0 . This indicates that the matrix A is singular, which leads to a contradiction.

B.2 Proof of Lemma 3

Lemma 3 (Non-Singular Solution). Let ˜ A be a non-singular matrix. Suppose ˆ A is a matrix such that ˆ A ˆ A ⊤ = ˜ A ˜ A ⊤ . Then, ˆ A is non-singular.

̸

Proof. A matrix is non-singular (or invertible) if and only if its determinant is nonzero. Therefore, we need to show that det ( ˆ A ) = 0 .

From the given, we have ˆ A ˆ A ⊤ = ˜ A ˜ A ⊤ . Taking determinants on both sides, we get

Since ˆ A and ˜ A are square matrices, we can simplify both sides:

̸

Since ˜ A is non-singular, we know that det( ˜ A ) = 0 . Therefore, det( ˆ A ) 2 = 0 . This implies that det( ˆ A ) = 0 . Thus, ˆ A is non-singular.

̸

̸

B.3 Proof of Lemma 5

Lemma 4. Let A be a matrix with all diagonal entries being nonzero. Then, there exists matrix B ∈ R n × n off and Ω ∈ diag( R n > 0 ) such that A = ( I -B ) Ω -1 2 D , where D is a diagonal matrix with diagonal entries being ± 1 .

Proof. Since the diagonal entries of A are nonzero, we can construct a diagonal matrix D such that the diagonal entries of AD are positive, by defining d i,i = 1 if a i,i > 0 and d i,i = -1 if a i,i < 0 . Let Ω be a diagonal matrix of the same size as matrix A , where ( Ω ) i,i = 1 / ( AD ) 2 i,i , Also, let B := I -ADΩ 1 2 . Since the diagonal entries of matrix ADΩ 1 2 are ones, the diagonal entries of matrix B are zeros. Therefore, we have B ∈ R n × n off , Ω ∈ diag( R n > 0 ) , and A = ( I -B ) Ω -1 2 D , where D is a diagonal matrix with entries being ± 1 .

Lemma 5. Let A be a non-singular matrix. Then, there exists matrix B ∈ R n × n off and Ω ∈ diag( R n > 0 ) such that A ∼ ( I -B ) Ω -1 2 .

Proof. Since matrix A is non-singular, by Lemma 2, there exists a permutation matrix P such that the diagonal entries of AP are nonzero. By Lemma 4, there exists matrix B ∈ R n × n off and Ω ∈ diag( R n > 0 ) such that AP = ( I -B ) Ω -1 2 D , where D is a diagonal matrix with diagonal entries being ± 1 . Clearly, we have A ∼ ( I -B ) Ω -1 2 .

B.4 Proof of Lemma 6

Lemma 6. Suppose A ∼ ( I -B ) Ω -1 2 for matrices A ∈ R n × n , B ∈ R n × n off , and Ω ∈ diag( R n > 0 ) . Then, we have ∥ A ∥ 0 = ∥ B ∥ 0 + n .

Proof. First notice that the diagonal entries of B are zeros, which implies

Since Ω -1 2 is a diagonal matrix, right multiplication of Ω -1 2 amounts to rescaling the columns of I -B , and does not affect its support. Therefore, we have

Since A ∼ ( I -B ) Ω -1 2 , matrices A and ( I -B ) Ω -1 2 differ only in signed column permutations. Therefore, their number of nonzero entries are the same, i.e.,

B.5 Proof of Lemma 8

We first provide the following lemma that will be used to prove Lemma 8.

Lemma 7. Let P 1 and P 2 be two permutation matrices. Then, P ⊤ 1 P 2 is lower triangular if and only if P 1 = P 2 .

Proof. The 'if part' is clear because permutation matrices are orthogonal matrices, and thus P ⊤ 1 P 1 = I is lower triangular. It remains to prove the 'only if part'. Since P ⊤ 1 P 2 is also a permutation matrix, this matrix being lower triangular implies that it is an identity matrix, because identity matrix is the only permutation matrix that is lower triangular. We then have P ⊤ 1 P 2 = I , which implies

We now provide the proof of Lemma 8.

Lemma 8. Given matrix B ∈ R n × n off , if I -B satisfies Assumption 2, then I -B is non-singular. Proof. Since I -B satisfies Assumption 2, there exist permutation matrices P 1 and P 2 such that

̸

is lower triangular. For all i, j ∈ [ n ] such that ( P ⊤ 1 P 2 ) i,j = 0 , we have ( P ⊤ 1 BP 2 ) i,j = 0 , which implies that P ⊤ 1 P 2 must be lower triangular, because otherwise, P ⊤ 1 P 2 -P ⊤ 1 BP 2 cannot be lower

triangular. By Lemma 7, we then have P 1 = P 2 . This indicates that I -P ⊤ 1 BP 1 is lower triangular, with diagonal entries equal to one. Therefore, we have

which implies

Since the determinant of I -B is nonzero, it is non-singular.

C Proof of Theorem 1

We first describe the notion of covariance equivalence that is needed for the proof of Theorem 1. Specifically, if two support matrices ξ 1 and ξ 2 entail the same set of covariance matrices, i.e., Σ ( ξ 1 ) = Σ ( ξ 2 ) , they are said to be covariance equivalent . This means that for any combination of parameter values in ξ 1 , there exists a corresponding set of parameter values in ξ 2 that leads to the same covariance matrix, and vice versa. Furthermore, two support matrices are covariance equivalent if and only if they lead to the same set of semialgebraic constraints on the covariance matrices.

We now give the proof of Theorem 1. The proof makes use of Proposition 2, Proposition 6, Proposition 7, and Corollary 1, which are provided in Appendices C.1, C.2, C.3, and C.4, respectively. Note that the proof is partly inspired by that of Ghassami et al. [20, Theorem 3].

Theorem 1 (Identifiability with Sparsity). Suppose that the true mixing matrix ˜ A satisfies Assumptions 1, 2, and 3. Let ˆ A be a solution of the following problem:

Then, we have ˆ A ∼ ˜ A .

Proof. Let ˆ A be a solution of Problem (2). This implies ˆ A ˆ A ⊤ = ˜ Σ = ˜ A ˜ A ⊤ and that ˆ A satisfies Assumption 2. Since ˜ A is non-singular, by Lemma 3, matrix ˆ A is non-singular.

Since ˆ A can entail the covariance matrix ˜ Σ , we have ˜ Σ ∈ Σ ( ξ ˆ A ) , which indicates that ˜ Σ contains all semialgebraic constraints of ξ ˆ A . Under Assumption 3, we have

The sparsity term in the objective function implies

because otherwise ˆ A will never be a solution of Problem (2).

We now show by contradiction that H ( ξ ˆ A ) ̸⊂ H ( ξ ˜ A ) . Suppose H ( ξ ˆ A ) ⊂ H ( ξ ˜ A ) , which indicates dim( Σ ( ξ ˆ A )) > dim( Σ ( ξ ˜ A )) . Since the support matrices ξ ˆ A and ξ ˜ A satisfy Assumption 2, Proposition 2 implies ∥ ξ ˆ A ∥ 0 > ∥ ξ ˜ A ∥ 0 , which is contradictory with Inequality (9). This implies

By Eqs. (8) and (10), we obtain H ( ξ ˆ A ) = H ( ξ ˜ A ) , which, by Proposition 7, implies that the support matrices ξ ˆ A and ξ ˜ A are covariance equivalent, because they satisfy Assumption 2. By Proposition 6, we conclude that the columns of ξ ˆ A are a permutation of those of ξ ˜ A .

Recall that matrices ˜ A and ˆ A are non-singular, and entail the same covariance matrix ˜ Σ . Since the columns of ξ ˆ A are a permutation of those of ξ ˜ A , by Corollary 1, we have ˆ A ∼ ˜ A .

C.1 Dimension of Covariance Set

We first define the Jacobian matrix of Σ = AA ⊤ w.r.t. the free parameters of A as ∂ Σ ∂ A , which is a n 2 ×∥ A ∥ 0 matrix, where the rows are indexed by ( i, j ) ∈ [ n ] × [ n ] , and columns are indexed by

( k, l ) ∈ supp( A ) . That is, for ( i, j ) ∈ [ n ] × [ n ] and ( k, l ) ∈ supp( A ) , we have

̸

̸

Given a matrix M , we denote by off( M ) and on( M ) the off-diagonal and diagonal entries of M , respectively. With a slight abuse of notation, we rewrite the above Jacobian matrix as the following form that consists of four submatrices, by permuting the corresponding columns and rows:

Each submatrix above represents the Jacobian matrix for different parts (i.e., off-diagonal and diagonal entries) of matrix Σ taken w.r.t. different free parameters (i.e., off-diagonal and diagonal free parameters) in matrix A . Note that column and row permutations do not affect the rank of the matrix. Since we are primarily interested in the rank of the above Jacobian matrix and its submatrices, we use the same notation to refer to different column and/or row permutations of the corresponding Jacobian matrix, depending on the context.

Lemma 9. Let A be a lower triangular matrix. Then, we have

Proof. Let d be the number of diagonal free parameters in matrix A . This indicates that off( A ) and on( A ) contain ∥ A ∥ 0 -d and d free parameters, respectively.

By specializing A = I n and using Eq. (11), we have

where the last equality is obtained after row permutations of the corresponding matrix. Also, since matrix A is lower triangular, each nonzero entry of ∂ off( Σ ) ∂ off( A ) ∣ ∣ A = I n corresponds to either

In this case, each column of ∂ off( Σ ) ∂ off( A ) ∣ ∣ A = I n contains precisely two nonzero entries, while each row either contains precisely one nonzero entry, or does not contain any nonzero entry. Therefore, ∂ off( Σ ) ∂ off( A ) ∣ ∣ A = I n can be rewritten after row permutations as

Substituting Eq. (13) into Eq. (12), we have

which yields which, with Eq. (14), implies

̸

̸

Proposition 2 (Dimension of Covariance Set). Let ξ be a support matrix that satisfies Assumption 2. Then, its covariance set has a dimension of ∥ ξ ∥ 0 , i.e., dim( Σ ( ξ )) = ∥ ξ ∥ 0 .

Proof. Since ξ satisfies Assumption 2, it can be permuted by column and row permutations to be lower triangular; in this case, the resulting covariance matrix and the original covariance matrix differ in equal row and column permutations. Note that the dimension of the covariance set remains the same after simultaneous equal row and column permutations of the covariance matrices. Therefore, it suffices to consider the case where ξ is lower triangular, and show that it has a dimension of ∥ ξ ∥ 0 .

As indicated by Geiger et al. [19, Theorem 10], the dimension of dim( Σ ( ξ )) equals the maximum rank of the corresponding Jacobian matrix. In this case, it suffices to consider the columns of the Jacobian matrix that correspond to the nonzero entries of support matrix ξ , i.e., the free parameters of matrix A , which we denote by ∂ Σ ∂ A . By Lemma 9, when A = I n , the Jacobian matrix ∂ Σ ∂ A has full column rank that is equal to ∥ ξ ∥ 0 . Therefore, the dimension of the covariance set is ∥ ξ ∥ 0 .

C.2 Covariance Equivalence and Column Permutation of Support

We now state a result that is adapted from Ghassami et al. [20, Proposition 5] to the context of ICA. In our proof of Theorem 1, only the 'only if part' of the following result is used.

Proposition 6 (Ghassami et al. [20, Proposition 5]). Consider two support matrices ξ 1 and ξ 2 . If every pair of columns of ξ 1 differ in more than one entry, then ξ 1 and ξ 2 are covariance equivalent if and only if the columns of ξ 2 are a permutation of columns of ξ 1 .

C.3 Equality Constraints and Covariance Equivalence

In this section, we show how Assumption 2 allows one to go from two support matrices ξ 1 and ξ 2 having the same equality constraints to covariance equivalence.

Following Ghassami et al. [20], we first denote the covariance set of a directed graph G by

where B G is the adjacency matrix of G . With a slight abuse of notation, we denote by H ( G ) the set of the equality constraints imposed by G on the resulting matrix ( I -B ) Ω -1 ( I -B ) ⊤ .

Lemma 10. Let ξ be a non-singular support matrix that satisfies Assumption 2. 2 Then, there exists a DAG G such that Σ ( ξ ) = Θ ( G ) .

Proof. Since ξ is non-singular, by Lemma 2, it can be mapped via column permutations to another support matrix ξ ′ with diagonal entries being nonzero. Since ξ satisfies Assumption 2, Proposition 10 implies that ξ ′ represents a DAG, say G , where the support of its adjacency matrix is supp( B G ) = supp(off( ξ ′ )) . Since column permutations of support matrices do not affect the resulting covariance matrices, we have Σ ( ξ ) = Σ ( ξ ′ ) . Therefore, it suffices to prove Σ ( ξ ′ ) = Θ ( G ) .

Since ξ ′ is a support matrix with diagonal entries being nonzero, we have

We consider both parts of the statements.

Proof of Θ ( G ) ⊆ Σ + ( ξ ′ ) :

Suppose M ∈ Θ ( G ) . By definition of Θ ( G ) , there exist B ∈ R n × n off and Ω ∈ diag( R n > 0 ) such that

Let A := ( I -B ) Ω -1 2 , which, with Eq. (16), implies AA ⊤ = M . Recall that G is a DAG, which indicates B also represents a DAG, Therefore, I -B , and thus A , are non-singular. Since right multiplication of Ω -1 2 does not affect the support of I -B , by Eqs. (15) and (16), we have

Therefore, we have supp( A ) ⊆ supp( ξ ′ ) , which, with the non-singularity of matrix A , implies M = AA ⊤ ∈ Σ ( ξ ′ ) .

Proof of Σ ( ξ ′ ) ⊆ Θ ( G ) :

Suppose M ∈ Σ ( ξ ′ ) . By definition of Σ ( ξ ′ ) , there exists non-singular matrix A ∈ R n × n such that

Since matrix A is non-singular, all of its diagonal entries must be nonzero, because otherwise the corresponding determinant will be zero, which contradicts its non-singularity. By Lemma 4, there exists matrix B ∈ R n × n off and Ω ∈ diag( R n > 0 ) such that A = ( I -B ) Ω -1 2 D , where D is a diagonal matrix with diagonal entries being ± 1 . By Eq. (17), we have ( I -B ) Ω -1 ( I -B ) ⊤ = M . Since right multiplication of Ω -1 2 D does not affect the support of I -B , by Eqs. (15) and (17), we have

Note that supp( I ) and supp( B ) are disjoint. Furthermore, supp( I ) and supp(off( ξ ′ )) are disjoint. Therefore, we have supp( B ) ⊆ supp(off( ξ ′ )) and thus M = ( I -B ) Ω -1 ( I -B ) ⊤ ∈ Θ ( G ) . Proposition 7. Let ξ 1 and ξ 2 be non-singular support matrices that satisfy Assumption 2. If they have the same set of equality constraints, i.e., H ( ξ 2 ) = H ( ξ 2 ) , then they are covariance equivalent. Proof. Since matrices ξ 1 and ξ 2 are non-singular and satisfy Assumption 2, by Lemma 10, there exist DAGs G 1 and G 2 such that

̸

We now provide a proof by contrapositive. Suppose that ξ 1 and ξ 2 are not covariance equivalent, i.e., Σ ( ξ 1 ) = Σ ( ξ 2 ) . By Eq. (18), we have Θ ( G 1 ) = Θ ( G 2 ) , i.e., DAGs G 1 and G 2 are not covariance equivalent. By Ghassami et al. [20, Proposition 1], DAGs G 1 and G 2 are not Markov equivalent, which indicates that they do not have the same skeleton and v-structures [43]. This implies that they lead to different sets of conditional independence constraints, which in this case correspond to different sets of polynomial equality constraints [39], i.e., H ( G 1 ) = H ( G 2 ) . By Eq. (19), we have H ( ξ 1 ) = H ( ξ 2 ) .

which imply

̸

̸

C.4 Identifiability of Parameters from Support

In the following, we provide a result that establish the identifiability of the parameters in mixing matrix from its support.

Proposition 8. Consider two non-singular matrices A 1 and A 2 that satisfy Assumption 2 and that entail the same covariance matrix, i.e., Σ = A 1 A ⊤ 1 = A 2 A ⊤ 2 . If matrices A 1 and A 2 have the same support, then they differ only in sign changes of columns.

Proof. Since matrix A 1 satisfies Assumption 2, there exist permutation matrices P 1 and P 2 such that P ⊤ 1 A 1 P 2 is lower triangular. Clearly, P ⊤ 1 A 2 P 2 is also lower triangular, because matrices A 1 and A 2 have the same support. Since matrices P ⊤ 1 A 1 P 2 and P ⊤ 1 A 2 P 2 are non-singular, all diagonal entries of these two matrices must be nonzero, because otherwise the corresponding determinant will be zero, which contradict their non-singularity. Let D 1 and D 2 be diagonal matrices with diagonal entries being ± 1 such that the diagonal entries of P ⊤ 1 A 1 D 1 P 2 and P ⊤ 1 A 2 D 2 P 2 are positive. (The procedure for constructing such diagonal matrices D 1 and D 2 is straightforward and omitted here.)

Furthermore, we have

Since matrix A 1 is non-singular, matrices Σ , and thus P ⊤ 1 ΣP 1 , are symmetric positive definite. Here, Eq. (20) can be viewed as the Cholesky decomposition of P ⊤ 1 ΣP 1 , where P ⊤ 1 A 1 D 1 P 2 and P ⊤ 1 A 2 D 2 P 2 are the Cholesky factors. Recall that they are lower triangular matrices with all diagonal entries being positive; in this case, it is known that such Cholesky factor is unique [42]. Therefore, we have which implies

Since D 2 D 1 is a diagonal matrix with diagonal entries being ± 1 , we conclude that matrices A 1 and A 2 differ only in sign changes of columns.

̸

Corollary 1 (Identifiability of Parameters from Support). Consider two non-singular matrices A 1 and A 2 that satisfy Assumption 2 and that entail the same covariance matrix, i.e., Σ = A 1 A ⊤ 1 = A 2 A ⊤ 2 . If the columns of ξ A 1 are a permutation of those of ξ A 2 , then we have A 1 ∼ A 2 .

Proof. Suppose by contradiction that A 1 ̸∼ A 2 . This implies that, for every permutation matrix P such that ξ A 1 = ξ A 2 P , matrices A 1 and A 2 P differ in more than sign changes of columns. By Proposition 8, this cannot happen.

D Proofs of Other Results

D.1 Proof of Example 1

Example 1 (Semialgebraic Constraints). Consider support matrices

The equality constraints imposed by ξ 1 include

while the inequality constraints imposed by ξ 2 include

Proof. We first consider matrix A 1 with the support ξ 1 . That is, matrix A 1 is of the form

The entries of the resulting covariance matrix Σ = A 1 A ⊤ 1 are then given by

We now consider matrix A 2 with the support ξ 2 . That is, matrix A 2 is of the form

The entries of the resulting covariance matrix Σ = A 2 A ⊤ 2 are then given by

Suppose we fix the value of a 3 , 2 . This leads to

which imply which can be rewritten as

(Σ 1 , 1 Σ 2 , 2 -Σ 2 1 , 2 ) a 4 3 , 2 +( -Σ 1 , 1 Σ 2 , 2 Σ 3 , 3 -Σ 1 , 1 Σ 2 2 , 3 +Σ 2 , 2 Σ 2 1 , 3 +Σ 3 , 3 Σ 2 1 , 2 ) a 2 3 , 2 +(Σ 1 , 1 Σ 3 , 3 Σ 2 2 , 3 -Σ 2 1 , 3 Σ 2 2 , 3 ) = 0 . Since the value of a 3 , 2 is a real number, we have

D.2 Proof of Proposition 1

Proposition 1. If the true mixing matrix ˜ A does not satisfy Assumption 1, then there exists a solution ˆ A to Problem (1) such that ˆ A ̸∼ ˜ A .

̸

Proof. Since the true mixing matrix ˜ A does not satisfy Assumption 1, there exist i, j ∈ [ n ] , i = j such that

This leads to the following two cases:

Case 1: | supp(˜ a i ) ∪ supp(˜ a j ) | - | supp(˜ a i ) ∩ supp(˜ a j ) | = 1 . In this case, since the mixing matrix is of full column rank, there must exist a k ∈ [ n ] such that ( ξ ˜ A ) k,i = ( ξ ˜ A ) k,j = × . Thus, we can always apply a reversible acute rotation (see Remark 3) to the i -th and j -th columns of matrix ˜ A . This operation leads to another matrix ¨ A with ∥ ¨ A ∥ 0 = ∥ ˜ A ∥ 0 and ¨ A ¨ A ⊤ = ˜ A ˜ A ⊤ . In the reversible acute rotation, we can set either ( ξ ˜ A ) k,i or ( ξ ˜ A ) k,j to 0 . This implies ∥ ¨ a k, : ∥ 0 < ∥ ˜ a k, : ∥ 0 , and therefore ¨ A ̸∼ ˜ A . Now, suppose that the true mixing matrix ˜ A is not a solution to Problem (1). Clearly, there must exist a solution ˆ A to Problem (1) such that ∥ ˆ A ∥ 0 < ∥ ˜ A ∥ 0 , which indicates ˆ A ̸∼ ˜ A . It remains to consider the case where matrix ˜ A is a solution to Problem (1). In this case, since ∥ ¨ A ∥ 0 = ∥ ˜ A ∥ 0 , matrix ¨ A is also a solution to Problem (1), and we have shown that ¨ A ̸∼ ˜ A .

Case 2: | supp(˜ a i ) ∪ supp(˜ a j ) | - | supp(˜ a i ) ∩ supp(˜ a j ) | = 0 . In this case, we can always apply a reduction (see Remark 3) to the i -th and j -th columns of matrix ˜ A . This operation leads to another matrix ¨ A with ∥ ¨ A ∥ 0 < ∥ ˜ A ∥ 0 and ¨ A ¨ A ⊤ = ˜ A ˜ A ⊤ . Therefore, there must exist a solution ˆ A to Problem (1) such that ∥ ˆ A ∥ 0 ≤ ∥ ¨ A ∥ 0 < ∥ ˜ A ∥ 0 , which indicates ˆ A ̸∼ ˜ A .

In either case, there exists a solution to Problem (1) whose columns are not signed permutations of the columns of ˜ A .

D.3 Proof of Example 2

Example 2. If the connective structure G A of mixing matrix A is a polytree, then matrix A satisfies Assumption 2.

Proof. Suppose that the matrix A does not satisfy Assumption 2. This means that there does not exist permutation matrices P 1 and P 2 such that P ⊤ 1 AP 2 is lower triangular, which implies that, in the connective structure G A , there exists a path that alternates between source and observed variable nodes, i.e., a sequence of nodes { s i 1 , x j 1 , s i 2 , x j 2 , ..., s i k , x j k , s i 1 } where each pair ( s i t , x j t ) corresponds to a nonzero entry a j t ,i t in A , for t = 1 , . . . , k . Thus, by replacing the directed edges on this path with undirected edges, we obtain a cycle. Therefore, G A cannot be a polytree.

D.4 Proof of Proposition 3

The proof of the following proposition is adapted from that of Ghassami et al. [20, Proposition 8].

Proposition 3 (Generic Property). Suppose that the nonzero coefficients of matrix A are randomly drawn from a distribution that is absolutely continuous with respect to Lebesgue measure. Then, matrix A satisfies Assumption 3 with probability one.

Proof. Let ϕ be the set of possible equality constraints of any covariance matrix with the same size as Σ , which is a finite set because the number of variables is finite. Consider a matrix ˆ A with the same support as A (i.e., ξ ˆ A = ξ A ) that violates Assumption 3, where the corresponding covariance matrix is ˆ Σ . To violate Assumption 3, ˆ Σ has to satisfy an equality constraint κ ∈ ϕ \ H ( ξ A ) . Thus, the set of possible matrices with the same support as A that violate Assumption 3 is a subset of

By the definition of equality constraint, each set in the union above has zero Lebesgue measure, and thus the finite union above also has zero Lebesgue measure. This implies that the set of possible

matrices with the same support as A that violate Assumption 3 has zero Lebesgue measure. Therefore, Assumption 3 is satisfied with probability one.

D.5 Proof of Theorem 2

We first prove the following proposition that will be used to prove both Theorems 2 and 3.

Proposition 9. If mixing matrix A satisfies Assumption 6, then it satisfies Assumption 1.

̸

Proof. We provide a proof by contrapositive. Suppose A does not satisfy Assumption 1. This means that there exist some i, j ∈ [ n ] with i = j such that

The difference can be either 0 or 1 .

Case 1: | supp( a i ) ∪ supp( a j ) | - | supp( a i ) ∩ supp( a j ) | = 0 . This implies supp( a i ) = supp( a j ) , i.e., the supports of a i and a j are identical. In this case, Assumption 6 is clearly violated, as supp( a i ) is a subset of supp( a j ) and vice versa.

Case 2: | supp( a i ) ∪ supp( a j ) | - | supp( a i ) ∩ supp( a j ) | = 1 . This implies that one of the columns is a proper subset of the other, meaning either supp( a i ) ⊂ supp( a j ) or supp( a j ) ⊂ supp( a i ) . Again, this violates Assumption 6.

Hence, in either case, if A does not satisfy Assumption 1, then it does not satisfy Assumption 6.

We now prove the following theorem.

Theorem 2. For mixing matrix A , we have the following chain of chain of implications:

Furthermore, there exists a matrix A satisfying Assumption 1 that does not satisfy Assumption 4. Proof. We first prove Assumption 4 = ⇒ Assumption 6.

̸

In Assumption 4, Eq. (3) is assumed to be satsified for all I ⊆ [ n ] where |I| > 1 . Thus, in order to prove that Assumption 4 implies Assumption 6, it is sufficient to only consider the case where |I| = 2 . That is, for every i, j ∈ [ n ] and i = j , we have

where we set i as the target index without loss of generality.

Because rank(overlap( A { i,j } )) ≥ 1 , we have

Suppose | supp( a i ) | ≥ | supp( a j ) | . Eq. (21) implies that supp( a j ) is not a subset of supp( a i ) . Similarly, if | supp( a j ) | ≥ | supp( a i ) | , we could also show that supp( a i ) is not a subset of supp( a j ) . Thus, Assumption 6 is satisfied.

According to Proposition 9, we have Assumption 6 = ⇒ Assumption 1, which completes the proof of the first part, i.e., Assumption 4 = ⇒ Assumption 6 = ⇒ Assumption 1.

We now provide an example of mixing matrix satisfying the Assumption 1 that does not satisfy Assumption 4. Suppose the mixing matrix ˜ A has a support as follows:

Clearly, Assumption 1 is satisfied since every pair of columns differ on more than one entry. However, for k = 1 and I = { 1 , 2 } , we have

Since | supp( a k ) | = | supp( a 1 ) | = 3 , it is not possible for Eq. (3) to hold for ˜ A when k = 1 and I = { 1 , 2 } . Thus, Assumption 4 is violated. The proof of part (b) is finished.

D.6 Proof of Theorem 3

Theorem 3. For mixing matrix A , we have the following chain of chain of implications:

Furthermore, there exists a matrix A satisfying Assumption 1 that does not satisfy Assumption 5.

Proof. We first prove the contrapositive of Assumption 5 = ⇒ Assumption 6. Suppose that Assumption 6 does not hold. This means that there exist distinct indices i, j ∈ [ n ] , such that supp( a i ) is a subset of supp( a j ) .

Now, for any set of row indices I ⊂ [ n ] , consider the intersection over the supports of all rows j ∈ I , denoted by ⋂ j ∈I supp( a j, : ) . Because supp( a i ) is a subset of supp( a j ) , it is clear that i cannot be the only element in this intersection. Therefore, it is impossible to satisfy ⋂ j ∈I supp( a j, : ) = { i } for any choice of I , which indicates that Assumption 5 is violated.

According to Proposition 9, we have Assumption 6 = ⇒ Assumption 1, which completes the proof of the first part, i.e., Assumption 5 = ⇒ Assumption 6 = ⇒ Assumption 1.

We now provide an example of mixing matrix satisfying the Assumption 1 that does not satisfy Assumption 5. Suppose the mixing matrix ˜ A has a support as follows:

Clearly, Assumption 1 is satisfied since the supports of each pair of columns differ on more than one entry. However, Assumption 5 is violated since there does not exist any set of rows such that the intersection of their nonzero indices is 2 or 3 .

D.7 Proof of Theorem 4

Theorem 4 (Equivalent Formulations). Suppose ˜ A = ( I -˜ B ) ˜ Ω -1 2 . Then, we have:

(a) Let ( ˆ B , ˆ Ω ) be a solution to Problem (4) . Then, ˆ A := ( I -ˆ B ) ˆ Ω -1 2 is a solution to Problem (1) .

(b) Let ˆ A be a solution to Problem (1) . Then, there exist matrices ˆ B ∈ R n × n off and ˆ Ω ∈ diag( R n > 0 ) such that ˆ A ∼ ( I -ˆ B ) ˆ Ω -1 2 , and ( ˆ B , ˆ Ω ) is a solution to Problem (4) .

Proof. We consider both parts of the statements.

Part (a):

We provide a proof by contrapositive. For matrices ˆ B and ˆ Ω , suppose ˆ A := ( I -ˆ B ) ˆ Ω -1 2 is not a solution to Problem (1). That is, there exists matrix ¨ A such that

and

By Lemma 3, matrix ¨ A is non-singular, and thus, by Lemma 5, there exist matrices ¨ B ∈ R n × n off and ¨ Ω ∈ diag( R n > 0 ) such that

which, with Inequality (22), indicate ∥ ¨ B ∥ 0 < ∥ ˆ B ∥ 0 . Furthermore, using Eq. (23) and the assumption ˜ A = ( I -˜ B ) ˜ Ω -1 2 , we have

Therefore, ( ¨ B , ¨ Ω ) satisfies the constraint of Problem (4) and leads to a smaller zero norm for the objective function, and thus ( ˆ B , ˆ Ω ) will never be a solution of Problem (4).

Lemma 6 implies

Part (b):

Let ˆ A be a solution to Problem (1). By Lemma 3, matrix ˆ A is non-singular, and thus, by Lemma 5, there exist matrices ˆ B ∈ R n × n off and ˆ Ω ∈ diag( R n > 0 ) such that

It then remains to prove that ( ˆ B , ˆ Ω ) is a solution to Problem (4), which we do so by contradiction. Suppose that ( ˆ B , ˆ Ω ) is not a solution to Problem (4). That is, there exists solution ( ¨ B , ¨ Ω ) such that

and

Define ¨ A := ( I -¨ B ) ˜ Ω -1 2 . Lemma 6 implies ∥ ˆ A ∥ 0 = ∥ ˆ B ∥ 0 + n and ∥ ¨ A ∥ 0 = ∥ ¨ B ∥ 0 + n , which, with Inequality (24), indicates ∥ ¨ A ∥ 0 < ∥ ˆ A ∥ 0 . Furthermore, using Eq. (25) and the assumption ˜ A = ( I -˜ B ) ˜ Ω -1 2 , we have

Therefore, ¨ A satisfies the constraint of Problem (1) and leads to a smaller zero norm for the objective function, and thus ˆ A will never be a solution of Problem (1), which is a contradiction.

D.8 Proof of Theorem 5

In this section, we first provide the proofs of Propositions 10 and 11, which together straightforwardly imply Theorem 5. Before proving Proposition 10, we state the following lemma from Shimizu et al. [38] that is useful for the proof.

Lemma 11 (Shimizu et al. [38, Lemma 1]). Let A be a lower triangular matrix with all diagonal entries being nonzero. Let P 1 and P 2 be two permutation matrices. Then, a permutation of rows and columns of A , i.e., P ⊤ 1 AP 2 , has only nonzero entries in the diagonal if and only if the row and column permutations are equal, i.e., P 1 = P 2 .

We now provide the proof of Proposition 10.

Proposition 10. Suppose A ∼ ( I -B ) Ω -1 2 for matrices A ∈ R n × n , B ∈ R n × n off , and Ω ∈ diag( R n > 0 ) . Then, A satisfies Assumption 2 if and only if matrix B represents a DAG.

Proof. Without loss of generality, we consider the case in which A and ( I -B ) Ω -1 2 differ only in column permutations, instead of signed column permutations. This is because, for the latter case, there exists a diagonal matrix D with diagonal entries being ± 1 such that AD and ( I -B ) Ω -1 2 differ only in column permutations, and furthermore, AD satisfies Assumption 2 if and only if A satisfies Assumption 2.

Therefore, suppose there exists a permutation matrix P 1 such that

We now consider both parts of the statements.

If part:

Suppose that matrix B represents a DAG. Then, there exists permutation matrix P 2 such that P ⊤ 2 BP 2 is strictly lower triangular. Therefore, P ⊤ 2 ( I -B ) P 2 = I -P ⊤ 2 BP 2 is lower triangular. This implies that P ⊤ 2 ( I -B ) Ω -1 2 P 2 is lower triangular because Ω -1 2 is a diagonal matrix and does not affect the support. By substituting Eq. (26), P ⊤ 2 AP -1 1 P 2 is lower triangular. Clearly, P -1 1 P 2 is also a permutation matrix. Therefore, matrix A can be permuted by row and column permutations to be lower triangular, and thus satisfies Assumption 2.

Only if part:

Suppose matrix A satisfies Assumption 2. Then, there exist permutation matrices P 2 and P 3 such that P ⊤ 2 AP 3 is lower triangular. Substituting Eq. (26), P ⊤ 2 ( I -B ) Ω -1 2 P 1 P 3 is lower triangular.

Since P 1 P 3 is also permutation matrix, this indicates that ( I -B ) Ω -1 2 , and thus I -B , satisfy Assumption 2. By Lemma 8, I -B is non-singular, which indicates that

̸

̸

̸

̸

It is known that the determinant of a lower triangular matrix is the product of its diagonal entries. Since P ⊤ 2 AP 3 is lower triangular and det( P ⊤ 2 AP 3 ) = 0 , all diagonal entries of P ⊤ 2 AP 3 must be nonzero. Now define both of which are permutation matrices. By some algebraic manipulations of Eq. (26) and further substituting the above definitions, we have

where all diagonal entries are nonzeros. Applying Lemma 11 w.r.t. matrix P ⊤ 2 AP 3 , we have

Since we have shown that P ⊤ 2 ( I -B ) Ω -1 2 P 1 P 3 is lower triangular, further substitution of P 2 = P 1 P 3 indicates that P ⊤ 2 ( I -B ) Ω -1 2 P 2 is lower triangular. Since right multiplication of Ω -1 2 does not affect the support of I -B , we see that

is also lower triangular. This indicates that P ⊤ 2 BP 2 is lower triangular, which, with the assumption that the diagonal entries of B are zeros, imply that P ⊤ 2 BP 2 is strictly lower triangular. Therefore, matrix B represents a DAG.

After proving Proposition 10, we now consider Proposition 11. Before that, we provide a result by Ghassami et al. [20] that is useful for the proof. We first describe the notion of parent exchange by Ghassami et al. [20]. Let △ be the symmetric difference operator that identifies the elements present in either of the sets but not in the intersection. For DAG G with weighted adjacency matrix B , its vertices x i and x j are said to be parent exchangeable if | supp(( I -B ) i ) △ supp(( I -B ) j ) | = 1 , i.e., there exists k ∈ [ n ] such that supp(( I -B ) i ) △ supp(( I -B ) j ) = { k } . In such case, a support rotation can be performed on columns ( ξ I -B ) i and ( ξ I -B ) j that sets a nonzero entry on those columns, except ( ξ I -B ) i,i and ( ξ I -B ) j,j , to zero. In other words, the parent of x i and x j that corresponds to the zeroed entry is removed. Furthermore, the entry ( ξ I -B ) k,i or ( ξ I -B ) k,j is set to × , which corresponds to adding the missing edge x k → x i or x k → x j . Ghassami et al. [20] defined such an operation to be a parent exchange . We then provide the following corollary that is straightforwardly derived from Ghassami et al. [20, Corollary 2 & Proposition 1].

Corollary 2 (Ghassami et al. [20, Corollary 2]). DAGs G 1 and G 2 are Markov equivalent if and only if there exists a sequence of parent exchanges that maps G 1 to G 2 , and one that maps G 2 to G 1 .

We now provide the proof of Proposition 11.

Proposition 11. Let G be a DAG with weighted adjacency matrix B . Then, matrix I -B satisfies Assumption 1 if and only if the Markov equivalence class of G is a singleton.

Proof. We consider both parts of the statements.

Note that

With Eq. (26), we have and therefore

which, by plugging into Eq. (27), implies

If part:

We provide a proof by contrapositive. Suppose that matrix I -B does not satisfy Assumption 2. That is, there exist i, j ∈ [ n ] and i = j such that

̸

̸

Clearly, we have supp(( I -B ) i ) = supp(( I -B ) j ) , because otherwise there will be a cycle with length of two over variables x i and x j , which contradicts the assumption that G is a acyclic. This implies

̸

By definition, x i and x j are parent exchangeable, and therefore there exists a parent exchange that maps DAG G to another directed graph G ′ where G ′ = G . We now show by contradiction that G ′ is a DAG. Suppose that G ′ is a not DAG. By Eq. (28), there exists variable x k such that

Here, we must have k = i or k = j , because otherwise we have

which leads to a cycle over variables x i and x j . Without loss of generality, we consider the case of k = j , which implies

and ( ξ I -B ) j,i = 0 . This indicates that the entry ( ξ I -B ) j,i is set to × (i.e., the edge x j → x i is added) after the parent exchange to DAG G ′ , which subsequently leads to a cycle in G ′ . In this case, there must exist a path x i → x l 1 → ··· → x l m → x j in DAG G , where m < n -2 and l 1 , . . . , l m ∈ [ n ] \ { i, j } , to which adding the edge x j → x i leads to a cycle in DAG G ′ . By Eq. (29), x l m is also a parent of x i , indicating that there exists a cycle x i → x l 1 →··· → x l m → x i in DAG G , which contradicts the assumption that G is acyclic. Therefore, G ′ must be a DAG.

Since there exists a parent exchange that maps DAG G to another DAG G ′ , clearly there also exists a (reversed) parent exchange that maps DAG G ′ back to DAG G . By Corollary 2, DAGs G and G ′ are Markov equivalent. Therefore, the Markov equivalence class of G contains at least two DAGs and is not a singleton.

Only if part:

Suppose that matrix I -B satisfies Assumption 2. That is, for all i, j ∈ [ n ] and i = j ,we have

̸

In this case, every pair of vertices are not parent exchangeable, and thus parent exchange cannot be applied for any pair of vertices in DAG G . Therefore, for any DAG G ′ = G , there exists no sequence of parent exchanges that maps G to G ′ , implying that they are not Markov equivalent. This indicates that all DAGs are not Markov equivalent to DAG G , except itself, and thus the Markov equivalence class of G is a singleton.

With Propositions 10 and 11 in place, we provide the proof of Theorem 5.

Theorem 5. Suppose A ∼ ( I -B ) Ω -1 2 for matrices A ∈ R n × n , B ∈ R n × n off , and Ω ∈ diag( R n > 0 ) . Then, A satisfies Assumptions 1 and 2 if and only if B represents a DAG whose Markov equivalence class is a singleton.

Proof. We consider both parts of the statements.

If part:

Suppose matrix B represents a DAG whose Markov equivalence class is a singleton. By Proposition 10, matrix A satisfies Assumption 2. Furthermore, Proposition 11 implies that I -B , and thus ( I -B ) Ω -1 2 , satisfy Assumption 1. Since A and ( I -B ) Ω -1 2 differ only in signed column permutations, and Assumption 1 involves only pairwise comparison of the support matrix, A must also satisfy Assumption 1.

Only if part:

Suppose A satisfies Assumptions 1 and 2. By Proposition 10, matrix B represents a DAG. Since A and ( I -B ) Ω -1 2 differ only in signed column permutations, and Assumption 1 involves only pairwise comparison of the support matrix, I -B also satisfies Assumption 1. By Proposition 11, the Markov equivalence class of the DAG represented by B is a singleton.

̸

D.9 Proof of Lemma 1

Before proving Lemma 1, we first provide another result that is useful for the proof. To ease further reasoning, we define the function

Zheng et al. [48], Zhang et al. [47] have shown that, for any matrix A , f ( A ) = 0 if and only if A represents the weighted adjacency matrix of a DAG. Also, it is known that the weighted adjacency matrix of a directed graph can be permuted via simultaneous equal row and column permutations to be strictly lower triangular if and only if the graph is a DAG. Therefore, we provide the following corollary that is straightforwardly implied by the results by Zheng et al. [48], Wei et al. [45].

Lemma 12 (Wei et al. [45, Theorem 1]). For any matrix A , f ( A ) = 0 if and only if it can be permuted via simultaneous equal row and column permutations to be strictly lower triangular.

We now provide the proof for Lemma 1.

Lemma 1. For any matrix A , g ( A ) = 0 if and only if it can be permuted via simultaneous equal row and column permutations to be lower triangular.

Proof. Wefirst define D A as a diagonal matrix of the same size as matrix A , where its diagonal entries are equal to those of A and its non-diagonal entries are zero. Clearly, we have A = off( A ) + D A . For any matrix P , This implies

and clearly we have

If part:

Suppose that there exists permutation matrix P such that P ⊤ AP is lower triangular. By Eq. (31), we have P ⊤ off( A ) P = P ⊤ AP -P ⊤ D A P . Clearly, the diagonal entries of P ⊤ AP are exactly the same as those of P ⊤ D A P , which cancel out each other. Therefore, the diagonal entries of P ⊤ off( A ) P are zeros, indicating that it is strictly lower triangular. By Lemma 12, this implies f (off( A )) = 0 , and thus g ( A ) = 0 by Eq. (30).

Only if part:

Suppose g ( A ) = f (off( A )) = 0 . By Lemma 12, there exists permutation matrix P such that P ⊤ off( A ) P is strictly lower triangular. Clearly, P ⊤ D A P is a diagonal matrix. By Eq. (31), P ⊤ AP is lower triangular, i.e., A can be permuted via simultaneous equal row and column permutations to be lower triangular.

D.10 Proof of Proposition 5

Proposition 5. The matrix A satisfies Assumption 2 if and only if there is a matrix ˆ A such that it is a column permutation of A and that g ( ˆ A ) = 0 .

Proof. We consider both parts of the statements.

If part:

Suppose that there exists matrix ˆ A such that it is a column permutation of A and that g ( ˆ A ) = 0 . By definition, there exists permutation matrix P 2 such that ˆ A = AP 2 . Also, by Lemma 1, there exists permutation matrix P 1 such that P ⊤ 1 ˆ AP 1 is lower triangular, which implies that P ⊤ 1 AP 2 P 1 is lower triangular. Clearly, P 2 P 1 is also a permutation matrix. Therefore, matrix A can be permuted by row and column permutations to be lower triangular, indicating that it satisfies Assumption 2.

Only if part:

Suppose that matrix A satisfies Assumption 2, i.e., there exist permutation matrices P 1 and P 2 such that P ⊤ 1 AP 2 is lower triangular. Defining P 3 := P 2 P -1 1 , which is also a permutation matrix,

and substituting it into the previous statement, we see that P ⊤ 1 AP 3 P 1 is lower triangular. Further substitution of ˆ A := AP 3 implies that P ⊤ 1 ˆ AP 1 is lower triangular, which, by Lemma 1, indicates g ( ˆ A ) = 0 . Clearly, ˆ A is a column permutation of A .

D.11 Proof of Theorem 6

Theorem 6 (Alternative Formulation of Identifiability). Suppose that the true mixing matrix ˜ A satisfies Assumptions 1, 2, and 3. Let ˆ A be a solution of the following problem:

Then, we have ˆ A ∼ ˜ A .

Proof. Let ˆ A be a solution to Problem (5). Suppose by contradiction that it is not a solution to Problem (2). That is, there exists matrix ¨ A satisfying Assumption 2 such that

By Proposition 5, there exists permutation matrix P such that g ( ¨ AP ) = 0 . Furthermore, by Eq. (32), we have

Therefore, ¨ AP satisfies the constraint of Problem (5) and leads to a smaller zero norm for the objective function, and thus ˆ A will never be a solution of Problem (5), which is a contradiction. Therefore, ˆ A must be a solution to Problem (2). Since matrix ˜ A also satisfies Assumptions 1, 2, and 3, applying Theorem 1 completes the proof.

D.12 Proof of Theorem 7

Theorem 7 (Likelihood-Based Method). Suppose that the true mixing matrix ˜ A satisfies Assumptions 1, 2, and 3. Let ˆ A be a solution of Problem (7) with sparsity regularizer ρ ( A ) = 0 . 5 ∥ A ∥ 0 log T . Then, we have ˆ A ∼ ˜ A in the large sample limit.

Proof. First, we have g ( ˆ A ) = 0 and, in the large sample limit, ¯ Σ = ˜ Σ . Similar to BIC [36], the likelihood term dominates in the large sample limit as the weight of the likelihood function increases much faster than that of the sparsity regularizer. Therefore, in the large sample limit, we have ˆ A ˆ A = ˜ Σ = ˜ A ˜ A . Also, for any matrix A that satisfies AA = ˜ A ˜ A and g ( A ) = 0 , the sparsity regularizer indicates ∥ ˆ A ∥ 0 ≤ ∥ A ∥ 0 , because otherwise ˆ A will never be a solution of Problem (7). This implies that ˆ A is also a solution to Problem (5). Since matrix ˜ A also satisfies Assumptions 1, 2, and 3, applying Theorem 1 completes the proof.

E Supplementary Discussion on Structural Assumptions

E.1 Examples Satisfying Structural Variability Assumption

To illustrate the intuition of the proposed assumption of structural variability (i.e., Assumption 1), we provide several examples on the connective structure from sources to observed variables (which corresponds to the support of mixing matrix) satisfying that assumption, as illustrated in Figure 3.

E.2 Efficient Approach for Verifying Assumption 2

Assumption 2 involves finding a certain combination of row and column permutations for mixing matrix A , which may at first appear inefficient to verify. We provide a more efficient way to do so, by leveraging the interpretation of our assumptions in the context of causal discovery (see Section 3.4 for detailed discussion). Specifically, we provide the following corollary that is a straightforward consequence of Lemma 2 and Proposition 10, whose proof is omitted.

̸

̸

Corollary 3 (Verification of Assumption 2). Let A be a non-singular matrix and P be a permutation matrix such that the diagonal entries of AP are nonzero. Let G = ( V , E ) be a directed graph where V = { v 1 , . . . , v n } and E = { v j → v i : ( AP ) i,j = 0 , i = j } . Then, directed graph G is acyclic if and only if matrix A satisfies Assumption 2.

s

1

s

2

s

3

s

4

x

1

x

2

x

3

x

4

(a)

s

1

s

2

s

3

s

4

x

1

x

2

x

3

x

4

(b)

s

1

s

2

s

3

s

4

x

1

x

2

x

3

x

4

(c)

Specifically, Corollary 3 implies that it suffices to find a column permutation such that the diagonal entries of the permuted matrix are nonzero. Note that such column permutation is guaranteed to exist as indicated by Lemma 2. We then construct a directed graph G based on such permuted matrix and check if G contains cycle, e.g., via depth-first search. Instead of searching for a certain combination of row and column permutations, this procedure may be more efficient because it involves only finding specific column permutations.

F Supplementary Estimation Details

We provide the estimation details for the methods described in Section 4.2. In our experiments, we use the average log-likelihood 1 T L ( A ; ¯ Σ ) as the objective (instead of L ( A ; ¯ Σ ) in Eq. (7)) for likelihoodbased method. For the sparsity term ρ ( A ) , we use MCP with hyperparameters λ = 1 , α = 40 and λ = 0 . 1 , α = 10 for decomposition-based and likelihood-based methods, respectively.

Furthermore, for both methods, we use the L-BFGS algorithm [11] implemented in SciPy [44] to solve each unconstrained optimization problem of quadartic penalty method. Since the formulations involve solving nonconvex optimization problems, we run L-BFGS with 30 random initializations, where each entry of the initial solution A 0 is sampled uniformly at random from [ -0 . 1 , 0 . 1] . In this case, the final solution is chosen via model selection. For quadratic penalty method, we use c 1 = 10 -5 and c 1 = 10 -2 for decomposition-based and likelihood-based methods, respectively, and use β = 1 . 5 for both methods.

Lastly, we also use a threshold of 0 . 01 to remove small weights in the estimated mixing matrix. We run each of the experiments on 12 CPUs and 8 GBs of memory.

Computational complexity. To compute the constraint term g ( A ) , a straightforward approach is to compute each matrix power in g ( A ) and then sum their traces up, which requires O ( n ) matrix multiplications. In our implementation, we adopt a more efficient approach with computational complexity of O (log n ) matrix multiplications, inspired by Zhang et al. (2020). The rough idea is to perform exponentiation by squaring (i.e., a procedure similar to binary search) and recursively compute the term g ( A ) .

Furthermore, each L-BFGS run has a computational complexity of O ( m 2 n 2 + m 3 + mn 2 t ) , where m ≪ n 2 is the memory size and t is the number of inner iterations of the L-BFGS run. Typically, we have t = 250 for each L-BFGS run, and 125 iterations for the quadratic penalty method.

G Supplementary Experimental Results

In addition to the MCC reported in Section 5, we report the Amari distance to evaluate the identification performance in Figures 4 and 5, respectively.

Vanilla

FastICA-D

FastICA

30

100

300

1000

3000

10000

Sample Size

1.0

1.5

2.0

Amari Distance

30

100

300

1000

3000

10000

Sample Size

0.5

1.0

1.5

2.0

Amari Distance

(a) Likelihood-based method.

(b) Decomposition-based method.

Vanilla

FastICA-D

FastICA

0.0

0.2

0.4

0.6

0.8

1.0

Ratio of Gaussian Sources

0.0

0.5

1.0

1.5

2.0

Amari Distance

(a) Likelihood-based method.

0.0

0.2

0.4

0.6

0.8

1.0

Ratio of Gaussian Sources

0.0

0.5

1.0

1.5

2.0

Amari Distance

(b) Decomposition-based method.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAiming to address rotational invariance of Gaussian sources, this work first proposed an ICA identifiability theory based on Structural Variability. It then proposed two methods based on sparsity regularization and continuous constrained optimization to estimate the mixing matrix. It also made connections between ICA and causal discovery. It finally showed preliminary results to validate the proposed theory and estimation methods.\n\nSTRENGTHS:\n1. The proposed method could be potentially useful as it claims to identify linear Gaussian sources with a weaker assumption compared to structural sparsity. \n\n2. The notations, theorems and proofs are clear in general. \n\n3. The proofs or explanations in the Appendix are very detailed and helpful. \n\n4. The author(s) conducted experiments to validate their theory and showed the effectiveness of the proposed SparseICA method in two simulated datasets.\n\nWEAKNESSES:\n1. As the author(s) pointed out, my main concern is that there is no sufficient empirical evaluation to demonstrate the effectiveness and generalizability of the proposed theory. Current experiments only used simulated data in two settings. How does the proposed theory work on real-world datasets? This work would be more convincing if there were experiments on real-world datasets.\n\n2. Additionally, the author(s) argued that the theory proposed by Zheng et al. 2022 couldn't capture hierarchical structures (lines 39 - 41), but it doesn't seem that the authors performed experiments to identify Gaussian sources which are organized in a hierarchical manner.\n\n3. I don't feel all theorems/assumptions/examples were explained very clearly, but I also note that the author(s) provided detailed proofs or explanations for each proposed theorem/assumption/example in the Appendix. So I would suggest that the author(s) try to organize the manuscript more logically and guide readers to the corresponding Appendix section in the main text. \n\n4. There are a lot of typos in the present manuscript. For example, line 71: $j$-th column by $a_{:, j}$; line 104: Section 3.2; line 248: there exists; line 254: repeated \"with\". Please proof-read the manuscript prior to submission.\n\n5. There is no statistical test to compare results. \n\n6. There is no code provided to replicate the results. Please consider making the code publicly available.\n\nQUESTIONS:\n1. Example 1: I understand the derivation in this example but I wonder how to identify such constraints. Do we have to derive these hard or non-hard constraints for each matrix? Are there any general principles to derive such constraints? If so, please briefly explain the principles. If not, it doesn't seem to be computationally efficient to derive these constraints as the dimension of A increases.\n\n2. Line 130: Please elaborate what \"sufficiently diverse effect\" means exactly.\n\n3. Do you have any explanations about the performance difference from two estimation methods in Figure 1? For example, decomposition method seems worse than likelihood method in vanilla case.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides theorems under which the mixing matrix of linear ICA with Gaussian sources can be identified. While Gaussian sources are known to be unidentifiable in the classical ica theory, identifiability is possible if certain sparse structure is assumed for the mixing matrix as was initailly shown in Zheng. The assumptions in Zheng were restrictive however. This work expands the work of Zheng to provide more general conditions, namely:\n- assumption of structural provided is much less restrictive on the sparsity pattern\n- necessity of this assumption is proven (under the problem considered here)\n- connection of this approach to causal discovery is shown\n- the theorems lead to new estimation methods based on second-order statistics that allow ICA to be applied on Gaussian sources\n\nSTRENGTHS:\nStrengths:\n- the paper is in general well written and of good quality\n- the assumptions on sparsity are more much more reasonable than in previous works and should allow for future works in this area. To this end theorems 2 and 3 provide relevant and useful comparison\n- theoretical assumptions are also often nicely illustrated with examples\n- some nice approach of transforming the theorems into algorithms in a justified manner (e.g. framing the search space of A; theorem 6)\n- connection between ica and causal discovery is well known and this paper further contributes to that similarity\n- addition of a necessary condition is a nice result and helps to understand the limits of this approach\n\nWEAKNESSES:\nWeaknesses:\n\nThe biggest conceptual problem I have is that the identifiability here is framed in terms of an optimization problem where the ground-truth covariance matrix is assumed. I find this idea hard to follow since identifiability should be a property of the data model. Typically the process is to assume that we have $\\log p(x; \\theta) = \\log p(x; \\hat{\\theta})$ and then show that this implies $\\theta \\sim \\hat{\\theta}$ -- this approach makes sense as the starting point, equality of likelihoods, can be justified on the basis of MLE and its guarantees (at least in theory). Here instead one seems to start with the assumption that $AA' = \\hat{A}\\hat{A}'$. But the justification for this starting point is missing -- you should write it out fully. Also, in practice the empirical covariance matrix is used (eq. 8) but what are the guarantees here? I'm quite happy to revise my score upwards once this, and below, are discussed. \n\nI think the practical motivations for this paper are lacking and weak, perhaps partly due to a bit of unclear writing. For example, the authors write that \"many biological traits ... are often normally distributed\" to justify the importance of doing ica on Gaussian latents. But those listed biological signals are *not* typically latent, but often observed (after noise) and can usually be handled by standard ICA as there is no problem of Gaussian observations. Of course, it is possible to have latent biological signals too but that should be explained more carefully. It should be stressed however that the work still has important contribution in terms of fundamental research -- practical appeal does not always need to be obvious, so this is not necessarily a big problem. Authors also write that one possible situation in which their theorems are more useful than previous works, is when \"an observed variable serves as a root cause\". This may be possible but goes against the typical idea of ICA where observations are all assumed mixtures of independent latents. I feel this is however more easily understood in causal framwork. This leads to my next point...\n\nI find in general that ideologically this work is in some sense difficult to place as it feels like the ideas are more relevant to causal discovery yet it's framed as an ICA paper. But the assumptions on sparsity can still be quite restrictive and a substantially sparse mixing matrix, again, goes against the whole concept of ICA in some sense -- we are not really solving the mixing matrix problem if the mixing that happens is quite limited. Despite the examples given, it is still hard to understand exactly how restrictive sparsity assumptions are *in practice*.\n\nI find the estimation algorithm of the decomposition-based method in general a bit difficult to justify; optimally you would use an $l_o$ regularizer as this would agree with theory but that seems difficult to optimize. The likelihood-based method seems much more principled and I wonder if this paper could have been better framed around likelihood-based methodology in general.\n\nI'm surprised by the poor experimental results -- typically linear ICA achieves 0.99 correlation to the ground-truths eg. with FastICA so I would have expected similar results here. There are also no experiments on real data (not necessarily a big problem due to the breadth of theory here).\n\nQUESTIONS:\nMore concerns and things I'd expect revised for me to improve my score:\n\nline 16-17: the examples+reference here are pretty much straight form Hyvärinen (Independent Component Analysis: Algorithms and Applications, section 7). Perhaps with a bit more originality and effort some more varied, newer, examples or at least references could be provided?\n\n\"distribution of Gaussian sources, the sparsity of the mixing matrix undergoes noticeable changes.\" Could you clarify what is meant here?\n\nPlease include \"links\" to relevant proofs in the text. At the moment one has to wonder in the appendix to see whether the proofs exist. \n\n329: \"...based method (Eq. (8) or Eq. (9)) on data where both Assumptions 1 and 2 do not hold\" The grammar could be polished as now it could mean \"where assumption 1 and 2 do not hold at the same time\"  or that \"neither assumptions 1 and 2 hold\"\n\nLIMITATIONS:\nAuthors do indeed point out the limited applications to real world data. Authors also admit that \" Since the true generating process of real-world data is inaccessible, it is challenging to quantitatively  evaluate the applicability of these sparsity assumptions.\" \n\nhowever there is no discussion on why the experimental results are not as good as one would expect, this would be welcome.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of estimating a mixing matrix $A^*$, given measuremnts $x = A^* s$. The authors show that under Gaussian $s$, some assumptions on $A$, and using infinite $x$, the sparsest $A$ that satisfies $AA^{T} = A^* A^{* T}$ recovers $A^*$ upto permutation and sign of the columns.\n\nThe authors also show connections to causal discovery in the case of structural equation models.\n\nSince the sparse norm is non-convex, the authors propose a convex relaxation using the $\\ell_1$-norm that can be run in practice. \n\nThe sparsity assumptions on $A$ induce polynomial constraints $H(A)$ on the elements of the matrix $\\Sigma = AA^T$. These constraints are typically hard to compute, but the authors define an auxilliary function $h(A) = \\text{tr} ( \\sum_{k=2}^{n} off(A) \\odot off(A) )$, where $off(A)$ is the off-diagonal version of $A$, and $\\odot$ is the Hadamard product. The constaint set now becomes $h(A) = 0$, and the authors show that convex solvers can work in this setting.\n\n\nMinor:\nUsing $H$ for the set of hard constraints and $h$ for the trace of the defined matrix is slightly confusing, please change it.\n\nSTRENGTHS:\n- The authors show that their assumptions on the matrix are strictly weaker than existing approaches (Section 3.3)\n- The authors show connections to causal discovery, which I thought were interesting.\n- Existing approaches cannot handle Gaussian sources $s$, as this induces rotational symmetry on $AA^T$, and hence we can only recover $A$ upto rotation. In contrast, the current work can recover it upto permutation and sign of the matrix.\n- The authors are able to show that the convex relaxation in Section 4 recovers $A$ in the asymptotic limit of infinite samples of $x$.\n- Experimental results are nice.\n\nWEAKNESSES:\n- The identifiability results are for infinite samples of $x$.\n\n- No runtime guarantees\n\n- The authors should spend more time commenting on the technical quality of the results. In some sense, the extension from the non-convex program to the convex relaxation is ``expected''. What new technical tools were required over traditional sparsity?\n\n- Experiments are a little simplistic, but that's fine for a theory paper.\n\nQUESTIONS:\n- I'm not an expert on causal discovery. Are the connections between ICA and identifiability of structural equations established results? Relatedly, can you explain your contributions over existing work.\n\n- The authors should spend more time commenting on the technical quality of the results. In some sense, the extension from the non-convex program to the convex relaxation is ``expected''. What new technical tools were required over traditional sparsity? Without commenting on this, it would seem that choosing the right assumptions (1-3) are the key for this paper.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors develop new identifiability conditions for ICA, based on sparsity constraints. These assumptions, in particular, do not require the sources to be non-Gaussian, as is usually the case in ICA.\nMostly, the authors take inspiration in previous work by Zheng et al, and weaken their main condition. Some assumption are shown to be necessary.\nAs is usual with sparsity-based analyses, the resulting optimization problem is quite non-trivial, here the main difficulty is to explore the space of directed adjacency matrices whose corresponding graph is a DAG. The authors provide some discussion and implementation procedures about this.\nThe proposed method is evaluated against FastICA on synthetic data. A (quick) ablation study is realized to examine the effect of the assumptions not being satisfied.\n\nSTRENGTHS:\n- the contribution is interesting for the long-standing problem of non-Gaussian ICA. The authors make a commendable effort to discuss previous results, their main inspiration, and the strengthening of the results/weakening of assumptions\n\n- the paper is well-written and pedagogical, with examples and discussion scattered throughout about the assumptions and the results\n\n- some aspects of the paper are quite developed (relation with causal discovery, implementation, necessity of the assumptions...) without sacrificing the clarity of the writing despite the space constraints\n\nWEAKNESSES:\n- the only assumption that less discussed than the other ones, in relation with previous work or its \"realistic aspect\" with respect to real data, seems to also be the \"main\" one, Assumption 2. The focus on hard constraints is theoretically understandable, but how realistic is it ? Also, the triple-presentation of the optimization problem is a bit redundant, the end problem being \"just\" optimizing among the space of DAGs\n\n- the paper focuses on exact identifiability, when the true covariance matrix is \"available\". This can be confusing for people not expert on ICA, can the effect of sample size and covariance estimation be discussed ? Does the resulting noise has an effect on \"admissible\" sparsity level (empirically, let's say), and for now the sparsity level $\\| A\\|_0$ does not seem to be constrained in any way (beyond assumption 1 and 2)\n\n- there is no discussion on computational complexity, it seems that the constraint $h(A)=0$ is very costly as it involves computing the $n$th power of an $n \\times n$ matrix, at each iteration. Is there any simplification over brute force ?\n\n- no evaluation on real data. When the assumptions are only approximately satisfied, and there is strong noise on the covariance, it is not clear what is the performance of the proposed method\n\nQUESTIONS:\nSee \"weaknesses\" above\n\nMinor questions/typos:\n\n- l22 : \"Kurtois\"\n- l 95 : \"statitsic\"\n- Assumption 3 is a bit unclear at first read (it seems like a definition), only one sense of the \"if and only if\" is important, perhaps it is possible to reformulate\n- what is the exact sense of \"asymptotically\" in theorem 7 ? (see also question/comment about sample size, which is not discussed in the paper)\n\nLIMITATIONS:\nThe authors have discussed some aspects extensively, eg Assumption 1 with respect to previous work, other are a bit scarce, eg Assumption 2. Also, the computational complexity is not discussed, and there is no experiments on real data.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}