SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ignavier  $\mathbf{Ng}^{*1}$ , Yujia Zheng $^{*1}$ , Xinshuai Dong $^{1}$ , Kun Zhang $^{1,2}$
{ignavierng, yujiazh, dongxinshuai, kunz1}@cmu.edu
Independent component analysis (ICA) is a fundamental statistical tool used to reveal hidden generative processes from observed data. However, traditional ICA approaches struggle with the rotational invariance inherent in Gaussian distributions, often necessitating the assumption of non-Gaussianity in the underlying sources. This may limit their applicability in broader contexts. To accommodate Gaussian sources, we develop an identifiability theory that relies on second-order statistics without imposing further preconditions on the distribution of sources, by introducing novel assumptions on the connective structure from sources to observed variables. Different from recent work that focuses on potentially restrictive connective structures, our proposed assumption of structural variability is both considerably less restrictive and provably necessary. Furthermore, we propose two estimation methods based on second-order statistics and sparsity constraint. Experimental results are provided to validate our identifiability theory and estimation methods.
Independent component analysis (ICA) [12] has emerged as an essential statistical tool in the scientific community, with application in various disciplines including neuroscience [27], biology [41, 9], and Earth science [32]. It aims to uncover the hidden generative processes that govern the observed data and separate mixed signals into independent sources. However, it is known that the traditional approaches of ICA struggle with Gaussian sources, which may limit their applicability in a wide variety of contexts. For instance, several biological traits and measurements such as height, blood pressure, and measurement errors in genetics, as well as the thermal noises in electronic circuits, may often be normally distributed [29, 34]. Classical identifiability results in ICA rely on higher-order statistics (e.g., Kurtois) [24], and cannot provide desired theoretical guarantees when there is more than one Gaussian source. Identifiability based on second-order statistic is thus essential in these scenarios.
The primary hurdle in applying ICA to Gaussian sources lies in the rotational invariance of Gaussian distribution [25]. To address this issue, earlier studies [30, 5, 35] incorporated additional information by assuming that the sources are nonstationary. However, this extra information may not always be readily available, limiting the generalizability of these approaches. Thus, recent research [49, 1] has started to delve into the connective structure between the sources and the observed variables, as opposed to solely focusing on distributional assumptions (e.g., non-Gaussianity and nonstationarity). This shift of focus is motivated by a key observation: despite the rotational invariance in the distribution of Gaussian sources, the sparsity of mixing matrix undergoes noticeable changes, i.e., it may be denser after rotation [49]. Building on this insight, Zheng et al. [49] introduced two assumptions on the support of the mixing matrix to achieve identifiability of Gaussian sources, leading to a novel perspective for tackling this long-standing challenge in the field of ICA. On the other hand, Abrahamsen and Rigollet [1] assumed that the mixing matrix is generated from a sparse Bernoulli-Gaussian ensemble.
Although the rotational invariance of Gaussian distribution can be resolved by the structural assumptions on the mixing matrix proposed by Zheng et al. [49], they may be deemed overly restrictive. For instance, both of their structural assumptions cannot deal with the case where the set of observed variables influenced by one source is a subset of those affected by another source, which may not be uncommon in practice. This may limit the applicability of ICA in complex real-world scenarios, thus underscoring the need for a weaker and more flexible structural assumption that is capable of addressing the rotational invariance in a more universally applicable manner.
To enhance the applicability with Gaussian sources, we develop an identifiability theory of ICA from second-order statistics under more flexible structural constraints. We introduce a novel assumption, namely structural variability, that is considerably weaker than existing ones. Notably, this assumption is proved to be among the necessary conditions for identifying Gaussian sources by focusing on the connective structure (i.e., the support of the mixing matrix). Moreover, we propose two estimation methods grounded in sparsity regularization and continuous constrained optimization. The efficacy of our proposed methods has been validated through experiments, which also reaffirm the validity of our theoretical result. Lastly, as a matter of independent interest, we establish the connection between our identifiability result of ICA with causal discovery from second-order statistics; our finding further bridges the gap between these two fields and provides insights into the interpretation of our result.
We consider the ICA setup given by  $\mathbf{x} = \tilde{\mathbf{A}}\mathbf{s}$ , where  $\mathbf{x} \in \mathbb{R}^n$  denotes the observed random vector,  $\mathbf{s} \in \mathbb{R}^n$  is the latent random vector representing the independent components, also called sources, and  $\tilde{\mathbf{A}} = [\tilde{\mathbf{a}}_1|\dots |\tilde{\mathbf{a}}_n] \in \mathbb{R}^{n\times n}$  denotes the unknown mixing matrix. We assume here that  $\tilde{\mathbf{A}}$  has full rank and all sources are standardized. For the estimated mixing matrix  $\hat{\mathbf{A}}$  and the ground-truth one  $\tilde{\mathbf{A}}$ , we write  $\hat{\mathbf{A}} \sim \tilde{\mathbf{A}}$  if they differ only in column permutations and sign changes of columns, and  $\hat{\mathbf{A}} \not\sim \tilde{\mathbf{A}}$  vice versa. The goal is then to estimate  $\hat{\mathbf{A}}$  such that  $\hat{\mathbf{A}} \sim \tilde{\mathbf{A}}$ ; in this case, one could identify a one-on-one mapping between the ground truth sources and the estimated ones, i.e., the unknown mixing process has been demixed during estimation. Since the support of a mixing matrix  $\mathbf{A}$  essentially represents the connective structure between sources and observed variables, we have the following definition for the ease of reference.
Definition 1 (Connective Structure). Given a mixing matrix  $\mathbf{A}$ , we define its connective structure as a directed bipartite graph  $\mathcal{G}_{\mathbf{A}} = (\mathcal{V}_{\mathbf{A}}, \mathcal{E}_{\mathbf{A}})$  from sources  $\mathbf{s}$  to observed variables  $\mathbf{x}$ , where the nodes and edges are defined as  $\mathcal{V}_{\mathbf{A}} := \{s_i\}_{i=1}^n \cup \{x_i\}_{i=1}^n$  and  $\mathcal{E}_{\mathbf{A}} := \{(s_j, x_i) : a_{ij} \neq 0\}$ , respectively.
Notations. We use bold capital letters (e.g.,  $\mathbf{A}$ ), bold lowercase letters (e.g.,  $\mathbf{a}$ ), and italic letters (e.g.,  $a$ ) to denote matrices, vectors, and scalar quantities, respectively. For any matrix  $\mathbf{A}$ , we denote its  $j$ -th column by  $\mathbf{a}_j$ ,  $i$ -th row by  $\mathbf{a}_{i,:}$ , and  $(i,j)$ -th entry by  $a_{i,j}$ . We also denote by  $\mathbf{A} \mathcal{I}$  the submatrix of  $\mathbf{A}$  by obtaining the columns indexed by set  $\mathcal{I}$ . For any vector  $\mathbf{a}$ , we denote its  $i$ -th entry by  $a_i$ . We define the support set of matrix  $\mathbf{A}$  as  $\mathrm{supp}(\mathbf{A}) := \{(i,j): a_{i,j} \neq 0\}$ , and its support matrix as  $\xi_{\mathbf{A}}$  which is of the same size as  $\mathbf{A}$ , where  $(\xi_{\mathbf{A}})_{i,j} = \times$  if  $a_{i,j} \neq 0$  and  $(\xi_{\mathbf{A}})_{i,j} = 0$  otherwise. The notations of support set and support matrix are similarly defined for vector  $\mathbf{a}$ . We denote by  $\|\mathbf{A}\|_0$  the number of nonzero entries in  $\mathbf{A}$ , and we have  $\|\mathbf{A}\|_0 = |\mathrm{supp}(\mathbf{A})| = \|\xi_{\mathbf{A}}\|_0$ . Furthermore, we denote the  $n \times n$  identity matrix and  $m \times n$  zero matrix by  $\mathbf{I}_n$  and  $\mathbf{0}_{m \times n}$ , respectively; to lighten the notation, we drop their subscripts when the context is clear. We also use  $[n]$  to denote  $\{1,2,\ldots,n\}$ .
3 Identifiability Result without Assuming Non-Gaussianity
By exploiting the non-Gaussianity of the sources, such as fourth-order cumulant, existing approaches are able to estimate the true mixing matrix  $\tilde{\mathbf{A}}$  up to signed column permutation when there is at most one Gaussian source [23, 2]. However, these approaches typically fail in the presence of more than one Gaussian source, because higher-order statistics cannot be utilized for full identifiability. The primary challenge of achieving identifiability for Gaussian sources lies in the rotational invariance of the Gaussian distribution. More specifically, the second-order statistics (or, more specifically, population-level covariance matrix)  $\tilde{\Sigma} = \tilde{\mathbf{A}}\tilde{\mathbf{A}}^{\top}$  remains unchanged if one replaces  $\tilde{\mathbf{A}}$  with  $\tilde{\mathbf{A}}\mathbf{U}$  for any orthogonal matrix  $\mathbf{U}$ . Therefore, considering only second-order statistics, the true mixing matrix  $\tilde{\mathbf{A}}$  is, generally speaking, only identifiable up to right orthogonal transformation without
further assumptions. In this section, we adopt a different perspective that departs from traditional distributional assumptions (i.e., non-Gaussianity and fourth-order cumulant), and instead introduces novel and precise assumptions on the connective structure, specifically the support of the mixing matrix. These assumptions enable identification of the true mixing matrix  $\tilde{\mathbf{A}}$  up to signed column permutation using second-order statistics and sparsity constraint. Roughly speaking, with the population-level covariance matrix  $\tilde{\Sigma}$ , we consider the following formulation:
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {n \times n}} \| \mathbf {A} \| _ {0} \quad \text {s u b j e c t t o} \quad \mathbf {A} \mathbf {A} ^ {\top} = \tilde {\boldsymbol {\Sigma}} = \tilde {\mathbf {A}} \tilde {\mathbf {A}} ^ {\top}. \tag {1}
$$
Note that we start with the assumption of  $\mathbf{A}\mathbf{A}^{\top} = \hat{\boldsymbol{\Sigma}}$  in the formulation above and our identifiability result (e.g., in Theorem 1); such an assumption can be obtained, e.g., from the equality of Gaussian likelihoods in the large sample limit. This also inspires our sparsity-regularized likelihood-based estimation method that will be described in Section 4.2, which is more inline with, e.g., model selection approaches based on sparsity-regularized likelihood [36, 18].
In Section 3.1, we first examine various types of constraints arising from second-order statistics. We then present the main identifiability result in Section 3.2. We show that our assumptions are strictly weaker than existing sparsity assumptions in Section 3.3, and establish the connection between our identifiability theory with causal discovery in Section 3.4. All proofs are given in Appendices C and D.
3.1 Semialgebraic Constraints Arising from Second-Order Statistics
We first discuss various notions related to constraints arising from covariance matrices of observed variables  $\mathbf{x}$ , which serve as a fundamental basis for introducing our assumptions and identifiability result of ICA in Section 3.2. These notions have been studied in the field of algebraic statistics [16], factor analysis [15], graphical models [19, 40], and causality [14, 20].
We begin with the following definition on the set of covariance matrices entailed by  $\xi$  for different values of the free parameters in  $\xi$ .
Definition 2 (Covariance Set). The covariance set of support matrix  $\xi$  is defined as
$$
\boldsymbol {\Sigma} (\boldsymbol {\xi}) := \{\mathbf {A} \mathbf {A} ^ {\top}: \mathbf {A} \in \mathbb {R} ^ {n \times n}, \operatorname {s u p p} (\mathbf {A}) \subseteq \operatorname {s u p p} (\boldsymbol {\xi}), \mathbf {A} \text {i s n o n - s i n g u l a r} \}.
$$
The support  $\xi$  of mixing matrix  $\mathbf{A}$  imposes certain constraints on the entries of the covariance matrix  $\boldsymbol{\Sigma} = \mathbf{A}\mathbf{A}^{\top}$ , which, by Tarski-Seidenberg theorem (see [6]), correspond to semialgebraic constraints, i.e., equality and inequality constraints. The covariance set  $\boldsymbol{\Sigma}(\boldsymbol{\xi})$  is then said to be a semialgebraic set, i.e., a set that can be described with a finite number of polynomial equations and inequalities [6]. Clearly, if a covariance matrix  $\boldsymbol{\Sigma}$  belongs to the covariance set  $\boldsymbol{\Sigma}(\boldsymbol{\xi})$ , then  $\boldsymbol{\Sigma}$  satisfies the semialgebraic constraints imposed by  $\boldsymbol{\xi}$ .
For an equality constraint, the set of values satisfying the constraint has zero Lebesgue measure over the parameter space involved. Given a support matrix  $\xi$ , we denote by  $H(\xi)$  the set of equality constraints it imposes on the corresponding covariance matrices. On the other hand, the set of values satisfying an inequality constraint has nonzero Lebesgue measure. To illustrate these constraints, we provide a three-variable example below. Note that the example only serves as illustrations of the constraints; our estimation methods (in Section 4) do not require deriving them in practice.
Example 1 (Semialgebraic Constraints). Consider support matrices
$$
\boldsymbol {\xi} _ {1} = \left[ \begin{array}{c c c} \times & 0 & 0 \\ \times & \times & 0 \\ \times & 0 & \times \end{array} \right] \qquad a n d \qquad \boldsymbol {\xi} _ {2} = \left[ \begin{array}{c c c} \times & 0 & \times \\ \times & \times & 0 \\ 0 & \times & \times \end{array} \right].
$$
The equality constraints imposed by  $\xi_{1}$  include
$$
\Sigma_ {1, 1} \Sigma_ {2, 3} - \Sigma_ {1, 2} \Sigma_ {1, 3} = 0,
$$
while the inequality constraints imposed by  $\xi_{2}$  include
$$
\left(\Sigma_ {1, 1} \Sigma_ {2, 2} \Sigma_ {3, 3} + \Sigma_ {1, 1} \Sigma_ {2, 3} ^ {2} - \Sigma_ {2, 2} \Sigma_ {1, 3} ^ {2} - \Sigma_ {3, 3} \Sigma_ {1, 2} ^ {2}\right) ^ {2} - 4 \left(\Sigma_ {1, 1} \Sigma_ {2, 2} - \Sigma_ {1, 2} ^ {2}\right) \left(\Sigma_ {1, 1} \Sigma_ {3, 3} \Sigma_ {2, 3} ^ {2} - \Sigma_ {1, 3} ^ {2} \Sigma_ {2, 3} ^ {2}\right) \geq 0.
$$
The detailed derivation can be found in Appendix D.1 and provides insights into how such constraints arise from the corresponding support matrices. In the example above, the covariance matrix  $\Sigma$  generated by any mixing matrix  $\mathbf{A}$  with support  $\xi_{1}$  must satisfy the corresponding equality constraint; similarly, the covariance matrix  $\Sigma$  generated by any mixing matrix  $\mathbf{A}$  with support  $\xi_{2}$  must satisfy the above inequality constraint. These constraints serve as footprints of the mixing matrix on the covariance matrix, and can be exploited for its identifiability, which we explain in the next section.
3.2 Identifiability Result from Second-Order Statistics
In this section, we present our identifiability result of ICA from second-order statistics. The core idea is to introduce precise and mild assumptions on the connective structure from sources to observed variables. These assumptions facilitate the identification of the mixing matrix through the application of a sparsity constraint, formulated in Problem (1). To begin, we describe our primary assumption concerning the connective structure as follows.
Assumption 1 (Structural Variability). Every pair of the columns in the support matrix of  $\mathbf{A}$  differ in more than one entry. That is, for every  $i$ ,  $j \in [n]$  and  $i \neq j$ , we have
$$
\left. \right.\left| \operatorname {s u p p} \left(\mathbf {a} _ {i}\right) \cup \operatorname {s u p p} \left(\mathbf {a} _ {j}\right)\right| - \left| \operatorname {s u p p} \left(\mathbf {a} _ {i}\right) \cap \operatorname {s u p p} \left(\mathbf {a} _ {j}\right)\right| > 1.
$$
The assumption above implies that every pair of sources should influence more than one different observed variable. Notably, in the field of nonlinear ICA with auxiliary variable, Hyvarinen and Morioka [22], Hyvarinen et al. [26] have adopted the assumption of sufficient variability which requires that the auxiliary variable has a sufficiently diverse effect on the distributions of sources; specifically, the conditional distributions of the sources given the auxiliary variable must vary sufficiently. In contrast, our assumption of structural variability requires that every pair of sources influence sufficiently diverse sets of observed variables, facilitating the disentanglement of each source.
We provide several examples in Appendix E.1 to illustrate the broad applicability of the assumption above. Furthermore, the following proposition justifies such an assumption because it is a necessary condition for identifiability via second-order statistics and sparsity. The intuition is that if Assumption 1 is violated, there exists a rotation that maps matrix  $\tilde{\mathbf{A}}$  to another matrix  $\hat{\mathbf{A}}$  which has equal or smaller number of nonzero entries and is not a column permutation of  $\tilde{\mathbf{A}}$ .
Proposition 1. If the true mixing matrix  $\tilde{\mathbf{A}}$  does not satisfy Assumption 1, then there exists a solution  $\hat{\mathbf{A}}$  to Problem (1) such that  $\hat{\mathbf{A}} \not\sim \tilde{\mathbf{A}}$ .
Remark 1 (Necessary Condition). Assumption 1 is a necessary condition for identifiability of ICA via second-order statistics and under sparsity constraint.
We also adopt the following assumption on the mixing matrix for the identifiability of ICA.
Assumption 2 (Permutations to Lower Triangular Matrix). The matrix  $\mathbf{A}$  can be permuted by independent row and column permutations to be lower triangular. That is, there exist permutation matrices  $\mathbf{P}_1$  and  $\mathbf{P}_2$  such that  $\mathbf{P}_1^\top \mathbf{A}\mathbf{P}_2$  is lower triangular.
As we show in the proof of identifiability result in Theorem 1, Assumption 2, loosely speaking, ensures that the resulting covariance matrix does not contain "nontrivial" inequality constraints. In Example 1, support matrix  $\xi_{1}$  satisfies Assumption 2 and leads to an equality constraint, while matrix  $\xi_{2}$  fails to meet this assumption, resulting in an inequality constraint. The Lebesgue measure of the parameters leading to such inequality constraint is not zero, thus requiring additional assumptions to handle such cases. Therefore, we adopt Assumption 2 in this work and focus on equality constraints.
A key ingredient of our identifiability result based on sparsity is the dimension of the covariance set  $\Sigma(\pmb{\xi})$ . It may be natural to expect that the dimension of  $\Sigma(\pmb{\xi})$ , denoted as  $\dim(\Sigma(\pmb{\xi}))$ , equals the number of parameters used to specify the mixing matrices, i.e.,  $\|\pmb{\xi}\|_0$ . This is not the case for general mixing matrices, but we show that such property holds under Assumption 2.
Proposition 2 (Dimension of Covariance Set). Let  $\pmb{\xi}$  be a support matrix that satisfies Assumption 2. Then, its covariance set has a dimension of  $\| \pmb {\xi}\| _0$ , i.e.,  $\dim (\Sigma (\pmb {\xi})) = \| \pmb {\xi}\| _0$
Note that Assumption 2 allows independent row and column permutations, which thus may be rather mild especially for sparse mixing matrix. Below we provide an example of the connective structure that satisfies this assumption. We also introduce an efficient approach to verify whether a mixing matrix satisfies Assumption 2 in Appendix E.2.
Example 2. If the connective structure  $\mathcal{G}_{\mathbf{A}}$  of mixing matrix  $\mathbf{A}$  is a polytree, then matrix  $\mathbf{A}$  satisfies Assumption 2.
Finally, the following assumption is needed to ensure that the equality constraints arising from the covariance matrix are entailed by the true mixing matrix, rather than accidental parameter cancellations. This establishes a correspondence between equality constraints in the covariance matrix and those imposed by the support of the mixing matrix. Similar assumption has been employed in various tasks such as causal discovery [39, 20], as discussed in Section 3.4.
Assumption 3 (Faithfulness). For a mixing matrix  $\mathbf{A}$  and the resulting covariance matrix  $\boldsymbol{\Sigma}$ ,  $\boldsymbol{\Sigma}$  satisfies an equality constraint  $\kappa$  only if  $\kappa \in H(\pmb{\xi}_{\mathbf{A}})$ .
The following proposition justifies Assumption 3 and demonstrates that it is a generic property in the sense that it holds almost everywhere in the space of possible mixing matrices. In other words, it is only violated for a set of mixing matrices with zero Lebesgue measure.
Proposition 3 (Generic Property). Suppose that the nonzero coefficients of matrix  $\mathbf{A}$  are randomly drawn from a distribution that is absolutely continuous with respect to Lebesgue measure. Then, matrix  $\mathbf{A}$  satisfies Assumption 3 with probability one.
With the aforementioned assumptions in place, we are ready to present our main identifiability result of ICA: by making use of second-order statistics and sparsity constraint, we show that the true mixing matrix can be identified up to signed column permutation. The key intuition is as follows: although in general there is rotational invariance in second-order statistics, the sparsity of the estimated mixing matrix, under our assumptions above, is guaranteed to be denser than the true mixing matrix after any rotation (except permutation). Note that the proof leverages the technical tools developed by Ghassami et al. [20] which involve different types of rotations.
Theorem 1 (Identifiability with Sparsity). Suppose that the true mixing matrix  $\tilde{\mathbf{A}}$  satisfies Assumptions 1, 2, and 3. Let  $\hat{\mathbf{A}}$  be a solution of the following problem:
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {n \times n}} \| \mathbf {A} \| _ {0} \quad \text {s u b j e c t t o} \quad \mathbf {A} \mathbf {A} ^ {\top} = \tilde {\mathbf {A}} \tilde {\mathbf {A}} ^ {\top} \quad \text {a n d} \quad \mathbf {A} \text {s a t i s f i e s A s s u m p t i o n 2 .} \tag {2}
$$
Then, we have  $\hat{\mathbf{A}}\sim \tilde{\mathbf{A}}$
3.3 Related Sparsity Assumptions
As briefly discussed in Section 1, Zheng et al. [49] have also provided identifiability result of ICA without assuming non-Gaussianity. Their approach, similar to ours, relies on assumptions related to the support of the mixing matrix, which are provided below.
Assumption 4 (Zheng et al. [49]). For every set  $\mathcal{I} \subseteq [n]$  where  $|\mathcal{I}| > 1$  and for all  $i \in \mathcal{I}$ , we have
$$
\left| \bigcup_ {j \in \mathcal {I}} \operatorname {s u p p} (\mathbf {a} _ {j}) \right| - \operatorname {r a n k} (\operatorname {o v e r l a p} (\mathbf {A} _ {\mathcal {I}})) > | \operatorname {s u p p} (\mathbf {a} _ {i}) |. \tag {3}
$$
Assumption 5 (Zheng et al. [49]). For every  $i \in [n]$ , there exists set  $\mathcal{I} \subset [n]$  such that
$$
\bigcap_ {j \in \mathcal {I}} \operatorname {s u p p} (\mathbf {a} _ {j,:}) = \{i \}.
$$
While Assumptions 4 and 5 shed light on resolving the long-standing challenge of the rotational indeterminacy of Gaussian sources, their general applicability remains unclear. One notable restriction is that they require each column of the support matrix to not be a subset of the other, formalized below. This may be overly restrictive in practical scenarios and is not the case for our Assumption 1.
Assumption 6 (Column Subset). Each column in the support of  $\mathbf{A}$  is not a subset of the other.
Since the true generating process of real-world data is inaccessible, it is challenging to quantitatively evaluate the applicability of these sparsity assumptions. In light of this challenge, we demonstrate the significance and advantage of our Assumption 1 by proving that it is strictly weaker than Assumptions 4 and 5. This also strengthens the validity of our result (i.e., Proposition 1) that Assumption 1 is a necessary condition for achieving identifiability with second-order statistics and sparsity constraint.