SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 2. For mixing matrix  $\mathbf{A}$ , we have the following chain of chain of implications:
$$
\text {A s s u m p t i o n} 4 \Longrightarrow \text {A s s u m p t i o n} 6 \Longrightarrow \text {A s s u m p t i o n} 1.
$$
Furthermore, there exists a matrix  $\mathbf{A}$  satisfying Assumption 1 that does not satisfy Assumption 4.
Theorem 3. For mixing matrix  $\mathbf{A}$ , we have the following chain of chain of implications:
$$
\text {A s s u m p t i o n} 5 \Longrightarrow \text {A s s u m p t i o n} 6 \Longrightarrow \text {A s s u m p t i o n} 1.
$$
Furthermore, there exists a matrix  $\mathbf{A}$  satisfying Assumption 1 that does not satisfy Assumption 5.
Remark 2. Assumption 1 is not only strictly weaker than both Assumption 4 and 5, but also accommodates cases where one column of mixing matrix  $\mathbf{A}$  is a subset of another (provided that their supports differ in more than one entry). On the other hand, the latter two assumptions cannot accommodate such cases, further demonstrating the flexibility and broader scope of Assumption 1.
3.4 Connection with Causal Discovery
ICA has emerged as a useful tool for causal discovery over the past two decades [38]. In particular, Shimizu et al. [38] demonstrated that the identifiability of ICA based on non-Gaussianity can be leveraged to discover the complete structure of a linear non-Gaussian structural equation model (SEM). In this section, we establish the connection and provide an analogy between ICA and causal discovery from second-order statistics. This connection further bridges the gap between these two fields, and provides insights into the interpretation of our identifiability result.
Let  $\mathbb{R}_{\mathrm{off}}^{n\times n}$  be the set of matrices whose diagonal entries are zero, and  $\mathrm{diag}(\mathbb{R}_{>0}^n)$  be the set of positive diagonal matrices. Consider the linear SEM  $\mathbf{x} = \tilde{\mathbf{B}}^\top \mathbf{x} + \mathbf{e}$ , where  $\mathbf{x}$  denotes the random vector,  $\tilde{\mathbf{B}}\in \mathbb{R}_{\mathrm{off}}^{n\times n}$  denotes the weighted adjacency matrix representing a directed graph without self-loop, and  $\mathbf{e}$  is the independent noise vector with covariance matrix  $\tilde{\Omega}\in \mathrm{diag}(\mathbb{R}_{>0}^n)$ . The graph is often assumed to be a directed acyclic graph (DAG); in this case, two DAGs are said to be Markov equivalent if they share the same skeleton and v-structures [43], resulting in the same set of conditional independencies. Also, the inverse covariance matrix of  $\mathbf{x}$  is given by  $\tilde{\Theta} = (\mathbf{I} - \tilde{\mathbf{B}})\tilde{\Omega}^{-1}(\mathbf{I} - \tilde{\mathbf{B}})^{\top}$ . We refer readers to Spirtes et al. [39], Glymour et al. [21] for more details and a review of causal discovery.
Score-based method is a major class of causal discovery methods that optimizes a goodness-of-fit measure under a sparsity constraint [21], e.g., BIC [36]. In essence, score-based causal discovery from second-order statistics can often be formulated in the large sample limit as the following optimization problem (the commonly used acyclicity constraint is omitted here and will be clarified subsequently):
$$
\min  _ {\substack {\mathbf {B} \in \mathbb {R} _ {\text {off}} ^ {n \times n}, \\ \boldsymbol {\Omega} \in \operatorname {diag} (\mathbb {R} _ {> 0} ^ {n})}} \| \mathbf {B} \| _ {0} \quad \text {subject to} \quad (\mathbf {I} - \mathbf {B}) \boldsymbol {\Omega} ^ {- 1} (\mathbf {I} - \mathbf {B}) ^ {\top} = \tilde {\boldsymbol {\Theta}} = (\mathbf {I} - \tilde {\mathbf {B}}) \tilde {\boldsymbol {\Omega}} ^ {- 1} (\mathbf {I} - \tilde {\mathbf {B}}) ^ {\top}. \tag{4}
$$
By substituting  $\mathbf{A} \coloneqq (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$  into the above formulation, we obtain the ICA formulation with second-order statistics and sparsity constraint introduced in Problem (1). To establish a precise connection between formulations (4) and (1), we present the following theorem which indicates that these formulations can be translated into each other.
Theorem 4 (Equivalent Formulations). Suppose  $\tilde{\mathbf{A}} = (\mathbf{I} - \tilde{\mathbf{B}})\tilde{\Omega}^{-\frac{1}{2}}$ . Then, we have:
(a) Let  $(\hat{\mathbf{B}},\hat{\Omega})$  be a solution to Problem (4). Then,  $\hat{\mathbf{A}}\coloneqq (\mathbf{I} - \hat{\mathbf{B}})\hat{\boldsymbol{\Omega}}^{-\frac{1}{2}}$  is a solution to Problem (1).
(b) Let  $\hat{\mathbf{A}}$  be a solution to Problem (1). Then, there exist matrices  $\hat{\mathbf{B}} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$  and  $\hat{\Omega} \in \mathrm{diag}(\mathbb{R}_{>0}^n)$  such that  $\hat{\mathbf{A}} \sim (\mathbf{I} - \hat{\mathbf{B}})\hat{\boldsymbol{\Omega}}^{-\frac{1}{2}}$ , and  $(\hat{\mathbf{B}}, \hat{\boldsymbol{\Omega}})$  is a solution to Problem (4).
Thus, the formulations of causal discovery and ICA via second-order statistics and sparsity constraint share inherent similarities. The key difference lies in their respective goals—the former aims to estimate the support of  $\tilde{\mathbf{B}}$  up to a Markov equivalence class [39], while the latter aims to estimate  $\tilde{\mathbf{A}}$  up to signed column permutation. The other difference is that  $(\mathbf{I} - \tilde{\mathbf{B}})\tilde{\boldsymbol{\Omega}}^{-1}(\mathbf{I} - \tilde{\mathbf{B}})^{\top}$  represents the inverse covariance matrix  $\tilde{\Theta}$  of  $\mathbf{x}$  in causal discovery, while  $\tilde{\mathbf{A}}\tilde{\mathbf{A}}^{\top}$  represents the covariance matrix  $\tilde{\Sigma}$  of  $\mathbf{x}$  in ICA.
In addition to establishing the connection between formulations (4) and (1), we show that the assumptions we employ for identifiability of ICA, namely Assumptions 1, 2, and 3, are inherently related to causal discovery. Notably, Assumption 3 has been used in causal discovery [39, 20] to ensure that the conditional independencies in the distribution are entailed by the true directed graph. We now present a result that establishes the connection of Assumptions 1 and 2 with causal discovery.
Theorem 5. Suppose  $\mathbf{A} \sim (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$  for matrices  $\mathbf{A} \in \mathbb{R}^{n \times n}$ ,  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$ , and  $\Omega \in \mathrm{diag}(\mathbb{R}_{>0}^n)$ . Then,  $\mathbf{A}$  satisfies Assumptions 1 and 2 if and only if  $\mathbf{B}$  represents a DAG whose Markov equivalence class is a singleton.
In causal discovery, it is rather common to assume that the true directed graph is acyclic and accordingly incorporate an acyclicity constraint to formulation (4). As indicated in Theorem 5 (and Proposition 10 in Appendix D.8), this acyclicity assumption corresponds to Assumption 2 in the context of ICA. Therefore, Theorem 4 can be straightforwardly extended to show the equivalence between formulations (2) and (4) with an additional acyclicity constraint on matrix  $\mathbf{B}$ . Furthermore, it is worth noting that the mapping from mixing matrix  $\mathbf{A}$  satisfying Assumption 2 to a DAG is unique, which is straightforwardly implied by Shimizu et al. [38, Appendix A].
Proposition 4 (Shimizu et al. [38]). Suppose matrix  $\mathbf{A}$  is non-singular and satisfies Assumption 2. Then, there exist unique matrices  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$  and  $\Omega \in \mathrm{diag}(\mathbb{R}_{>0}^n)$  such that  $\mathbf{A} \sim (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$ . Furthermore, matrix  $\mathbf{B}$  represents a DAG.
Moreover, as indicated in Theorem 5 (and Proposition 11 in Appendix D.8), Assumption 1 implies that the Markov equivalence class of  $\mathbf{B}$  is a singleton; in this case, the true DAG can be completely identified. In particular, the Markov equivalence class of DAG is a singleton when all edges are either part of a v-structure or required to be oriented to avoid forming new v-structures or cycles [31, 4].
4 Estimation Methods with Second-Order Statistics
Building upon the identifiability result provided in Section 3, we propose two estimation methods that leverage second-order statistics and sparsity. These methods involve solving a continuous constrained optimization problem, which we discuss in detail in this section. First, in Section 4.1, we introduce a novel approach to formulate the search space in Problem (2) that enables the application of continuous optimization techniques. We then describe the proposed estimation methods in Section 4.2. All proofs are provided in Appendix D.
4.1 Characterization of Search Space
The key to achieving the identifiability result presented in Theorem (1) lies in the optimization problem (2), where the search space involves the matrices  $\mathbf{A}$  that satisfy Assumption 2. Consequently, a crucial question arises: is there an efficient approach for exploring the space of matrices  $\mathbf{A}$  that satisfy Assumption 2? Inspired by Zheng et al. [48], Wei et al. [45], Zhang et al. [47], we introduce the following function to characterize the search space:
$$
g (\mathbf {A}) = \operatorname {t r} \left(\sum_ {k = 2} ^ {n} (\operatorname {o f f} (\mathbf {A}) \odot \operatorname {o f f} (\mathbf {A})) ^ {k}\right), \quad \text {w h e r e} \quad (\operatorname {o f f} (\mathbf {A})) _ {i, j} = \left\{ \begin{array}{l l} 0, & \text {i f} i = j, \\ a _ {i, j}, & \text {o t h e r w i s e}. \end{array} \right.
$$
Here, symbol  $\odot$  denotes the Hadamard product. We then provide the following lemma that establishes the relationship between function  $g(\mathbf{A})$  and a specific type of permutation, namely simultaneous equal row and column permutation.
Lemma 1. For any matrix  $\mathbf{A}$ ,  $g(\mathbf{A}) = 0$  if and only if it can be permuted via simultaneous equal row and column permutations to be lower triangular.
Intuitively speaking, if we interpret matrix  $\mathbf{A}$  as a weighted adjacency matrix of a directed graph, say  $\mathcal{G}$ , then  $\mathrm{tr}((\mathrm{off}(\mathbf{A})\odot \mathrm{off}(\mathbf{A}))^k)$  counts the number of length- $k$  weighted closed walks in  $\mathcal{G}$  excluding the self-loops. Therefore,  $g(\mathbf{A})$  counts the total number of weighted closed walks in  $\mathcal{G}$  without including self-loops.  $g(\mathbf{A}) = 0$  then implies that  $\mathcal{G}$  does not contain any cycle longer than one (i.e., it may contain self-loops). It is known that a directed graph is acyclic if and only if its weighted adjacency matrix can be permuted via simultaneous equal row and column permutations to be strictly lower triangular. In our case,  $\mathcal{G}$  may contain self-loops, and thus can be permuted to a lower triangular form. This distinction elucidates the difference between our characterization and that introduced by Zheng et al. [48], Zhang et al. [47], i.e., their characterization focuses on matrices that are strictly lower triangular, while ours focuses on lower triangular matrices.
The following proposition sheds light on the connection between  $g(\mathbf{A})$  and Assumption 2.
Proposition 5. The matrix  $\mathbf{A}$  satisfies Assumption 2 if and only if there is a matrix  $\hat{\mathbf{A}}$  such that it is a column permutation of  $\mathbf{A}$  and that  $g(\hat{\mathbf{A}}) = 0$ .
The above proposition indicates that the search for matrices  $\mathbf{A}$  satisfying Assumption 2 can be effectively conducted by considering the constraint  $g(\mathbf{A}) = 0$ . Accordingly, we establish an alternative formulation of the identifiability result presented in Section 3.2. In the following section, we will introduce efficient approaches for solving Problem (5).
Theorem 6 (Alternative Formulation of Identifiability). Suppose that the true mixing matrix  $\tilde{\mathbf{A}}$  satisfies Assumptions 1, 2, and 3. Let  $\hat{\mathbf{A}}$  be a solution of the following problem:
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {n \times n}} \| \mathbf {A} \| _ {0} \quad \text {s u b j e c t t o} \quad \mathbf {A} \mathbf {A} ^ {\top} = \tilde {\mathbf {A}} \tilde {\mathbf {A}} ^ {\top} \quad \text {a n d} \quad g (\mathbf {A}) = 0. \tag {5}
$$
Then, we have  $\hat{\mathbf{A}}\sim \bar{\mathbf{A}}$
Algorithm 1 Decomposition-Based Method
Require: initial penalty coefficient  $c_{1} > 0$ ; multiplicative factor  $\beta > 1$ ; maximum number of iterations  $k_{\mathrm{max}} > 0$ ; tolerance  $\epsilon_{1}, \epsilon_{2} > 0$ ; initial solution  $\mathbf{A}_0$ ; empirical covariance matrix  $\bar{\Sigma}$
1: for  $k = 1,2,\ldots ,k_{\max}$  do
2: Solve  $\mathbf{A}_k\coloneqq \arg \min_{\mathbf{A}\in \mathbb{R}^{n\times n}}\rho (\mathbf{A}) + \frac{c_k}{2}\| \mathbf{A}\mathbf{A}^\top -\bar{\mathbf{\Sigma}}\| _F^2 +\frac{c_k}{2} g(\mathbf{A})^2$  initialized at  $\mathbf{A}_{k - 1}$
3: if  $\| \mathbf{A}_k\mathbf{A}_k^\top -\bar{\mathbf{\Sigma}}\| _F^2 <  \epsilon_1$  and  $g(\mathbf{A}_k) <   \epsilon_2$  then break
4: Update penalty coefficient  $c_{k+1} \coloneqq \beta c_k$
5: Output solution  $\mathbf{A}_k$
Algorithm 2 Likelihood-Based Method
Require: initial penalty coefficient  $c_{1} > 0$ ; multiplicative factor  $\beta > 1$ ; maximum number of iterations  $k_{\mathrm{max}} > 0$ ; tolerance  $\epsilon > 0$ ; initial solution  $\mathbf{A}_0$ ; empirical covariance matrix  $\bar{\Sigma}$
1: for  $k = 1,2,\ldots ,{k}_{\max }$  do
2: Solve  $\mathbf{A}_k\coloneqq \arg \min_{\mathbf{A}\in \mathbb{R}^{n\times n}}L(\mathbf{A};\bar{\boldsymbol{\Sigma}}) + \rho (\mathbf{A}) + \frac{c_k}{2} g(\mathbf{A})^2$  initialized at  $\mathbf{A}_{k - 1}$
3: if  $g(\mathbf{A}_k) < \epsilon$  then break
4: Update penalty coefficient  $c_{k+1} \coloneqq \beta c_k$
5: Output solution  $\mathbf{A}_k$
4.2 Estimation Methods
Based on the identifiability results of Theorems 1 and 6, we propose two estimation methods, called SparseICA, to perform ICA from second-order statistics that leverage sparsity regularization and continuous constrained optimization. To proceed, we define  $\bar{\Sigma}$  as the empirical covariance matrix of observed variables  $\mathbf{x}$  and  $T$  as the sample size.
Decomposition-based method. Given the formulation in Eq. (5), we consider the following constrained optimization problem
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {n \times n}} \rho (\mathbf {A}) \quad \text {s u b j e c t t o} \quad \mathbf {A} \mathbf {A} ^ {\top} - \bar {\boldsymbol {\Sigma}} = \mathbf {0} \quad \text {a n d} \quad g (\mathbf {A}) = 0, \tag {6}
$$
where  $\rho(\mathbf{A})$  is a suitable sparsity regularizer, often expressible as  $\rho(\mathbf{A}) = \sum_{i,j} \rho(a_{i,j})$ . Formulation (5) indicates that one should apply the  $\ell_0$  regularizer  $\rho(\mathbf{A}) = \| \mathbf{A} \|_0$ . Alternatively, other possible choices include the  $\ell_1$  regularizer  $\rho(\mathbf{A}) = \| \mathbf{A} \|_1$  that supports continuous optimization. Further details regarding our specific choice of sparsity regularizer will be elaborated later in this section. On the other hand, we simply use the empirical covariance matrix  $\tilde{\Sigma}$  as an estimate of the true covariance matrix  $\tilde{\Sigma}$ , which is found to work well across different sample sizes in our experiments. One may also adopt a regularized estimator of the form  $\tilde{\Sigma} + \eta \mathbf{I}$  with a proper choice of  $\eta$ , which may have notable advantage in certain cases [13, 28, 37].
Likelihood-based method. In addition to the decomposition-based method above, we introduce a likelihood-based estimation method formulated by the following constrained optimization problem:
$$
\min  _ {\mathbf {A} \in \mathbb {R} ^ {n \times n}} L (\mathbf {A}; \bar {\boldsymbol {\Sigma}}) + \rho (\mathbf {A}) \quad \text {s u b j e c t t o} \quad g (\mathbf {A}) = 0, \tag {7}
$$
$$
\text {w h e r e} L (\mathbf {A}; \bar {\boldsymbol {\Sigma}}) = \frac {T}{2} \operatorname {t r} \left(\left(\mathbf {A} ^ {\top} \mathbf {A}\right) ^ {- 1} \bar {\boldsymbol {\Sigma}}\right) + T \log | \det  \mathbf {A} |
$$
is the negative Gaussian log-likelihood function and  $\rho (\mathbf{A})$  is a sparsity regularizer. The following result establishes the theoretical guarantee of this likelihood-based method.
Theorem 7 (Likelihood-Based Method). Suppose that the true mixing matrix  $\tilde{\mathbf{A}}$  satisfies Assumptions 1, 2, and 3. Let  $\hat{\mathbf{A}}$  be a solution of Problem (7) with sparsity regularizer  $\rho(\mathbf{A}) = 0.5\|\mathbf{A}\|_0\log T$ . Then, we have  $\hat{\mathbf{A}} \sim \tilde{\mathbf{A}}$  in the large sample limit.
Implementation. Based on Theorems 6 and 7, ideally one should adopt the  $\ell_0$  regularizer  $\rho(\mathbf{A}) = \lambda \| \mathbf{A} \|_0$  and develop an exact discrete search procedure over the support space of matrix  $\mathbf{A}$ . However, such approach may pose computational challenges in practice. Since the functions  $L(\mathbf{A}; \bar{\Sigma})$  and  $g(\mathbf{A})$  are differentiable, in this work we develop an estimation procedure that leverages efficient continuous optimization techniques. Therefore, some possible choices for  $\rho(\mathbf{A})$  are  $\ell_1$ , smoothly
clipped absolute deviation (SCAD) [17], and minimax concave penalty (MCP) [46] regularizers. The  $\ell_1$  regularizer has been shown to exhibit bias during estimation [17, 10], especially for large coefficients. Here, we adopt the MCP regularizer that is less susceptible to such issue, given by
$$
\rho (a _ {i, j}) = \left\{ \begin{array}{l l} \lambda | a _ {i, j} | - \frac {a _ {i , j} ^ {2}}{2 \alpha}, & \text {i f} | a _ {i, j} | \leq \alpha \lambda , \\ \frac {\alpha \lambda^ {2}}{2}, & \text {o t h e r w i s e}, \end{array} \right.
$$
where  $\lambda$  and  $\alpha$  are hyperparameters.
To solve Eqs. (6) and (7), standard constrained optimization methods can be used, such as quadratic penalty method, augmented Lagrangian method, and barrier method [7, 8, 33]. In this work, we adopt the quadratic penalty method that converts each constrained problem into a sequence of unconstrained optimization problems where the constraint violations are increasingly penalized. We describe the full procedure of the decomposition-based and likelihood-based methods based on quadratic penalty method in Algorithms 1 and 2, respectively. The unconstrained problem in each iteration can be solved using different continuous optimization solvers, including first-order methods such as gradient descent and steepest descent, as well as second-order methods such as quasi-Newton methods. In our experiments presented in the subsequent section, we employ L-BFGS [11], a quasi-Newton method, to solve the unconstrained optimization problem.
It is worth noting that the formulations in Eqs. (6) and (7) involve solving nonconvex optimization problems; in practice, the optimization procedure may return stationary points that correspond to suboptimal local solutions. Therefore, we run the method for a number of times and choose the final mixing matrix via model selection. Further details regarding the optimization procedure and implementation are provided in Appendix F.
To empirically validate our proposed identifiability results, we carry out experiments under various settings. We also conduct ablation studies to verify the necessity of the proposed assumptions and include FastICA [23] as a representative baseline. Specifically, we consider the following methods:
- SparseICA: Decomposition-based (Eq. (6)) or likelihood-based (Eq. (7)) method on data where both Assumptions 1 and 2 hold;
- Vanilla: Decomposition-based (Eq. (6)) or likelihood-based (Eq. (7)) method without the constraint  $g(\mathbf{A}) = 0$ , on data where neither Assumption 1 nor Assumption 2 holds;
- FastICA-D: FastICA on data where both Assumptions 1 and 2 hold;
- FastICA: FastICA on data where neither Assumption 1 nor Assumption 2 holds.
For all experiments, we simulate 10 sources, and generate the supports of the true mixing matrices  $\tilde{\mathbf{A}}$  according to the assumptions required by each method above. The nonzero entries of  $\tilde{\mathbf{A}}$  are sampled uniformly at random from  $[-0.8, -0.2] \cup [0.2, 0.8]$ . We use mean correlation coefficient (MCC) and Amari distance [3] as evaluation metrics, where all results are reported for 10 random trials.
Different sample sizes. We first consider entirely Gaussian sources and different sample sizes. The empirical results of MCC are shown in Figure 1, while those of Amari distance are given in Figure 4 in Appendix G. By comparing SparseICA with Vanilla and FastICA, it is evident that the identification performance is much better across different sample sizes when the required assumptions on the connective structure are satisfied, as validated by Wilcoxon signed-rank test at  $5\%$  significance level. Furthermore, the unsatisfactory results of FastICA-D indicate that our estimation methods are also essential for ensuring the quality of the identification, which further validates the proposed identifiability theory. Since FastICA-D performs similarly to FastICA, it suggests that the data-generating process, while meeting our assumption, may not be inherently simpler to recover without considering specific procedure to handle Gaussian sources. In addition, as expected, the performance of SparseICA improves in terms of both MCC and Amari distance as the sample size increases.
Different ratios of Gaussian sources. We now conduct empirical study to investigate the performance in the presence of Gaussian and non-Gaussian sources. Here, the non-Gaussian sources follow exponential distributions. We consider different ratios of Gaussian sources, which are specifically 0, 0.2, 0.4, 0.6, 0.8, and 1. For instance, ratio of 0.6 indicates that there are 6 Gaussian sources
(a) Likelihood-based method.
(b) Decomposition-based method.
(a) Likelihood-based method.
Figure 2: Empirical results of MCC across different ratios of Gaussian sources. Error bars indicate the standard errors calculated based on 10 random trials.
Figure 1: Empirical results of MCC across different sample sizes. Error bars indicate the standard errors calculated based on 10 random trials.
(b) Decomposition-based method.
and 4 non-Gaussian sources. The empirical results of MCC based on 1000 samples are depicted in Figure 1, while those of Amari distance are provided in Figure 4 in Appendix G. One observes that the identification performance SparseICA is rather stable across different ratios of Gaussian sources, which may not be surprising as it leverages only second-order statistics. On the other hand, the performance of FastICA-D and FastICA deteriorates as the ratio of Gaussian sources increases, because it relies on non-Gaussianity of the sources. It is also observed that, in the presence of Gaussian sources, FastICA-D and FastICA may perform well, provided that the ratio (or number) of Gaussian sources is not large. This suggests a potential future direction to integrate our method based on second-order statistics with existing methods that rely on non-Gaussianity, which may better handle both Gaussian and non-Gaussian sources.
We develop an identifiability theory of ICA from second-order statistics without relying on non-Gaussianity. Specifically, we introduce novel and precise assumptions on the connective structure from sources and observed variables, and show that our proposed assumption of structural variability is strictly weaker than the previous ones. Importantly, we prove that this assumption is one of the necessary conditions for achieving identifiability in the investigated setting. We further propose two estimation methods based on second-order statistics that leverage sparsity regularization. Moreover, we establish a precise connection between our identifiability result of ICA and causal discovery from second-order statistics, which may open up avenues for exploring the interplay between ICA and causal discovery with linear Gaussian SEM. Our theoretical claims have also been empirically validated across different settings. The limitations include the lack of finite sample analysis and broader application of our theory in more real-world tasks, which are worth exploring in future work.