SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 12 (Wei et al. [45, Theorem 1]). For any matrix  $\mathbf{A}$ ,  $f(\mathbf{A}) = 0$  if and only if it can be permuted via simultaneous equal row and column permutations to be strictly lower triangular.
We now provide the proof for Lemma 1.
Proof. We first define  $\mathbf{D}_{\mathbf{A}}$  as a diagonal matrix of the same size as matrix  $\mathbf{A}$ , where its diagonal entries are equal to those of  $\mathbf{A}$  and its non-diagonal entries are zero. Clearly, we have  $\mathbf{A} = \mathrm{off}(\mathbf{A}) + \mathbf{D}_{\mathbf{A}}$ . For any matrix  $\mathbf{P}$ , This implies
$$
\mathbf {P} ^ {\top} \mathbf {A} \mathbf {P} = \mathbf {P} ^ {\top} \operatorname {o f f} (\mathbf {A}) \mathbf {P} + \mathbf {P} ^ {\top} \mathbf {D} _ {\mathbf {A}} \mathbf {P}. \tag {31}
$$
Suppose that there exists permutation matrix  $\mathbf{P}$  such that  $\mathbf{P}^{\top}\mathbf{A}\mathbf{P}$  is lower triangular. By Eq. (31), we have  $\mathbf{P}^{\top}\mathrm{off}(\mathbf{A})\mathbf{P} = \mathbf{P}^{\top}\mathbf{A}\mathbf{P} - \mathbf{P}^{\top}\mathbf{D}_{\mathbf{A}}\mathbf{P}$ . Clearly, the diagonal entries of  $\mathbf{P}^{\top}\mathbf{A}\mathbf{P}$  are exactly the same as those of  $\mathbf{P}^{\top}\mathbf{D}_{\mathbf{A}}\mathbf{P}$ , which cancel out each other. Therefore, the diagonal entries of  $\mathbf{P}^{\top}\mathrm{off}(\mathbf{A})\mathbf{P}$  are zeros, indicating that it is strictly lower triangular. By Lemma 12, this implies  $f(\mathrm{off}(\mathbf{A})) = 0$ , and thus  $g(\mathbf{A}) = 0$  by Eq. (30).
Suppose  $g(\mathbf{A}) = f(\mathrm{off}(\mathbf{A})) = 0$ . By Lemma 12, there exists permutation matrix  $\mathbf{P}$  such that  $\mathbf{P}^{\top}\mathrm{off}(\mathbf{A})\mathbf{P}$  is strictly lower triangular. Clearly,  $\mathbf{P}^{\top}\mathbf{D}_{\mathbf{A}}\mathbf{P}$  is a diagonal matrix. By Eq. (31),  $\mathbf{P}^{\top}\mathbf{A}\mathbf{P}$  is lower triangular, i.e.,  $\mathbf{A}$  can be permuted via simultaneous equal row and column permutations to be lower triangular.
D.10 Proof of Proposition 5
Suppose that there exists matrix  $\hat{\mathbf{A}}$  such that it is a column permutation of  $\mathbf{A}$  and that  $g(\hat{\mathbf{A}}) = 0$ . By definition, there exists permutation matrix  $\mathbf{P}_2$  such that  $\hat{\mathbf{A}} = \mathbf{AP}_2$ . Also, by Lemma 1, there exists permutation matrix  $\mathbf{P}_1$  such that  $\mathbf{P}_1^\top \hat{\mathbf{A}}\mathbf{P}_1$  is lower triangular, which implies that  $\mathbf{P}_1^\top \mathbf{AP}_2\mathbf{P}_1$  is lower triangular. Clearly,  $\mathbf{P}_2\mathbf{P}_1$  is also a permutation matrix. Therefore, matrix  $\mathbf{A}$  can be permuted by row and column permutations to be lower triangular, indicating that it satisfies Assumption 2.
Suppose that matrix  $\mathbf{A}$  satisfies Assumption 2, i.e., there exist permutation matrices  $\mathbf{P}_1$  and  $\mathbf{P}_2$  such that  $\mathbf{P}_1^\top \mathbf{A}\mathbf{P}_2$  is lower triangular. Defining  $\mathbf{P}_3 \coloneqq \mathbf{P}_2\mathbf{P}_1^{-1}$ , which is also a permutation matrix,
and substituting it into the previous statement, we see that  $\mathbf{P}_1^\top \mathbf{A}\mathbf{P}_3\mathbf{P}_1$  is lower triangular. Further substitution of  $\hat{\mathbf{A}} \coloneqq \mathbf{A}\mathbf{P}_3$  implies that  $\mathbf{P}_1^\top \hat{\mathbf{A}}\mathbf{P}_1$  is lower triangular, which, by Lemma 1, indicates  $g(\hat{\mathbf{A}}) = 0$ . Clearly,  $\hat{\mathbf{A}}$  is a column permutation of  $\mathbf{A}$ .
D.11 Proof of Theorem 6
Proof. Let  $\hat{\mathbf{A}}$  be a solution to Problem (5). Suppose by contradiction that it is not a solution to Problem (2). That is, there exists matrix  $\hat{\mathbf{A}}$  satisfying Assumption 2 such that
$$
\left\| \ddot {\mathbf {A}} \right\| _ {0} <   \left\| \hat {\mathbf {A}} \right\| _ {0} \quad \text {a n d} \quad \ddot {\mathbf {A}} \ddot {\mathbf {A}} ^ {\top} = \tilde {\mathbf {A}} \tilde {\mathbf {A}} ^ {\top}. \tag {32}
$$
By Proposition 5, there exists permutation matrix  $\mathbf{P}$  such that  $g(\mathbf{\dot{A}}\mathbf{P}) = 0$ . Furthermore, by Eq. (32), we have
$$
\| \ddot {\mathbf {A}} \mathbf {P} \| _ {0} <   \| \hat {\mathbf {A}} \| _ {0} \quad \text {a n d} \quad (\ddot {\mathbf {A}} \mathbf {P}) (\ddot {\mathbf {A}} \mathbf {P}) ^ {\top} = \tilde {\mathbf {A}} \tilde {\mathbf {A}} ^ {\top}.
$$
Therefore,  $\mathbf{A}\mathbf{P}$  satisfies the constraint of Problem (5) and leads to a smaller zero norm for the objective function, and thus  $\hat{\mathbf{A}}$  will never be a solution of Problem (5), which is a contradiction. Therefore,  $\hat{\mathbf{A}}$  must be a solution to Problem (2). Since matrix  $\tilde{\mathbf{A}}$  also satisfies Assumptions 1, 2, and 3, applying Theorem 1 completes the proof.
D.12 Proof of Theorem 7
Theorem 7 (Likelihood-Based Method). Suppose that the true mixing matrix  $\tilde{\mathbf{A}}$  satisfies Assumptions 1, 2, and 3. Let  $\hat{\mathbf{A}}$  be a solution of Problem (7) with sparsity regularizer  $\rho(\mathbf{A}) = 0.5\|\mathbf{A}\|_0 \log T$ . Then, we have  $\hat{\mathbf{A}} \sim \tilde{\mathbf{A}}$  in the large sample limit.
Proof. First, we have  $g(\hat{\mathbf{A}}) = 0$  and, in the large sample limit,  $\tilde{\Sigma} = \tilde{\Sigma}$ . Similar to BIC [36], the likelihood term dominates in the large sample limit as the weight of the likelihood function increases much faster than that of the sparsity regularizer. Therefore, in the large sample limit, we have  $\hat{\mathbf{A}}\hat{\mathbf{A}} = \tilde{\Sigma} = \tilde{\mathbf{A}}\tilde{\mathbf{A}}$ . Also, for any matrix  $\mathbf{A}$  that satisfies  $\mathbf{A}\mathbf{A} = \tilde{\mathbf{A}}\tilde{\mathbf{A}}$  and  $g(\mathbf{A}) = 0$ , the sparsity regularizer indicates  $\| \hat{\mathbf{A}} \|_0 \leq \| \mathbf{A} \|_0$ , because otherwise  $\hat{\mathbf{A}}$  will never be a solution of Problem (7). This implies that  $\hat{\mathbf{A}}$  is also a solution to Problem (5). Since matrix  $\tilde{\mathbf{A}}$  also satisfies Assumptions 1, 2, and 3, applying Theorem 1 completes the proof.
E Supplementary Discussion on Structural Assumptions
E.1 Examples Satisfying Structural Variability Assumption
To illustrate the intuition of the proposed assumption of structural variability (i.e., Assumption 1), we provide several examples on the connective structure from sources to observed variables (which corresponds to the support of mixing matrix) satisfying that assumption, as illustrated in Figure 3.
E.2 Efficient Approach for Verifying Assumption 2
Assumption 2 involves finding a certain combination of row and column permutations for mixing matrix  $\mathbf{A}$ , which may at first appear inefficient to verify. We provide a more efficient way to do so, by leveraging the interpretation of our assumptions in the context of causal discovery (see Section 3.4 for detailed discussion). Specifically, we provide the following corollary that is a straightforward consequence of Lemma 2 and Proposition 10, whose proof is omitted.
Corollary 3 (Verification of Assumption 2). Let  $\mathbf{A}$  be a non-singular matrix and  $\mathbf{P}$  be a permutation matrix such that the diagonal entries of  $\mathbf{AP}$  are nonzero. Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  be a directed graph where  $\mathcal{V} = \{v_{1},\dots,v_{n}\}$  and  $\mathcal{E} = \{v_{j}\to v_{i}:(\mathbf{AP})_{i,j}\neq 0,i\neq j\}$ . Then, directed graph  $\mathcal{G}$  is acyclic if and only if matrix  $\mathbf{A}$  satisfies Assumption 2.
(c)
Figure 3: Graphical representations of examples that satisfy Assumption 1.
Specifically, Corollary 3 implies that it suffices to find a column permutation such that the diagonal entries of the permuted matrix are nonzero. Note that such column permutation is guaranteed to exist as indicated by Lemma 2. We then construct a directed graph  $\mathcal{G}$  based on such permuted matrix and check if  $\mathcal{G}$  contains cycle, e.g., via depth-first search. Instead of searching for a certain combination of row and column permutations, this procedure may be more efficient because it involves only finding specific column permutations.
F Supplementary Estimation Details
We provide the estimation details for the methods described in Section 4.2. In our experiments, we use the average log-likelihood  $\frac{1}{T} L(\mathbf{A};\bar{\Sigma})$  as the objective (instead of  $L(\mathbf{A};\bar{\Sigma})$  in Eq. (7)) for likelihood-based method. For the sparsity term  $\rho (\mathbf{A})$ , we use MCP with hyperparameters  $\lambda = 1,\alpha = 40$  and  $\lambda = 0.1,\alpha = 10$  for decomposition-based and likelihood-based methods, respectively.
Furthermore, for both methods, we use the L-BFGS algorithm [11] implemented in SciPy [44] to solve each unconstrained optimization problem of quadratic penalty method. Since the formulations involve solving nonconvex optimization problems, we run L-BFGS with 30 random initializations, where each entry of the initial solution  $\mathbf{A}_0$  is sampled uniformly at random from  $[-0.1, 0.1]$ . In this case, the final solution is chosen via model selection. For quadratic penalty method, we use  $c_1 = 10^{-5}$  and  $c_1 = 10^{-2}$  for decomposition-based and likelihood-based methods, respectively, and use  $\beta = 1.5$  for both methods.
Lastly, we also use a threshold of 0.01 to remove small weights in the estimated mixing matrix. We run each of the experiments on 12 CPUs and 8 GBs of memory.
Computational complexity. To compute the constraint term  $g(\mathbf{A})$ , a straightforward approach is to compute each matrix power in  $g(\mathbf{A})$  and then sum their traces up, which requires  $O(n)$  matrix multiplications. In our implementation, we adopt a more efficient approach with computational complexity of  $O(\log n)$  matrix multiplications, inspired by Zhang et al. (2020). The rough idea is to perform exponentiation by squaring (i.e., a procedure similar to binary search) and recursively compute the term  $g(\mathbf{A})$ .
Furthermore, each L-BFGS run has a computational complexity of  $O(m^{2}n^{2} + m^{3} + mn^{2}t)$ , where  $m \ll n^2$  is the memory size and  $t$  is the number of inner iterations of the L-BFGS run. Typically, we have  $t = 250$  for each L-BFGS run, and 125 iterations for the quadratic penalty method.
G Supplementary Experimental Results
In addition to the MCC reported in Section 5, we report the Amari distance to evaluate the identification performance in Figures 4 and 5, respectively.
(b) Decomposition-based method.
Figure 4: Empirical results of Amari distance across different sample sizes. Error bars indicate the standard errors calculated based on 10 random trials.
(a) Likelihood-based method.
Figure 5: Empirical results of Amari distance across different ratios of Gaussian sources. Error bars indicate the standard errors calculated based on 10 random trials.