SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The authors would like to thank the anonymous reviewers for helpful comments and suggestions. This project is partially supported by NSF Grant 2229881, the National Institutes of Health (NIH) under Contract R01HL159805, a grant from Apple Inc., a grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
[1] N. Abrahamsen and P. Rigollet. Sparse gaussian ICA. arXiv preprint arXiv:1804.00408, 2018.
[2] L. Albera, A. Ferreol, P. Chevalier, and P. Comon. ICAR: a tool for blind source separation using fourth-order statistics only. IEEE Transactions on Signal Processing, 53(10):3633-3643, 2005.
[3] S.-i. Amari, A. Cichocki, and H. Yang. A new learning algorithm for blind signal separation. In Advances in Neural Information Processing Systems, 1995.
[4] S. A. Andersson, D. Madigan, and M. D. Perlman. A characterization of Markov equivalence classes for acyclic digraphs. The Annals of Statistics, 25(2):505-541, 1997.
[5] A. Belouchrani, K. Abed-Meraim, J.-F. Cardoso, and E. Moulines. A blind source separation technique using second-order statistics. IEEE Transactions on signal processing, 45(2):434-444, 1997.
[6] R. Benedetti and J.-J. Risler. Real algebraic and semi-algebraic sets. Actualités mathématiques. Hermann, Paris, 1990.
[7] D. P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods. Academic Press, 1982.
[8] D. P. Bertsekas. Nonlinear Programming. Athena Scientific, 2nd edition, 1999.
[9] A. Biton, I. Bernard-Pierrot, Y. Lou, C. Krucker, E. Chapeaublanc, C. Rubio-Pérez, N. López-Bigas, A. Kamoun, Y. Neuzillet, P. Gestraud, L. Grieco, S. Rebouissou, A. de Reyniès, S. Benhamou, T. Lebret, J. Southgate, E. Barillot, Y. Allory, A. Zinovyev, and F. Radvanyi. Independent component analysis uncovers the landscape of the bladder tumor transcriptome and reveals insights into luminal and basal subtypes. Cell Reports, 9(4):1235-1245, Nov 2014.
[10] P. Breheny and J. Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The Annals of Applied Statistics, 5(1):232-253, 2011.
[11] R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5):1190-1208, 1995.
[12] P. Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
[13] M. J. Daniels and R. E. Kass. Shrinkage estimators for covariance matrices. Biometrics, 57(4): 1173-1184, 2001.
[14] M. Drton. Algebraic problems in structural equation modeling. Advanced Studies in Pure Mathematics, 77, 2018.
[15] M. Drton, B. Sturmfels, and S. Sullivant. Algebraic factor analysis: Tetrads, pentads and beyond. Probability Theory and Related Fields, 138, 09 2005.
[16] M. Drton, B. Sturmfels, and S. Sullivant. Lectures on Algebraic Statistics, volume 39 of Oberwolfach Seminars. Springer, 2009.
[17] J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456):1348-1360, 2001.
[39] P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction, and Search. MIT press, 2nd edition, 2001.
[40] S. Sullivant. Algebraic geometry of gaussian Bayesian networks. Advances in Applied Mathematics, 40(4):482-513, 2008. ISSN 0196-8858.
[41] A. E. Teschendorff, M. Journée, P. A. Absil, R. Sepulchre, and C. Caldas. Elucidating the altered transcriptional programs in breast cancer using independent component analysis. *PLoS Comput. Biol.*, 3(8):1539–1554, 2007.
[42] L. N. Trefethen and D. Bau. Numerical linear algebra, volume 50. SIAM, 1997.
[43] T. Verma and J. Pearl. Equivalence and synthesis of causal models. In Conference on Uncertainty in Artificial Intelligence, 1990.
[44] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. Jarrod Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. Carey, I. Polat, Y. Feng, E. W. Moore, J. Vand erPlas, D. Laxalde, J. Perktold, R. Cirmrnan, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and S... Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17: 261-272, 2020.
[45] D. Wei, T. Gao, and Y. Yu. DAGs with no fears: A closer look at continuous optimization for learning Bayesian networks. In Advances in Neural Information Processing Systems, 2020.
[46] C.-H. Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2):894-942, 2010.
[47] Z. Zhang, I. Ng, D. Gong, Y. Liu, E. M. Abbasnejad, M. Gong, K. Zhang, and J. Q. Shi. Truncated matrix power iteration for differentiable DAG learning. In Advances in Neural Information Processing Systems, 2022.
[48] X. Zheng, B. Aragam, P. Ravikumar, and E. P. Xing. DAGs with NO TEARS: Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, 2018.
[49] Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022.
A Preliminaries on Support Rotation 14
B Proofs of Useful Lemmas 15
B.1 Proof of Lemma 2 15
B.2 Proof of Lemma 3 15
B.3 Proof of Lemma 5 16
B.4 Proof of Lemma 6 16
B.5 Proof of Lemma 8 16
C Proof of Theorem 1 17
C.1 Dimension of Covariance Set 17
C.2 Covariance Equivalence and Column Permutation of Support 19
C.3 Equality Constraints and Covariance Equivalence 19
C.4 Identifiability of Parameters from Support 20
D Proofs of Other Results 21
D.1 Proof of Example 1 21
D.2 Proof of Proposition 1 22
D.3 Proof of Example 2 22
D.4 Proof of Proposition 3 22
D.5 Proof of Theorem 2 23
D.6 Proof of Theorem 3 24
D.7 Proof of Theorem 4 24
D.8 Proof of Theorem 5 25
D.9 Proof of Lemma 1 28
D.10 Proof of Proposition 5 28
D.11 Proof of Theorem 6 29
D.12 Proof of Theorem 7 29
E Supplementary Discussion on Structural Assumptions 29
E.1 Examples Satisfying Structural Variability Assumption 29
E.2 Efficient Approach for Verifying Assumption 2 29
F Supplementary Estimation Details 30
G Supplementary Experimental Results 30
A Preliminaries on Support Rotation
Before presenting the proofs of the theoretical results, we review the definition of support rotation and its potential effects on a support matrix, given by Ghassami et al. [20]. Note that the definition and remark below are quoted from Ghassami et al. [20, Section 3.1], with only minor modifications.
Definition 3 (Support Rotation [20]). The support rotation, denoted as  $R(i,j,k)$ , is a transformation that modifies a support matrix  $\pmb{\xi}$  by applying a Givens rotation in the  $(j,k)$  plane, setting the element  $\xi_{i,j}$  to zero. The outcome of is the support matrix of  $\mathbf{Q}G(j,k,\tan^{-1}(-q_{i,j} / q_{i,k}))$ , where  $\mathbf{Q} \in \arg \max_{\mathbf{Q}'} \left| \operatorname{supp}\left(\mathbf{Q}'G(j,k,\tan^{-1}\left(-q_{i,j}' / q_{i,k}'\right))\right)\right|$  such that the support matrix of  $\mathbf{Q}'$  is  $\pmb{\xi}$ .
Note that  $G\left(j,k,\tan^{-1}\left(-q_{ij}^{\prime} / q_{i,k}^{\prime}\right)\right)$  is the Givens rotation in the  $(j,k)$  plane that makes the  $q_{i,j}^{\prime}$  entry zero.
Remark 3 (Ghassami et al. [20]). The effects of applying a support rotation  $R(i,j,k)$  can be categorized into the following four cases:
- Reduction: If  $\xi_{i,j} = \xi_{i,k} = \times$  and  $\xi_{l,j} = \xi_{l,k}$  for all  $l\in [p]\backslash \{i\}$ , then only  $\xi_{i,j}$  becomes zero.
- Reversible acute rotation: If  $\xi_{i,j} = \xi_{i,k} = \times$  and there exists a row  $i'$  such that the  $j$ -th and  $k$ -th columns differ only in that row, then  $\xi_{i,j}$  becomes zero and both  $\xi_{i',j}$  and  $\xi_{i',k}$  become  $\times$ .
- Irreversible acute rotation: If  $\xi_{i,j} = \xi_{i,k} = \times$  and the  $j$ -th and  $k$ -th columns differ in at least two rows, then  $\xi_{i,j}$  becomes zero and all entries on the  $j$ -th and  $k$ -th columns become  $\times$  on the rows on which they differed.
- Column swap: If  $\xi_{i,j} = \times$  and  $\xi_{i,k} = 0$ , then columns  $j$  and  $k$  are swapped.
B Proofs of Useful Lemmas
We provide several lemmas that will be useful for subsequent proofs.
Lemma 2 (Nonzero Diagonal Entries). For any non-singular matrix  $\mathbf{A}$ , there exists a permutation matrix  $\mathbf{P}$  such that the diagonal entries of  $\mathbf{AP}$  are nonzero.
Proof. We prove it by contradiction. Suppose that there always exists a zero diagonal entry in every column permutation.
We first represent the determinant of the matrix  $\mathbf{A}$  as its Leibniz formula:
$$
\det (\mathbf {A}) = \sum_ {\sigma \in \mathcal {S} _ {n}} \left(\operatorname {s g n} (\sigma) \prod_ {i = 1} ^ {n} a _ {i, \sigma (i)}\right),
$$
where  $S_{n}$  is the set of  $n$ -permutations. Because for every permutation, there always exists a diagonal entry with value zero, we have
$$
\prod_ {i = 1} ^ {n} a _ {i, \sigma (i)} = 0, \quad \forall \sigma \in \mathcal {S} _ {n}.
$$
Thus, it follows that  $\operatorname{det}(\mathbf{A}) = 0$ . This indicates that the matrix  $\mathbf{A}$  is singular, which leads to a contradiction.
B.2 Proof of Lemma 3
Lemma 3 (Non-Singular Solution). Let  $\tilde{\mathbf{A}}$  be a non-singular matrix. Suppose  $\hat{\mathbf{A}}$  is a matrix such that  $\hat{\mathbf{A}}\hat{\mathbf{A}}^{\top} = \bar{\mathbf{A}}\bar{\mathbf{A}}^{\top}$ . Then,  $\hat{\mathbf{A}}$  is non-singular.
Proof. A matrix is non-singular (or invertible) if and only if its determinant is nonzero. Therefore, we need to show that  $\operatorname{det}(\hat{\mathbf{A}}) \neq 0$ .
From the given, we have  $\hat{\mathbf{A}}\hat{\mathbf{A}}^{\top} = \tilde{\mathbf{A}}\tilde{\mathbf{A}}^{\top}$ . Taking determinants on both sides, we get
$$
\det (\hat {\mathbf {A}} \hat {\mathbf {A}} ^ {\top}) = \det (\tilde {\mathbf {A}} \tilde {\mathbf {A}} ^ {\top}).
$$
Since  $\hat{\mathbf{A}}$  and  $\bar{\mathbf{A}}$  are square matrices, we can simplify both sides:
$$
\det (\hat {\mathbf {A}}) ^ {2} = \det (\tilde {\mathbf {A}}) ^ {2}.
$$
Since  $\tilde{\mathbf{A}}$  is non-singular, we know that  $\operatorname*{det}(\tilde{\mathbf{A}}) \neq 0$ . Therefore,  $\operatorname*{det}(\hat{\mathbf{A}})^2 \neq 0$ . This implies that  $\operatorname*{det}(\hat{\mathbf{A}}) \neq 0$ . Thus,  $\hat{\mathbf{A}}$  is non-singular.
B.3 Proof of Lemma 5
Lemma 4. Let  $\mathbf{A}$  be a matrix with all diagonal entries being nonzero. Then, there exists matrix  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$  and  $\Omega \in \mathrm{diag}(\mathbb{R}_{>0}^n)$  such that  $\mathbf{A} = (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}\mathbf{D}$ , where  $\mathbf{D}$  is a diagonal matrix with diagonal entries being  $\pm 1$ .
Proof. Since the diagonal entries of  $\mathbf{A}$  are nonzero, we can construct a diagonal matrix  $\mathbf{D}$  such that the diagonal entries of  $\mathbf{AD}$  are positive, by defining  $d_{i,i} = 1$  if  $a_{i,i} > 0$  and  $d_{i,i} = -1$  if  $a_{i,i} < 0$ . Let  $\Omega$  be a diagonal matrix of the same size as matrix  $\mathbf{A}$ , where  $(\Omega)_{i,i} = 1 / (\mathbf{AD})_{i,i}^2$ , Also, let  $\mathbf{B} := \mathbf{I} - \mathbf{AD}\Omega^{\frac{1}{2}}$ . Since the diagonal entries of matrix  $\mathbf{AD}\Omega^{\frac{1}{2}}$  are ones, the diagonal entries of matrix  $\mathbf{B}$  are zeros. Therefore, we have  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n\times n}$ ,  $\Omega \in \mathrm{diag}(\mathbb{R}_{>0}^n)$ , and  $\mathbf{A} = (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}\mathbf{D}$ , where  $\mathbf{D}$  is a diagonal matrix with entries being  $\pm 1$ .
Lemma 5. Let  $\mathbf{A}$  be a non-singular matrix. Then, there exists matrix  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$  and  $\Omega \in \mathrm{diag}(\mathbb{R}_{>0}^n)$  such that  $\mathbf{A} \sim (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$ .
Proof. Since matrix  $\mathbf{A}$  is non-singular, by Lemma 2, there exists a permutation matrix  $\mathbf{P}$  such that the diagonal entries of  $\mathbf{AP}$  are nonzero. By Lemma 4, there exists matrix  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$  and  $\Omega \in \mathrm{diag}(\mathbb{R}_{>0}^n)$  such that  $\mathbf{AP} = (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}\mathbf{D}$ , where  $\mathbf{D}$  is a diagonal matrix with diagonal entries being  $\pm 1$ . Clearly, we have  $\mathbf{A} \sim (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$ .
B.4 Proof of Lemma 6
Lemma 6. Suppose  $\mathbf{A} \sim (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$  for matrices  $\mathbf{A} \in \mathbb{R}^{n \times n}$ ,  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$ , and  $\Omega \in \mathrm{diag}(\mathbb{R}_{>0}^n)$ . Then, we have  $\| \mathbf{A} \|_0 = \| \mathbf{B} \|_0 + n$ .
Proof. First notice that the diagonal entries of  $\mathbf{B}$  are zeros, which implies
$$
\| \mathbf {I} - \mathbf {B} \| _ {0} = \| \mathbf {I} \| _ {0} + \| \mathbf {B} \| _ {0} = \| \mathbf {B} \| _ {0} + n.
$$
Since  $\Omega^{-\frac{1}{2}}$  is a diagonal matrix, right multiplication of  $\Omega^{-\frac{1}{2}}$  amounts to rescaling the columns of  $\mathbf{I} - \mathbf{B}$ , and does not affect its support. Therefore, we have
$$
\left\| (\mathbf {I} - \mathbf {B}) \boldsymbol {\Omega} ^ {- \frac {1}{2}} \right\| _ {0} = \left\| \mathbf {I} - \mathbf {B} \right\| _ {0} = \left\| \mathbf {B} \right\| _ {0} + n.
$$
Since  $\mathbf{A} \sim (\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$ , matrices  $\mathbf{A}$  and  $(\mathbf{I} - \mathbf{B})\Omega^{-\frac{1}{2}}$  differ only in signed column permutations. Therefore, their number of nonzero entries are the same, i.e.,
$$
\| \mathbf {A} \| _ {0} = \| \mathbf {B} \| _ {0} + n.
$$
B.5 Proof of Lemma 8
We first provide the following lemma that will be used to prove Lemma 8.
Lemma 7. Let  $\mathbf{P}_1$  and  $\mathbf{P}_2$  be two permutation matrices. Then,  $\mathbf{P}_1^\top \mathbf{P}_2$  is lower triangular if and only if  $\mathbf{P}_1 = \mathbf{P}_2$ .
Proof. The "if part" is clear because permutation matrices are orthogonal matrices, and thus  $\mathbf{P}_1^\top \mathbf{P}_1 = \mathbf{I}$  is lower triangular. It remains to prove the "only if part". Since  $\mathbf{P}_1^\top \mathbf{P}_2$  is also a permutation matrix, this matrix being lower triangular implies that it is an identity matrix, because identity matrix is the only permutation matrix that is lower triangular. We then have  $\mathbf{P}_1^\top \mathbf{P}_2 = \mathbf{I}$ , which implies
$$
\mathbf {P} _ {1} = \mathbf {P} _ {2} ^ {- \top} = \mathbf {P} _ {2}.
$$
We now provide the proof of Lemma 8.
Lemma 8. Given matrix  $\mathbf{B} \in \mathbb{R}_{\mathrm{off}}^{n \times n}$ , if  $\mathbf{I} - \mathbf{B}$  satisfies Assumption 2, then  $\mathbf{I} - \mathbf{B}$  is non-singular. Proof. Since  $\mathbf{I} - \mathbf{B}$  satisfies Assumption 2, there exist permutation matrices  $\mathbf{P}_1$  and  $\mathbf{P}_2$  such that
$$
\mathbf {P} _ {1} ^ {\top} (\mathbf {I} - \mathbf {B}) \mathbf {P} _ {2} = \mathbf {P} _ {1} ^ {\top} \mathbf {P} _ {2} - \mathbf {P} _ {1} ^ {\top} \mathbf {B} \mathbf {P} _ {2}
$$
is lower triangular. For all  $i,j\in [n]$  such that  $(\mathbf{P}_1^\top \mathbf{P}_2)_{i,j}\neq 0$  , we have  $(\mathbf{P}_1^\top \mathbf{B}\mathbf{P}_2)_{i,j} = 0$  , which implies that  $\mathbf{P}_1^\top \mathbf{P}_2$  must be lower triangular, because otherwise,  $\mathbf{P}_1^\top \mathbf{P}_2 - \mathbf{P}_1^\top \mathbf{B}\mathbf{P}_2$  cannot be lower
triangular. By Lemma 7, we then have  $\mathbf{P}_1 = \mathbf{P}_2$ . This indicates that  $\mathbf{I} - \mathbf{P}_1^\top \mathbf{B}\mathbf{P}_1$  is lower triangular, with diagonal entries equal to one. Therefore, we have
$$
\det  (\mathbf {I} - \mathbf {P} _ {1} ^ {\top} \mathbf {B} \mathbf {P} _ {1}) = 1,
$$
$$
\det (\mathbf {I} - \mathbf {B}) = \det (\mathbf {P} _ {1} (\mathbf {I} - \mathbf {B}) \mathbf {P} _ {1} ^ {\top}) = \det (\mathbf {I} - \mathbf {P} _ {1} ^ {\top} \mathbf {B} \mathbf {P} _ {1}) = 1.
$$
Since the determinant of  $\mathbf{I} - \mathbf{B}$  is nonzero, it is non-singular.
We first describe the notion of covariance equivalence that is needed for the proof of Theorem 1. Specifically, if two support matrices  $\pmb{\xi}_1$  and  $\pmb{\xi}_2$  entail the same set of covariance matrices, i.e.,  $\pmb{\Sigma}(\pmb{\xi}_1) = \pmb{\Sigma}(\pmb{\xi}_2)$ , they are said to be covariance equivalent. This means that for any combination of parameter values in  $\pmb{\xi}_1$ , there exists a corresponding set of parameter values in  $\pmb{\xi}_2$  that leads to the same covariance matrix, and vice versa. Furthermore, two support matrices are covariance equivalent if and only if they lead to the same set of semialgebraic constraints on the covariance matrices.
We now give the proof of Theorem 1. The proof makes use of Proposition 2, Proposition 6, Proposition 7, and Corollary 1, which are provided in Appendices C.1, C.2, C.3, and C.4, respectively. Note that the proof is partly inspired by that of Ghassami et al. [20, Theorem 3].
Proof. Let  $\hat{\mathbf{A}}$  be a solution of Problem (2). This implies  $\hat{\mathbf{A}}\hat{\mathbf{A}}^{\top} = \tilde{\boldsymbol{\Sigma}} = \tilde{\mathbf{A}}\tilde{\mathbf{A}}^{\top}$  and that  $\hat{\mathbf{A}}$  satisfies Assumption 2. Since  $\tilde{\mathbf{A}}$  is non-singular, by Lemma 3, matrix  $\hat{\mathbf{A}}$  is non-singular.
Since  $\hat{\mathbf{A}}$  can entail the covariance matrix  $\hat{\Sigma}$ , we have  $\tilde{\Sigma} \in \Sigma(\xi_{\hat{\mathbf{A}}})$ , which indicates that  $\hat{\Sigma}$  contains all semialgebraic constraints of  $\xi_{\hat{\mathbf{A}}}$ . Under Assumption 3, we have
$$
H \left(\boldsymbol {\xi} _ {\tilde {\mathbf {A}}}\right) \subseteq H \left(\boldsymbol {\xi} _ {\tilde {\mathbf {A}}}\right). \tag {8}
$$
The sparsity term in the objective function implies
$$
\| \hat {\mathbf {A}} \| _ {0} \leq \| \tilde {\mathbf {A}} \| _ {0}, \tag {9}
$$
because otherwise  $\hat{\mathbf{A}}$  will never be a solution of Problem (2).
We now show by contradiction that  $H(\pmb{\xi}_{\hat{\mathbf{A}}}) \not\subset H(\pmb{\xi}_{\tilde{\mathbf{A}}})$ . Suppose  $H(\pmb{\xi}_{\hat{\mathbf{A}}}) \subset H(\pmb{\xi}_{\tilde{\mathbf{A}}})$ , which indicates  $\dim(\Sigma(\pmb{\xi}_{\hat{\mathbf{A}}})) > \dim(\pmb{\Sigma}(\pmb{\xi}_{\tilde{\mathbf{A}}}))$ . Since the support matrices  $\pmb{\xi}_{\hat{\mathbf{A}}}$  and  $\pmb{\xi}_{\tilde{\mathbf{A}}}$  satisfy Assumption 2, Proposition 2 implies  $\| \pmb{\xi}_{\hat{\mathbf{A}}} \|_0 > \| \pmb{\xi}_{\tilde{\mathbf{A}}} \|_0$ , which is contradictory with Inequality (9). This implies
$$
H \left(\boldsymbol {\xi} _ {\tilde {\mathbf {A}}}\right) \notin H \left(\boldsymbol {\xi} _ {\tilde {\mathbf {A}}}\right). \tag {10}
$$
By Eqs. (8) and (10), we obtain  $H(\pmb{\xi}_{\hat{\mathbf{A}}}) = H(\pmb{\xi}_{\hat{\mathbf{A}}})$ , which, by Proposition 7, implies that the support matrices  $\pmb{\xi}_{\hat{\mathbf{A}}}$  and  $\pmb{\xi}_{\hat{\mathbf{A}}}$  are covariance equivalent, because they satisfy Assumption 2. By Proposition 6, we conclude that the columns of  $\pmb{\xi}_{\hat{\mathbf{A}}}$  are a permutation of those of  $\pmb{\xi}_{\hat{\mathbf{A}}}$ .
Recall that matrices  $\tilde{\mathbf{A}}$  and  $\hat{\mathbf{A}}$  are non-singular, and entail the same covariance matrix  $\tilde{\Sigma}$ . Since the columns of  $\xi_{\hat{\mathbf{A}}}$  are a permutation of those of  $\xi_{\tilde{\mathbf{A}}}$ , by Corollary 1, we have  $\hat{\mathbf{A}} \sim \tilde{\mathbf{A}}$ .
C.1 Dimension of Covariance Set
We first define the Jacobian matrix of  $\boldsymbol{\Sigma} = \mathbf{A}\mathbf{A}^{\top}$  w.r.t. the free parameters of  $\mathbf{A}$  as  $\frac{\partial\boldsymbol{\Sigma}}{\partial\mathbf{A}}$ , which is a  $n^2\times \| \mathbf{A}\| _0$  matrix, where the rows are indexed by  $(i,j)\in [n]\times [n]$ , and columns are indexed by
$(k,l)\in \operatorname {supp}(\mathbf{A})$  . That is, for  $(i,j)\in [n]\times [n]$  and  $(k,l)\in \operatorname {supp}(\mathbf{A})$  , we have
$$
\left(\frac {\partial \boldsymbol {\Sigma}}{\partial \mathbf {A}}\right) _ {(i, j), (k, l)} = \frac {\partial \Sigma_ {i , j}}{\partial a _ {k , l}} = \left\{ \begin{array}{l l} 2 a _ {i, l} & \text {i f} i = j \text {a n d} k = i, \\ 0 & \text {i f} i = j \text {a n d} k \neq i, \\ a _ {j, l} & \text {i f} i \neq j \text {a n d} k = i, \\ a _ {i, l} & \text {i f} i \neq j \text {a n d} k = j, \\ 0 & \text {i f} i \neq j \text {a n d} k \notin \{i, j \}. \end{array} \right. \tag {11}
$$
Given a matrix  $\mathbf{M}$ , we denote by  $\mathrm{off}(\mathbf{M})$  and  $\mathrm{on}(\mathbf{M})$  the off-diagonal and diagonal entries of  $\mathbf{M}$ , respectively. With a slight abuse of notation, we rewrite the above Jacobian matrix as the following form that consists of four submatrices, by permuting the corresponding columns and rows:
$$
\frac {\partial \boldsymbol {\Sigma}}{\partial \mathbf {A}} = \left[ \begin{array}{l l} \frac {\partial \operatorname {o f f} (\boldsymbol {\Sigma})}{\partial \operatorname {o f f} (\mathbf {A})} & \frac {\partial \operatorname {o f f} (\boldsymbol {\Sigma})}{\partial \operatorname {o n} (\mathbf {A})} \\ \frac {\partial \operatorname {o n} (\boldsymbol {\Sigma})}{\partial \operatorname {o f f} (\mathbf {A})} & \frac {\partial \operatorname {o n} (\boldsymbol {\Sigma})}{\partial \operatorname {o n} (\mathbf {A})} \end{array} \right]. \tag {12}
$$
Each submatrix above represents the Jacobian matrix for different parts (i.e., off-diagonal and diagonal entries) of matrix  $\pmb{\Sigma}$  taken w.r.t. different free parameters (i.e., off-diagonal and diagonal free parameters) in matrix  $\mathbf{A}$ . Note that column and row permutations do not affect the rank of the matrix. Since we are primarily interested in the rank of the above Jacobian matrix and its submatrices, we use the same notation to refer to different column and/or row permutations of the corresponding Jacobian matrix, depending on the context.
Lemma 9. Let  $\mathbf{A}$  be a lower triangular matrix. Then, we have
$$
\operatorname {r a n k} \left(\left. \frac {\partial \boldsymbol {\Sigma}}{\partial \mathbf {A}} \right| _ {\mathbf {A} = \mathbf {I} _ {n}}\right) = \| \mathbf {A} \| _ {0}.
$$
Proof. Let  $d$  be the number of diagonal free parameters in matrix  $\mathbf{A}$ . This indicates that  $\mathrm{off}(\mathbf{A})$  and  $\mathrm{on}(\mathbf{A})$  contain  $\| \mathbf{A}\| _0 - d$  and  $d$  free parameters, respectively.
By specializing  $\mathbf{A} = \mathbf{I}_n$  and using Eq. (11), we have
$$
\left. \frac {\partial \text {o f f} (\boldsymbol {\Sigma})}{\partial \operatorname {o n} (\mathbf {A})} \right| _ {\mathbf {A} = \mathbf {I} _ {n}} = \mathbf {0}, \quad \left. \frac {\partial \operatorname {o n} (\boldsymbol {\Sigma})}{\partial \operatorname {o f f} (\mathbf {A})} \right| _ {\mathbf {A} = \mathbf {I} _ {n}} = \mathbf {0}, \quad \text {a n d} \quad \left. \frac {\partial \operatorname {o n} (\boldsymbol {\Sigma})}{\partial \operatorname {o n} (\mathbf {A})} \right| _ {\mathbf {A} = \mathbf {I} _ {n}} = 2 \mathbf {I} _ {d}, \tag {13}
$$
where the last equality is obtained after row permutations of the corresponding matrix. Also, since matrix  $\mathbf{A}$  is lower triangular, each nonzero entry of  $\left.\frac{\partial\mathrm{off}(\boldsymbol{\Sigma})}{\partial\mathrm{off}(\mathbf{A})}\right|_{\mathbf{A} = \mathbf{I}_n}$  corresponds to either
$$
\frac {\partial \Sigma_ {k , l}}{\partial a _ {k , l}} = 1 \quad \text {o r} \quad \frac {\partial \Sigma_ {l , k}}{\partial a _ {k , l}} = 1 \quad \text {f o r s o m e} k > l.
$$
In this case, each column of  $\left.\frac{\partial\mathrm{off}(\pmb{\Sigma})}{\partial\mathrm{off}(\mathbf{A})}\right|_{\mathbf{A} = \mathbf{I}_n}$  contains precisely two nonzero entries, while each row either contains precisely one nonzero entry, or does not contain any nonzero entry. Therefore,  $\left.\frac{\partial\mathrm{off}(\pmb{\Sigma})}{\partial\mathrm{off}(\mathbf{A})}\right|_{\mathbf{A} = \mathbf{I}_n}$  can be rewritten after row permutations as