SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Image datasets. To study how reasonable the proposed theories are w.r.t. the practical generating process of observational data in complex scenarios, we conduct experiments on "Triangles" (Yang et al., 2022) and EMNIST (Cohen et al., 2017) datasets. The "Triangles" dataset consists of 60,000 synthetic  $28 \times 28$  images of triangles, which are generated from 4 factors: rotation, height, width, and brightness. By fixing the number of pixels (observed variables) and generating factors (sources), we can guarantee that the images are generated according to an undercomplete process, although the exact generating process is still unknown (e.g., a pixel could be (indirectly) influenced by multiple factors in a complicated way). For the real-world dataset, EMNIST contains 240,000  $28 \times 28$  images of handwritten digits and is a larger version of the classical MNIST dataset. Although we do not know the exact number of sources, it is highly possible that it is smaller than the number of pixels (784). We present the identified sources with the top four standard deviations (SDs) from both datasets in Fig. 6 and Fig. 7. In both figures, each row represents a source identified by our model, with it varying from  $-4$  to  $+4$  SDs to illustrate its influence. The rightmost column is a heat map given by the absolute pixel difference between  $-1$  and  $+1$  SDs. By observing the identified sources with the top four standard deviations, one could find that it is possible to identify semantically meaningful attributes from practical image datasets, which further suggests the potential of our theory in real-world scenarios. Additional results are available in Appx. B.2.
We establish a set of new identifiability results of nonlinear ICA in general settings with undercompleteness, partial sparsity and source dependence, and flexible grouping structures, thereby extending the identifiability theory to a wide range of real-world scenarios. Specifically, we prove the identifiability when there are more observed variables than underlying sources, and when sparsity and/or independence are not met for a subset of sources. Moreover, by leveraging various dependence structures among sources, further identifiability guarantees can also be obtained. Theoretical results have been validated through a combination of extensive previous studies and our own experiments, which involve both synthetic and real-world datasets. Future work includes adopting the theoretical framework for related tasks, such as disentanglement, transfer learning, and causal discovery. Furthermore, the proposed identifiability guarantees on generalized latent variable models bolster our confidence in uncovering hidden truths across diverse real-world settings in scientific discovery. We have only explored the visual disentanglement task, and the lack of other applications is a limitation of this work.
We are grateful to everyone involved in the anonymous reviewing process for their insightful feedback. This project is partially supported by NSF Grant 2229881, the National Institutes of Health (NIH) under Contract R01HL159805, a grant from Apple Inc., a grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
L. Ardizzone, T. Bungert, F. Draxler, U. Kothe, J. Kruse, R. Schmier, and P. Sorrenson. Framework for Easily Invertible Architectures (FrEIA), 2018-2022. URL https://github.com/vislearn/FrEIA.
P. Breheny and J. Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. The Annals of Applied Statistics, 5(1):232-253, 2011.
S. Buchholz, M. Besserve, and B. Schölkopf. Function classes for identifiable nonlinear independent component analysis. arXiv preprint arXiv:2208.06406, 2022.
C. P. Burgess, I. Higgins, A. Pal, L. Matthew, N. Watters, G. Desjardins, and A. Lerchner. Understanding disentangling in beta-VAE. Workshop on Learning Disentangled Representations at the 31st Conference on Neural Information Processing Systems, 2018.
M.-A. Carbonneau, J. Zaidi, J. Boilard, and G. Gagnon. Measuring disentanglement: A review of metrics. IEEE Transactions on Neural Networks and Learning Systems, 2022.
J.-F. Cardoso. Multidimensional independent component analysis. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP'98 (Cat. No. 98CH36181), volume 4, pages 1941-1944. IEEE, 1998.
R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emmist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
P. Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
S. Duan, L. Matthey, A. Saraiva, N. Watters, C. Burgess, A. Lerchner, and I. Higgins. Unsupervised model selection for variational disentangled representation learning. In *ICLR*, 2020.
F. Falck, H. Zhang, M. Willetts, G. Nicholson, C. Yau, and C. C. Holmes. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Systems, 34:8676-8690, 2021.
J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American statistical Association, 96(456):1348-1360, 2001.
H. Halva, S. Le Corff, L. Lehéricy, J. So, Y. Zhu, E. Gassiat, and A. Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems, 34, 2021.
A. Hyvarinen and P. Hoyer. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation, 12(7):1705-1720, 2000.
A. Hyvarinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems, 29:3765-3773, 2016.
A. Hyvarinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In International Conference on Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
A. Hyvarinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999.
F. Theis. Towards a general independent subspace analysis. Advances in Neural Information Processing Systems, 19, 2006.
J. Von Kugelgen, Y. Sharma, L. Gresele, W. Brendel, B. Scholkopf, M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451-16467, 2021.
M. J. Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using  $\ell_1$ -constrained quadratic programming (Lasso). IEEE Transactions on Information Theory, 55(5): 2183-2202, 2009.
X. Yang, Y. Wang, J. Sun, X. Zhang, S. Zhang, Z. Li, and J. Yan. Nonlinear ICA using volume-preserving transformations. In International Conference on Learning Representations, 2022.
W. Yao, Y. Sun, A. Ho, C. Sun, and K. Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2110.05428, 2021.
W. Yao, G. Chen, and K. Zhang. Temporally disentangled representation learning. In Advances in Neural Information Processing Systems, 2022.
C.-H. Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2):894-942, 2010.
Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022.
A.1 Proof of Theorem 3.1 14
A.2 Proof of Theorem 4.1 16
A.3 Proof of Theorem 4.2 17
A.4 Proof of Theorem 4.3 20
A.5 Proof of Theorem 4.4 23
B Experiments 28
B.1 Supplementary experimental settings 28
B.2 Supplementary experimental results 29
i. For each  $i \in \{1, \ldots, n\}$ , there exist  $\{\mathbf{s}^{(\ell)}\}_{\ell=1}^{\left|\mathcal{F}_{i,:}\right|}$  and a matrix  $T \in \mathcal{T}$  s.t.  $\operatorname{span}\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i,:}\}_{\ell=1}^{\left|\mathcal{F}_{i,:}\right|} = \mathbb{R}_{\mathcal{F}_{i,:}}^{n}$  and  $\left[\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})\mathbf{T}\right]_{i,:} \in \mathbb{R}_{\hat{\mathcal{F}}_{i,:}}^{n}$ .
ii. (Structural Sparsity) For each  $k \in \{1, \dots, n\}$ , there exists  $C_k$  s.t.  $\bigcap_{i \in C_k} \mathcal{F}_{i,:} = \{k\}$ .
Proof. Let  $\mathbf{h}:\mathbf{s}\rightarrow \hat{\mathbf{s}}$  denotes the transformation between the true sources and estimated sources. We can apply the chain rule repeatedly to get:
$$
\begin{array}{l} \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) = \mathbf {J} _ {\hat {\mathbf {f}} \circ \mathbf {h}} (\mathbf {s}) \tag {6} \\ = \mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) \mathbf {J} _ {\mathbf {h}} (\mathbf {s}). \\ \end{array}
$$
Since  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}})$  and  $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$  both possess full column rank,  $\mathbf{J}_{\mathbf{h}}(\mathbf{s})$  should have a non-zero determinant. From this, we can deduce by incorporating the inverse of  $\mathbf{h}$ :
$$
\mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) \mathbf {J} _ {\mathbf {h}} (\mathbf {s}) ^ {- 1}. \tag {7}
$$
Our objective here is to demonstrate that the function  $\mathbf{h}$  is a composition of a permutation and a component-wise invertible transformation. Let  $\mathbf{D}(\mathbf{s})$  denote a diagonal matrix and  $\mathbf{P}$  denote a permutation matrix, our goal can be rewritten as demonstrating that  $\mathbf{J}_{\mathbf{h}}(\mathbf{s})^{-1} = \mathbf{D}(\mathbf{s})\mathbf{P}$ . This leads us to demonstrate that:
$$
\mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) \mathbf {D} (\mathbf {s}) \mathbf {P}. \tag {8}
$$
Further, we can express:
$$
\mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) \mathbf {T} (\mathbf {s}), \tag {9}
$$
where  $\mathbf{T}(\mathbf{s})\in \mathbb{R}^{n\times n}$  is a square matrix. Here, we define  $\mathcal{F}$  as the support of  $\mathbf{J}_{\mathbf{f}}(\mathbf{s}),\hat{\mathcal{F}}$  as the support of  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}})$  and  $\mathcal{T}$  as a set of matrices with the same support of  $\mathbf{T}(\mathbf{s})$ . Furthermore,  $\mathrm{T}\in \mathcal{T}$  is a matrix with the same support as  $\mathbf{T}(\mathbf{s})$ . Based on Assumption i, we have:
$$
\operatorname {s p a n} \left\{\mathbf {J} _ {\mathbf {f}} \left(\mathbf {s} ^ {(\ell)}\right) _ {i,:} \right\} _ {\ell = 1} ^ {\left| \mathcal {F} _ {i,:} \right|} = \mathbb {R} _ {\mathcal {F} _ {i,:}} ^ {n}. \tag {10}
$$
Given that the set  $\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i,:}\}_{\ell = 1}^{\left|\mathcal{F}_{i,:}\right|}$  forms a basis of  $\mathbb{R}_{\mathcal{F}_{i,:}}^n$ , we can express any vector in this space as a linear combination of these basis vectors. In particular, for any  $j_0 \in \mathcal{F}_{i,:}$ , the one-hot vector  $e_{j_0} \in \mathbb{R}_{\mathcal{F}_{i,:}}^n$  can be written as
$$
e _ {j _ {0}} = \sum_ {\ell \in \mathcal {F} _ {i,:}} \alpha_ {\ell} \mathbf {J} _ {\mathbf {f}} \left(\mathbf {s} ^ {(\ell)}\right) _ {i,:}, \tag {11}
$$
where  $\alpha_{\ell}$  denotes the respective coefficient.
With this in mind, we can find the transformation of  $e_{j_0}$  under  $\mathrm{T}$  as
$$
\mathrm {T} _ {j _ {0},:} = e _ {j _ {0}} \mathrm {T} = \sum_ {\ell \in \mathcal {F} _ {i,:}} \alpha_ {\ell} \mathbf {J} _ {\mathbf {f}} \left(\mathbf {s} ^ {(\ell)}\right) _ {i,:} \mathrm {T}. \tag {12}
$$
According to Assumption i, each term in the above summation belongs to the space  $\mathbb{R}_{\hat{\mathcal{F}}_{i,:}}^n$ . Therefore,  $\mathrm{T}_{j_0,:}$  itself resides in  $\mathbb{R}_{\hat{\mathcal{F}}_{i,:}}^n$ , i.e.,  $\mathrm{T}_{j_0,:} \in \mathbb{R}_{\hat{\mathcal{F}}_{i,:}}^n$ . Thus
$$
\forall j \in \mathcal {F} _ {i,:}, \mathrm {T} _ {j,:} \in \mathbb {R} _ {\hat {\mathcal {F}} _ {i,:}} ^ {n}. \tag {13}
$$
Then the connections between these supports can be established according to Defn. 2.3
$$
\forall (i, j) \in \mathcal {F}, \{i \} \times \mathcal {T} _ {j,:} \subset \hat {\mathcal {F}}. \tag {14}
$$
It is noteworthy that a similar strategy to derive Eq. 14 has been applied in (Zheng et al., 2022) and part of the proof technique is inspired by that work. In contrast to the proof by Zheng et al. (2022), which assumes the invertibility of  $f$ , we only necessitate its injectivity. This distinction allows for the inclusion of undercomplete cases.
Since  $\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})$  and  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}}^{(\ell)})$  have full column rank  $n$ ,  $\mathbf{T}(\mathbf{s}^{(\ell)})$  must have a non-zero determinant. Otherwise, it would follow that the rank of  $\mathbf{T}(\mathbf{s}^{(\ell)})$  is less than  $n$ , which would imply a contradiction that  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}}^{(\ell)}) = \mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})\mathbf{T}(\mathbf{s}^{(\ell)})$  has a column rank less than  $n$ . Representing the determinant of the matrix  $\mathbf{T}(\mathbf{s}^{(\ell)})$  as its Leibniz formula yields
$$
\det  (\mathbf {T} (\mathbf {s} ^ {(\ell)})) = \sum_ {\sigma \in S _ {n}} \left(\operatorname {s g n} (\sigma) \prod_ {i = 1} ^ {n} \mathbf {T} (\mathbf {s} ^ {(\ell)}) _ {i, \sigma (i)}\right) \neq 0, \tag {15}
$$
where  $S_{n}$  is the set of  $n$ -permutations. Thus, there is at least one term in the sum that is non-zero, i.e.,
$$
\exists \sigma \in \mathcal {S} _ {n}, \forall i \in \{1, \dots , n \}, \operatorname {s g n} (\sigma) \prod_ {i = 1} ^ {n} \mathbf {T} \left(\mathbf {s} ^ {(\ell)}\right) _ {i, \sigma (i)} \neq 0, \tag {16}
$$
$$
\exists \sigma \in \mathcal {S} _ {n}, \forall i \in \{1, \dots , n \}, \mathrm {T} \left(\mathrm {s} ^ {(\ell)}\right) _ {i, \sigma (i)} \neq 0. \tag {17}
$$
Then we can conclude that this  $\sigma$  is in the support of  $\mathbf{T}(\mathbf{s})$  since  $\mathbf{s}^{(\ell)}\in \mathbf{s}$ . Therefore, it follows that
$$
\forall j \in \{1, \dots , n \}, \sigma (j) \in \mathcal {T} _ {j,:}. \tag {18}
$$
Together with Eq. (14), we have
$$
\forall (i, j) \in \mathcal {F}, (i, \sigma (j)) \in \{i \} \times \mathcal {T} _ {j,:} \subset \hat {\mathcal {F}}. \tag {19}
$$
$$
\sigma (\mathcal {F}) = \{(i, \sigma (j)) \mid (i, j) \in \mathcal {F} \}. \tag {20}
$$
$$
\sigma (\mathcal {F}) \subset \hat {\mathcal {F}}. \tag {21}
$$
Because of the sparsity regularization on the estimated Jacobian, we further have
$$
| \hat {\mathcal {F}} | \leq | \mathcal {F} | = | \sigma (\mathcal {F}) |. \tag {22}
$$
Combining this with Eq. (21), we derive
$$
\sigma (\mathcal {F}) = \hat {\mathcal {F}}. \tag {23}
$$
Suppose  $\mathbf{T}(\mathbf{s})\neq \mathbf{D}(\mathbf{s})\mathbf{P}$  , then
$$
\exists j _ {1} \neq j _ {2}, \mathcal {T} _ {j _ {1},:} \cap \mathcal {T} _ {j _ {2},:} \neq \emptyset . \tag {24}
$$
Additionally, consider  $j_3 \in \{1, \dots, n\}$  for which
$$
\sigma \left(j _ {3}\right) \in \mathcal {T} _ {j _ {1},:} \cap \mathcal {T} _ {j _ {2},:}. \tag {25}
$$
Since  $j_1 \neq j_2$ , we can assume  $j_3 \neq j_1$  without loss of generality. A similar strategy has been used previously in (Lachapelle et al., 2022; Zheng et al., 2022). Based on Assumption ii, there exists  $\mathcal{C}_{j_1} \ni j_1$  such that  $\bigcap_{i \in \mathcal{C}_{j_1}} \mathcal{F}_{i,:} = \{j_1\}$ . Because
$$
j _ {3} \notin \{j _ {1} \} = \bigcap_ {i \in \mathcal {C} _ {j _ {1}}} \mathcal {F} _ {i,:}, \tag {26}
$$
there must exist  $i_3 \in \mathcal{C}_{j_1}$  such that
$$
j _ {3} \notin \mathcal {F} _ {i _ {3},:}. \tag {27}
$$
Since  $j_{1}\in \mathcal{F}_{i_{3},:}$  , it follows that  $(i_3,j_1)\in \mathcal{F}$  . Therefore, according to Eq. (14), we have
$$
\left\{i _ {3} \right\} \times \mathcal {T} _ {j _ {1},;} \subset \hat {\mathcal {F}}. \tag {28}
$$
Notice that  $\sigma(j_3) \in \mathcal{T}_{j_1,:} \cap \mathcal{T}_{j_2,:}$  implies
$$
\left(i _ {3}, \sigma \left(j _ {3}\right)\right) \in \left\{i _ {3} \right\} \times \mathcal {T} _ {j _ {1}, \dots}. \tag {29}
$$
Then by Eqs. (28) and (29), we have
$$
\left(i _ {3}, \sigma \left(j _ {3}\right)\right) \in \hat {\mathcal {F}}. \tag {30}
$$
This further implies  $(i_3,j_3)\in \mathcal{F}$  by Eq. (20) and (23), which contradicts Eq. (27). Therefore, we have proven by contradiction that  $\mathbf{T}(\mathbf{s}) = \mathbf{D}(\mathbf{s})\mathbf{P}$ . By replacing  $\mathbf{T}(\mathbf{s})$  with  $\mathbf{D}(\mathbf{s})\mathbf{P}$  in Eq. (9), we obtain Eq. (8), which is the goal.
A.2 Proof of Theorem 4.1
Proof. Let  $h: \mathbf{s} \to \hat{\mathbf{s}}$  denotes the transformation between the true and estimated sources. By using chain rule repeatedly, we have
$$
\begin{array}{l} \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) = \mathbf {J} _ {\hat {\mathbf {f}} \circ \mathbf {h}} (\mathbf {s}) \tag {31} \\ = \mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) \mathbf {J} _ {\mathbf {h}} (\mathbf {s}). \\ \end{array}
$$
Since  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}})$  and  $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$  both possess full column rank,  $\mathbf{J}_{\mathbf{h}}(\hat{\mathbf{s}})$  should have a non-zero determinant. Thus,  $\mathbf{J}_{\mathbf{h}}(\mathbf{s})$  must be invertible and have a non-zero determinant. Otherwise, one of them would not be of full column rank, which leads to a contradiction.
Applying the change of variable rule, we have
$$
p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u}) | \det  (\mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}})) | = p _ {\mathbf {s}} (\hat {\mathbf {s}} | \mathbf {u}). \tag {32}
$$
Taking the logarithm on both sides yields
$$
\log p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u}) + \log | \det  (\mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}})) | = \log p _ {\tilde {\mathbf {s}} | \mathbf {u}} (\hat {\mathbf {s}} | \mathbf {u}). \tag {33}
$$
Note that according to the model defined in Eqs. (2) and (3), the joint densities can be factorized as
$$
p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u}) = p _ {\mathbf {s} _ {D} | \mathbf {u}} \left(\mathbf {s} _ {D} | \mathbf {u}\right) \prod_ {\substack {i = 1 \\ s _ {i}}} ^ {n _ {I}} p _ {s _ {i}} \left(s _ {i}\right), \tag{34}
$$
$$
p _ {\hat {\mathbf {s}} | \mathbf {u}} (\hat {\mathbf {s}} | \mathbf {u}) = p _ {\hat {\mathbf {s}} _ {D} | \mathbf {u}} (\hat {\mathbf {s}} _ {D} | \mathbf {u}) \prod_ {i = 1} ^ {n _ {I}} p _ {\hat {s} _ {i}} (\hat {s} _ {i}).
$$
Then we define the difference across domains as follows
$$
q (\mathbf {s}, \mathbf {u} _ {j}) = \log p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u} _ {j}) - \log p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u} _ {0}). \tag {35}
$$
By taking the difference on both sides of Eq. (33) corresponding to  $\mathbf{u}_j$  and  $\mathbf{u}_0$ , we obtain
$$
q \left(\hat {\mathbf {s}} _ {D}, \mathbf {u} _ {j}\right) = q \left(\mathbf {s} _ {D}, \mathbf {u} _ {j}\right), \tag {36}
$$
where, for  $i \in \{1, \dots, n_I\}$ ,  $\log p_{s|\mathbf{u}}(\hat{s}_i|\mathbf{u}_j)$  and  $\log p_{s|\mathbf{u}}(s_i|\mathbf{u}_j)$  have been canceled because sources in  $\mathbf{s}_I$  are not dependent on  $\mathbf{u}_j$ . That is,  $q(s_i, \mathbf{u}_j) = 0$  if  $i \in \{1, \dots, n_I\}$ .
Taking the derivatives of both sides of Eq. (36) w.r.t.  $\hat{s}_k$  where  $k\in \{1,\dots ,n_I\}$ , we have
$$
\frac {\partial q \left(\hat {\mathbf {s}} _ {D} , \mathbf {u} _ {j}\right)}{\partial \hat {s} _ {k}} = \sum_ {i = n _ {I} + 1} ^ {n} \left(\frac {\partial q \left(\mathbf {s} _ {D} , \mathbf {u} _ {j}\right)}{\partial s _ {i}} \frac {\partial s _ {i}}{\partial \hat {s} _ {k}}\right), \tag {37}
$$
Clearly, LHS of Eq. (37) equals zero. By considering each  $j \in \{1, \dots, n_D\}$  for  $\mathbf{u}_j$ , we have  $n_D$  equations like Eq. (37), which constitute a linear system with a  $n_D \times n_D$  coefficient matrix.
According to the assumption, the coefficient matrix of the linear system has full rank. Thus, the only solution of Eq. (37) is  $\frac{\partial s_i}{\partial\hat{s}_k} = 0$  for  $i\in \{n_I + 1,\dots ,n\}$  and  $k\in \{1,\ldots ,n_I\}$ .
As  $\mathbf{h}^{-1}(\cdot)$  is smooth, its Jacobian can be written as:
$$
\mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) = \left[ \begin{array}{c c} \mathbf {A} := \frac {\partial \mathbf {s} _ {I}}{\partial \hat {\mathbf {s}} _ {I}} & \mathbf {B} := \frac {\partial \mathbf {s} _ {I}}{\partial \hat {\mathbf {s}} _ {D}} \\ \hline \mathbf {C} := \frac {\partial \mathbf {s} _ {D}}{\partial \hat {\mathbf {s}} _ {I}} & \mathbf {D} := \frac {\partial \mathbf {s} _ {D}}{\partial \hat {\mathbf {s}} _ {D}} \end{array} \right]. \tag {38}
$$
Since  $\frac{\partial\mathbf{s}_j}{\partial\hat{\mathbf{s}}_k} = 0$  for  $j\in \{n_I + 1,\dots ,n\}$  and  $k\in \{1,\ldots ,n_I\}$ , entries in the submatrix  $\mathbf{C}$  must all be zero. Thus, the submatrix  $\mathbf{D}$  must be invertible, otherwise  $\mathbf{J}_{\mathbf{h}^{-1}}(\hat{\mathbf{s}})$  will not be invertible, which is a contradiction. Besides, based on Assumption ii, one can show that all entries in the submatrix  $\mathbf{B}$  are zero according to part of the proof of Theorem 4.2 in (Kong et al., 2022) (Steps 1, 2, and 3). Therefore,  $\hat{\mathbf{s}}_D$  is an invertible transformation of  $\mathbf{s}_D$ .
i. For each  $i \in \{1, \dots, n_I\}$ , there exist  $\{\mathbf{s}^{(\ell)}\}_{\ell=1}^{\left|\mathcal{F}_{i:n_I}\right|}$  and a matrix  $T \in \mathcal{T}$  s.t.  $\operatorname{span}\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i:n_I}\}_{\ell=1}^{\left|\mathcal{F}_{i:n_I}\right|} = \mathbb{R}_{\mathcal{F}_{i:n_I}}^{n_I}$  and  $\left[\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})\mathbf{T}\right]_{i:n_I} \in \mathbb{R}_{\hat{\mathcal{F}}_{i:n_I}}^{n_I}$ .
ii. (Structural Sparsity) For all  $k \in \{1, \dots, n_I\}$ , there exists  $\mathcal{C}_k$  s.t.  $\bigcap_{i \in \mathcal{C}_k} \mathcal{F}_{i:n_I} = \{k\}$ .
Proof. Our goal here is to show that  $\hat{\mathbf{s}}_I$  is a composition of a permutation and a component-wise invertible transformation of sources in  $\mathbf{s}_I$ . By using chain rule repeatedly, we have
$$
\begin{array}{l} \mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) = \mathbf {J} _ {\mathbf {f} \circ \mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) \\ = \mathbf {J} _ {\mathbf {f}} \left(\mathbf {h} ^ {- 1} (\hat {\mathbf {s}})\right) \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) \tag {39} \\ = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}). \\ \end{array}
$$
$$
\mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) = \left[ \begin{array}{c c} \mathbf {A} := \frac {\partial \mathbf {s} _ {I}}{\partial \hat {\mathbf {s}} _ {I}} & \mathbf {B} := \frac {\partial \mathbf {s} _ {I}}{\partial \hat {\mathbf {s}} _ {D}} \\ \hline \mathbf {C} := \frac {\partial \mathbf {s} _ {D}}{\partial \hat {\mathbf {s}} _ {I}} & \mathbf {D} := \frac {\partial \mathbf {s} _ {D}}{\partial \hat {\mathbf {s}} _ {D}} \end{array} \right]. \tag {40}
$$
In the proof of Thm. 4.1, we have shown that all entries in the submatrix  $\mathbf{C}$  are zero. Then we have
$$
\mathbf {J} _ {\hat {\mathbf {f}}} (\hat {\mathbf {s}}) _ {:,: n _ {I}} = \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) _ {:,: n _ {I}} \tag {41}
$$
$$
\stackrel {(\star)} {=} \mathbf {J} _ {\mathbf {f}} (\mathbf {s}) _ {:,: n _ {I}} \mathbf {J} _ {\mathbf {h} ^ {- 1}} (\hat {\mathbf {s}}) _ {: n _ {I},: n _ {I}},
$$