SlowGuess/NeurIPS_2023_OCR huggingface dataset:
i. For each  $i \in \{1, \dots, n_I\}$ , there exist  $\{\mathbf{s}^{(\ell)}\}_{\ell=1}^{\left|\mathcal{F}_{i:n_I}\right|}$  and a matrix  $T \in \mathcal{T}$  s.t.  $\operatorname{span}\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i:n_I}\}_{\ell=1}^{\left|\mathcal{F}_{i:n_I}\right|} = \mathbb{R}_{\mathcal{F}_{i:n_I}}^{n_I}$  and  $\left[\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})\mathbf{T}\right]_{i:n_I} \in \mathbb{R}_{\hat{\mathcal{F}}_{i:n_I}}^{n_I}$ .
ii. (Structural Sparsity) For all  $k \in \{1, \dots, n_I\}$ , there exists  $\mathcal{C}_k$  s.t.  $\bigcap_{i \in \mathcal{C}_k} \mathcal{F}_{i,:n_I} = \{k\}$ .
Then  $\mathbf{s}_I$  is identifiable up to an element-wise invertible transformation and a permutation.
The proofs are presented in Appx. A.2 and Appx. A.3. We tackle the challenge of partial sparsity and dependence by necessitating Structural Sparsity and independence only on a subset of sources in  $\mathbf{s}_I$  and prove that these sources can be identified up to trivial indeterminacies. For the remaining sources in  $\mathbf{s}_D$ , they only need to be dependent on an auxiliary variable  $\mathbf{u}$  without necessitating conditional independence among sources or distributional assumption. This extends previous models that assume all sources to be conditionally independent given  $\mathbf{u}$  (Hyvärinen et al., 2019; Lachapelle et al., 2022) or require the conditional distribution of the sources to be of a specific form (Khemakhem et al., 2020b).
The assumption on  $p(\mathbf{s}_D|\mathbf{u})$  in Thm. 4.1 indicates that the auxiliary variable  $\mathbf{u}$  should have a sufficiently diverse impact on sources without independence assumption (i.e.,  $\mathbf{s}_D$ ). It follows a similar spirit to the standard assumption of variability (Hyvärinen et al., 2019) but we further relax it. Specifically, we only need  $n_D + 1$  values of  $\mathbf{u}$  for the identifiability of sources in  $\mathbf{s}_I$ . This is intuitively reasonable since the fewer changes (smaller  $n_D$ ) a system has, the easier (fewer required values, i.e.,  $n_D + 1$ ) that a larger part of it (larger  $n_I$ , i.e.,  $n - n_D$ ) is identifiable. In contrast, most previous works require all sources to be dependent on an auxiliary variable  $\mathbf{u}$  with  $2n + 1$  distinct values of  $\mathbf{u}$ : no identifiability for any subset of sources can be provided if there exists any degree of violations, either on the number of sources dependent on  $\mathbf{u}$  or the number of values of  $\mathbf{u}$ . This limits the application of these results to ideal scenarios where all sources are influenced by the same auxiliary variable with sufficient changes without any type of compromise. In practice, however, it is often the case that only a subset of sources benefit from the additional information provided by auxiliary variables, different auxiliary variables may affect different sources, or auxiliary variables do not contain sufficient information. Assumption ii in Thm. 4.1 is originally from (Kong et al., 2022) and also necessitate the presence of change. Intuitively, the chance of having a subset  $A_{\mathrm{s}}$  on which all domain distributions have an equal probability measure is very slim, which has been verified empirically in (Kong et al., 2022). For both theorems, we consider the more challenging undercomplete case, for which the related identifiability results are lacking in the literature. Additionally, unlike previous works assuming specific distributions of sources such as exponential families, we do not have similar distributional assumptions on the sources.
4.1 Results with flexible grouping structures
If we further have access to the dependence structure among variables in  $\mathbf{s}_D$ , additional identifiability results for these sources may also be established. For example, consider the setting that  $\mathbf{s}_D = (s_{n_I + 1},\dots ,s_n)$  can be decomposed to  $d$  irreducible independent subspaces  $\{\mathbf{s}_{c_1},\ldots ,\mathbf{s}_{c_d}\}$ , of which each is a multi-dimensional vector consisting multiple sources. We denote the  $j$ -th consecutive  $d$ -dimensional vector ( $j$ -th subspace) in  $\mathbf{s}$  as  $\mathbf{s}_{c_j} = (s_{(j - 1)d + 1},\dots ,s_{jd}) = (s_{c_j^{(l)}},\dots ,s_{c_j^{(h)}})$ , where  $s_{c_j^{(l)}}$  and  $s_{c_j^{(h)}}$  are the first and the last sources in  $\mathbf{s}_{c_j}$ , respectively. Then we have
$$
p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u}) = \prod_ {i = 1} ^ {n _ {i}} p _ {s _ {i}} \left(s _ {i}\right) \prod_ {j = c _ {1}} ^ {c _ {d}} p _ {\mathbf {s} _ {c _ {j}} | \mathbf {u}} \left(\mathbf {s} _ {c _ {j}} | \mathbf {u}\right). \tag {4}
$$
This is similar to Independent Subspace Analysis (ISA) (Hyvärinen and Hoyer, 2000; Theis, 2006) but we allow only a subset of sources as a composition of (conditionally) independent subspaces instead of all, which formalizes the tasks of blind source separation or uncovering latent variable models with mixtures of both high-dimensional and one-dimensional signals. The considered general setting essentially covers ICA and ISA as special cases: if  $n_I = n$ , it is consistent with the ICA problem; if  $n_I = 0$ , all sources can be decomposed into irreducible independent subspaces, and thus it becomes an ISA problem. The identifiability result under this setting is shown in the following theorem with its proof provided in Appx. A.4:
Theorem 4.3. Let the observed data be a large enough sample generated from an undercomplete nonlinear ICA model as defined in Eqs. (2) and (4). Suppose the following assumptions hold:
i. For each  $i \in \{1, \dots, n_I\}$ , there exist  $\{\mathbf{s}^{(\ell)}\}_{\ell=1}^{\left|\mathcal{F}_{i:n_I}\right|}$  and a matrix  $T \in \mathcal{T}$  s.t.  $\operatorname{span}\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i:n_I}\}_{\ell=1}^{\left|\mathcal{F}_{i:n_I}\right|} = \mathbb{R}_{\mathcal{F}_{i:n_I}}^{n_I}$  and  $\left[\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})\mathbf{T}\right]_{i:n_I} \in \mathbb{R}_{\hat{\mathcal{F}}_{i:n_I}}^{n_I}$ .
ii. There exist  $2n_{D} + 1$  values of  $\mathbf{u}$ , i.e.,  $\mathbf{u}_i$  with  $i \in \{0,1,\dots,2n_{D}\}$ , s.t. the  $2n_{D}$  vectors  $\mathbf{w}(\mathbf{s}_{D},\mathbf{u}_{i}) - \mathbf{w}(\mathbf{s}_{D},\mathbf{u}_{0})$  with  $i \in \{1,\dots,2n_{D}\}$  are linearly independent, where vector  $\mathbf{w}(\mathbf{s}_D,\mathbf{u}_i)$  is defined as follows:
$$
\mathbf {w} \left(\mathbf {s} _ {D}, \mathbf {u} _ {i}\right) = \left(\mathbf {v} \left(\mathbf {s} _ {c _ {1}}, \mathbf {u} _ {i}\right), \dots , \mathbf {v} \left(\mathbf {s} _ {c _ {d}}, \mathbf {u} _ {i}\right), \mathbf {v} ^ {\prime} \left(\mathbf {s} _ {c _ {1}}, \mathbf {u} _ {i}\right), \dots , \mathbf {v} ^ {\prime} \left(\mathbf {s} _ {c _ {d}}, \mathbf {u} _ {i}\right)\right),
$$
$$
\mathbf {v} \left(\mathbf {s} _ {c _ {j}}, \mathbf {u} _ {i}\right) = \left(\frac {\partial \log p \left(\mathbf {s} _ {c _ {j}} \mid \mathbf {u} _ {i}\right)}{\partial s _ {c _ {j} (l)}}, \dots , \frac {\partial \log p \left(\mathbf {s} _ {c _ {j}} \mid \mathbf {u} _ {i}\right)}{\partial s _ {c _ {j} (h)}}\right),
$$
$$
\mathbf {v} ^ {\prime} (\mathbf {s} _ {c _ {j}}, \mathbf {u} _ {i}) = \left(\frac {\partial^ {2} \log p (\mathbf {s} _ {c _ {j}} | \mathbf {u} _ {i})}{\left(\partial s _ {c _ {j} (l)}\right) ^ {2}}, \dots , \frac {\partial^ {2} \log p (\mathbf {s} _ {c _ {j}} | \mathbf {u} _ {i})}{\left(\partial s _ {c _ {j} (h)}\right) ^ {2}}\right).
$$
iii. There exist  $\mathbf{u}_1, \mathbf{u}_2 \in \mathbf{u}$ , s.t., for any set  $A_{\mathbf{s}} \subseteq S$  with nonzero probability measure and cannot be expressed as  $B_{\mathbf{s}_I} \times S_D$  for any  $B_{\mathbf{s}_I} \subset S_I$ , we have
iv. (Structural Sparsity) For all  $k \in \{1, \dots, n_I\}$ , there exists  $\mathcal{C}_k$  s.t.  $\bigcap_{i \in \mathcal{C}_k} \mathcal{F}_{i,:n_I} = \{k\}$ .
Then  $\mathbf{s}_I$  is identifiable up to an element-wise invertible transformation and a permutation, and  $\mathbf{s}_D$  is identifiable up to a subspace-wise invertible transformation and a subspace-wise permutation.
All assumptions align with the same principles as those elaborated in the theorems proposed above and have been adapted to cater to the flexible grouping structure. Specifically, in addition to the identifiability of sources in  $\mathbf{s}_I$ , we prove that we can also identify sources in  $\mathbf{s}_D$  up to an indeterminacy that, for each  $c_i \in \{c_1, \ldots, c_d\}$ , there exists an invertible transformation  $\mathbf{h}_{c_i}$  s.t.  $\mathbf{h}_{c_i}(\mathbf{s}_{c_i}) = \hat{\mathbf{s}}_{c_i}$ , which is analogous to the previous element-wise indeterminacy. Consequently, even when dealing with mixtures of high and one-dimensional sources, like in the case of multi-modal data, we can still recover the hidden generating process to some extent. Based on the aforementioned theoretical results, which consider undercompleteness, partial sparsity, and partial source dependence, Thm. 4.3 further generalizes the identifiability of nonlinear ICA by relaxing the dimensionality constraint of the latent generating factors.
In this vein, it is natural to consider another dependence structure, i.e., sources in  $s_D$  are not marginally but conditionally independent given an auxiliary variable  $\mathbf{u}$ . This is similar to the assumption made in most previous works on identifiable nonlinear ICA with surrogate information, which assume that all sources are conditionally independent of each other given the auxiliary variable. However, our setting is more flexible in the sense that we do not assume all sources to be influenced by the auxiliary variable. Specifically, sources in  $\mathbf{s}_I$  are mutually independent as in the original ICA setting, while only sources in  $\mathbf{s}_D$  have access to the side information from the conditional independence given  $\mathbf{u}$ , i.e.,
$$
p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u}) = \prod_ {i = 1} ^ {n _ {I}} p _ {s _ {i}} \left(s _ {i}\right) \prod_ {j = n _ {I} + 1} ^ {n} p _ {s _ {j} | \mathbf {u}} \left(s _ {j} | \mathbf {u}\right). \tag {5}
$$
The identifiability result for all sources  $(\mathbf{s}_I$  and  $\mathbf{s}_D)$  is as follows with proof in Appx. A.5:
Theorem 4.4. Let the observed data be a large enough sample generated from an undercomplete nonlinear ICA model as defined in Eqs. (2) and (5), suppose the following assumptions hold:
i. For each  $i \in \{1, \dots, n_I\}$ , there exist  $\{\mathbf{s}^{(\ell)}\}_{\ell=1}^{|F_{i:n_I}|}$  and a matrix  $T \in \mathcal{T}$  s.t.  $\operatorname{span}\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i:n_I}\}_{\ell=1}^{|F_{i:n_I}|} = \mathbb{R}_{\mathcal{F}_{i:n_I}}^{n_I}$  and  $[\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})T]_{i:n_I} \in \mathbb{R}_{\hat{\mathcal{F}}_{i:n_I}}^{n_I}$ .
ii. There exist  $2n_{D} + 1$  values of  $\mathbf{u}$ , i.e.,  $\mathbf{u}_i$  with  $i \in \{0,1,\dots,2n_{D}\}$ , s.t. the  $2n_{D}$  vectors  $\mathbf{w}(\mathbf{s}_{D},\mathbf{u}_{i}) - \mathbf{w}(\mathbf{s}_{D},\mathbf{u}_{0})$  with  $i \in \{1,\dots,2n_{D}\}$  are linearly independent, where vector  $\mathbf{w}(\mathbf{s}_{D},\mathbf{u})$  is defined as follows:
$$
\mathbf {w} \left(\mathbf {s} _ {D}, \mathbf {u} _ {i}\right) = \left(\mathbf {v} \left(\mathbf {s} _ {D}, \mathbf {u} _ {i}\right), \mathbf {v} ^ {\prime} \left(\mathbf {s} _ {D}, \mathbf {u} _ {i}\right)\right),
$$
$$
\mathbf {v} (\mathbf {s} _ {D}, \mathbf {u} _ {i}) = \left(\frac {\partial \log p (s _ {n _ {I} + 1} | \mathbf {u} _ {i})}{\partial s _ {n _ {I} + 1}}, \dots , \frac {\partial \log p (s _ {n} | \mathbf {u} _ {i})}{\partial s _ {n}}\right),
$$
$$
\mathbf {v} ^ {\prime} (\mathbf {s} _ {D}, \mathbf {u} _ {i}) = \left(\frac {\partial^ {2} \log p (s _ {n _ {I} + 1} | \mathbf {u} _ {i})}{(\partial s _ {n _ {I} + 1}) ^ {2}}, \dots , \frac {\partial^ {2} \log p (s _ {n} | \mathbf {u} _ {i})}{(\partial s _ {n}) ^ {2}}\right).
$$
iii. There exist  $\mathbf{u}_1, \mathbf{u}_2 \in \mathbf{u}$ , s.t., for any set  $A_{\mathbf{s}} \subseteq \mathcal{S}$  with nonzero probability measure and cannot be expressed as  $B_{\mathbf{s}_I} \times \mathcal{S}_D$  for any  $B_{\mathbf{s}_I} \subset \mathcal{S}_I$ , we have
$$
\int_ {\mathbf {s} \in A _ {\mathbf {s}}} p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u} _ {1}) d \mathbf {s} \neq \int_ {\mathbf {s} \in A _ {\mathbf {s}}} p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u} _ {2}) d \mathbf {s}.
$$
With different assumptions on different sets of sources, one could view this theorem as an expansion of both previous theoretical findings that impose distributional constraints on sources with auxiliary variables (e.g., (Hyvärinen et al., 2019)) and those that constrain the mixing function with Structural Sparsity (Zheng et al., 2022). This is particularly helpful in the context of self-supervised learning (Von Kugelgen et al., 2021) or transfer learning (Kong et al., 2022), where latent representations are modeled as a changing part and an invariant part. In (Kong et al., 2022), the component-wise identifiability for variables changing across domains (i.e.,  $\mathbf{s}_D$  with multiple values of  $\mathbf{u}$  in our setting) are provided but not those in the invariant part (i.e.,  $\mathbf{s}_I$  in our setting). With the help of Thm. 4.4, we can show identifiability up to an element-wise invertible transformation and a permutation for each source, regardless of whether it changes across domains or not, which may help some related tasks where full identifiability is necessary. Furthermore, for causal reasoning or disentanglement with observational time-series data, our theorem benefits the identifiability of temporal processes involving instantaneous relations. Previous works in that area can only deal with time-delayed/changing influences, as they rely on the global conditional independence of all sources given the changing time index as the auxiliary variable (Hyvärinen and Morioka, 2016, 2017; Yao et al., 2022). Our theorem, on the other hand, provides the added ability to identify unconditional sources, thanks to the partially satisfied sparsity assumption, and thus aids the uncovering of latent processes with instantaneous relations.
In order to validate the proposed identifiability results, we conduct experiments using both simulated data and real-world images. It is noteworthy that there has been extensive research that has empirically verified that deep latent variable models are likely to be identifiable in complex scenarios, particularly in the disentanglement task (Kumar et al., 2017; Klys et al., 2018; Locatello et al., 2018; Rubenstein et al., 2018; Chen et al., 2018; Burgess et al., 2018; Duan et al., 2020; Falck et al., 2021; Carbonneau et al., 2022). While we are not sure of the exact inductive biases or side information that are available during the real-world application, which has been proved to be necessary (Hyvärinen and Pajunen, 1999; Locatello et al., 2019), the empirical success of these methods sheds light on the possibility of identification in the general settings considered in this work.
Setup. For settings with the auxiliary variable  $\mathbf{u}$ ,  $\mathbf{u}$  is always available during estimation and we consider the dataset as  $\mathcal{D} = \left\{\left(\mathrm{x}^{(1)},\mathrm{u}^{(1)}\right),\dots ,\left(\mathrm{x}^{(N)},\mathrm{u}^{(N)}\right)\right\}$ , where  $N$  is the sample size and  $\mathbf{u}^{(i)}$  is the value of  $\mathbf{u}$  (or class label) corresponding to the data point  $\mathbf{x}^{(i)}$ . Given the estimated model  $\hat{f}$  parameterized by  $\theta$ , similar to (Sorrenson et al., 2020), we consider a regularized maximum-likelihood approach for the required sparsity regularization during estimation with the objective function as:  $\mathcal{L}(\theta) = \mathbb{E}_{(\mathbf{x},\mathbf{u})\in \mathcal{D}}\left[\log p_{\hat{f}^{-1}}(\mathbf{x}|\mathbf{u}) - \lambda \mathbf{R}\right]$ , where  $\lambda \in [0,1]$  is a regularization parameter and  $\mathbf{R}$  is the regularization term on the Jacobian of the estimated mixing function, i.e.,  $\mathbf{J}_{\hat{f}}$ . Based on our experimental results (Fig. 8 in Appx. B.2), we adopt the minimax concave penalty (MCP) (Zhang, 2010) as the regularization term. For settings without the auxiliary variable, we remove the access of  $\mathbf{u}$  and follow the same objective function in (Zheng et al., 2022). We train a General Incompressible-flow Network (GIN) (Sorrenson et al., 2020), which is a flow-based generative model, to maximize the objective function  $\mathcal{L}(\cdot)$ . Following (Sorrenson et al., 2020), where necessary, we concatenate the latent sources with independent Gaussian noises to meet the dimensionality requirements. All results are from 20 trials with random seeds. Additional details of the experimental setup are included in Appx. B.
Ablation study. We perform an ablation study to verify the necessity of the proposed assumptions. Specifically, we focus on the following models corresponding to different assumptions: (UCSS) The assumption of Structural Sparsity, as well as other assumptions in the undercomplete case (Thm. 3.1), are satisfied; (Mixed) The assumption of Structural Sparsity with undercompleteness and the required dependence structure among sources influenced by the auxiliary variable, as well as other assumptions in the partial sparsity and dependence case (Thm. 4.4), are satisfied; (Base) The vanilla baseline in the undercomplete case, where the assumption of Structural Sparsity is not satisfied compared to UCSS. The datasets are generated according to the required assumptions, the details of which are included in Appx. B. All experiments are conducted in the undercomplete case, where the number of observed variables is twice the number of sources. For datasets that contain both sources in  $\mathbf{s}_I$  and  $\mathbf{s}_D$  (Mixed case), we set half as  $\mathbf{s}_I$  and the other half as  $\mathbf{s}_D$ , and the minimum required
iv. (Structural Sparsity) For all  $k \in \{1, \dots, n_I\}$ , there exists  $\mathcal{C}_k$  s.t.  $\bigcap_{i \in \mathcal{C}_k} \mathcal{F}_{i:n_I} = \{k\}$ .
Figure 2: MCC of UCSS w.r.t. different number of sources.
Figure 4: Percentage of random structures satisfying Structural Sparsity w.r.t. different degree of undercompleteness (i.e.,  $m / n$ ).
Figure 3: MCC of Mixed w.r.t. different number of sources.
Figure 5: Percentage of sources satisfying Structural Sparsity w.r.t. different numbers of sources in the bijective setting  $(m / n = 1)$
numbers of required distinct values have been assigned to the auxiliary variable  $\mathbf{u}$ . Following previous works (Hyvarinen and Morioka, 2016; Lachapelle et al., 2022), we use the mean correlation coefficient (MCC) between the true sources and the estimated ones as the evaluation metric.
Results for each model are summarised in Fig. 2 and Fig. 3. It can be observed that when the proposed assumptions are met (UCSS and Mixed), our models achieve higher MCCs than Base. This indicates that it is indeed possible to identify sources from nonlinear mixtures up to trivial indeterminacy in the general settings with undercompleteness, partial sparsity, and partial source dependence. Additionally, we conduct experiments with different numbers of sources  $n$  to evaluate the stability of the identification. Our results show that both models consistently outperform Base across all values of  $n$ , further supporting the theoretical claims.
Undercomplete Structural Sparsity. As previously discussed, the assumption of Structural Sparsity (Assumption ii in Thm. 3.1) is far more plausible in an undercomplete setting considered in our theory (i.e., the number of observed variables  $m$  is larger than the number of sources  $n$ ) than in the more restrictive bijective scenario required in (Zheng et al., 2022) (i.e., the numbers are equal,  $m = n$ ). Consequently, extending the identifiability with structural sparsity from a bijective to an undercomplete setting significantly broadens its applicability in real-world contexts. In order to validate the necessity of the proposed generalization empirically, we construct several experiments studying the Structural Sparsity assumption in the undercomplete case. We consider different numbers of sources  $n$  with different degrees of undercompleteness ( $m/n$ , where  $m$  is the number of observed variables). For each setting, we generate 50 random matrices where each entry is independently determined with an equal probability to be either zero or non-zero. The results of the percentages of matrices satisfying the assumption of Structural Sparsity are presented in Fig. 4. We could observe that there exists a significant gap on the percentages between the cases where  $m/n = 1$ , i.e., the bijective setting, and the undercomplete settings where  $m/n > 1$ . Thus, it is clear that the assumption is much more likely to hold true when we have more observed variables than sources. Furthermore, when the degree of undercompleteness increases, the percentage of cases satisfying structural converges to 1. This further suggests that the assumption will almost always hold with a sufficient degree of undercompleteness, which is rather common in practice. For instance, a photo can easily have millions of pixels (observed variables) but only a dozen of hidden concepts (sources).
Partial Structural Sparsity. Moreover, as previously noted, it is not uncommon for Structural Sparsity to be violated for a subset of sources. For instance, certain sources (such as high-decibel sound sources) may exert influence over all observed variables (microphones). Nonetheless, the
Figure 6: Results on Triangles. The rows may correspond to rotation, height, width, and brightness, respectively.
Figure 7: Results on EMNIST. The rows may correspond to line thickness, angle, upper width, and height, respectively.
prior study (Zheng et al., 2022) necessitates that the sparsity assumption holds true for all sources, providing no identifiability assurance in cases of any degree of violation. To confront this practical obstacle, we propose Thm. 4.1 and Thm. 4.2 to demonstrate that the remaining sources (i.e.,  $n_I$  sources) can still be identified even when the sparsity assumption does not universally hold for some sources. These results can also be motivated empirically. For instance, from Fig. 4, one may find that Structural Sparsity does not likely to hold for all sources when  $m/n = 1$ , which is the bijective setting considered in (Zheng et al., 2022). However, as discussed above, if a subset of sources satisfies the assumption, at least the identifiability for these sources could be guaranteed by our proposed theorems (Thm. 4.1 and Thm. 4.2) under certain conditions. To illustrate this, we conduct experiments in the bijective setting  $(m/n = 1)$  and report the percentage of sources satisfying Structural Sparsity in Fig. 4. We consider datasets with different number of sources and generate 50 random matrices for each of these. Each entry is independently determined with an equal probability to be either zero or non-zero. Combining results from both Fig. 4 and Fig. 5, we observe that, even in scenarios where Structural Sparsity is rarely satisfied for all sources  $(m/n = 1$  in Fig. 4), it is almost always satisfied for a significant fraction of sources (Fig. 5). Consequently, our generalization also proves helpful even within the confines of the earlier bijective setting.