SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yujia Zheng $^{1}$ , Kun Zhang $^{1,2}$
1 Carnegie Mellon University
$^{2}$  Mohamed bin Zayed University of Artificial Intelligence
{yujiazh, kunz1}@cmu.edu
Nonlinear independent component analysis (ICA) aims to uncover the true latent sources from their observable nonlinear mixtures. Despite its significance, the identifiability of nonlinear ICA is known to be impossible without additional assumptions. Recent advances have proposed conditions on the connective structure from sources to observed variables, known as Structural Sparsity, to achieve identifiability in an unsupervised manner. However, the sparsity constraint may not hold universally for all sources in practice. Furthermore, the assumptions of bijectivity of the mixing process and independence among all sources, which arise from the setting of ICA, may also be violated in many real-world scenarios. To address these limitations and generalize nonlinear ICA, we propose a set of new identifiability results in the general settings of undercompleteness, partial sparsity and source dependence, and flexible grouping structures. Specifically, we prove identifiability when there are more observed variables than sources (undercomplete), and when certain sparsity and/or source independence assumptions are not met for some changing sources. Moreover, we show that even in cases with flexible grouping structures (e.g., part of the sources can be divided into irreducible independent groups with various sizes), appropriate identifiability results can also be established. Theoretical claims are supported empirically on both synthetic and real-world datasets.
The unveiling of the true generating process of observations is fundamental to scientific discovery. Nonlinear independent component analysis (ICA) provides a statistical framework that represents a set of observed variables  $\mathbf{x}$  as a nonlinear mixture of independent latent sources  $\mathbf{s}$ , i.e.,  $\mathbf{x} = \mathbf{f}(\mathbf{s})$ . Unlike linear ICA (Comon, 1994), the mixing function  $\mathbf{f}$  can be an unknown nonlinear function, thus generalizing the theory to more real-world tasks. However, the identifiability of nonlinear ICA has been a long-standing problem for decades. The main obstacle is that, without additional assumptions, there exist infinite spurious solutions returning independent variables that are mixtures of the true sources (Hyvarinen and Pajunen, 1999). In the context of machine learning, this makes the theoretical analysis of unsupervised learning of disentangled representations difficult (Locatello et al., 2019).
To overcome this challenge, recent work has introduced the auxiliary variable  $\mathbf{u}$ , and assumed that all sources are conditionally independent given  $\mathbf{u}$ . Most of these methods require auxiliary variables to be observable, such as class labels and domain indices (Hyvärinen and Morioka, 2016; Hyvärinen et al., 2019; Khemakhem et al., 2020a; Sorrenson et al., 2020; Lachapelle et al., 2022; Lachapelle and Lacoste-Julien, 2022), with the exceptions being those for time series (Hyvärinen and Morioka, 2017; Hälvä et al., 2021; Yao et al., 2021, 2022). While the use of the auxiliary variable  $\mathbf{u}$  allows for the identifiability of nonlinear ICA with mild restrictions on the mixing process, it also necessitates a large number of distinct values of  $\mathbf{u}$ , which can be difficult to obtain in tasks with insufficient side informa
tion. Moreover, since these results assume that all sources are dependent on  $\mathbf{u}$ , they cannot accommodate a subset of sources with invariant distributions (e.g., content may not change with different styles).
Another possible direction is to impose appropriate conditions on the mixing process, but limited results are available in the literature. For example, it has been shown that conformal maps are identifiable up to specific indeterminacies (Hyvärinen and Pajunen, 1999; Buchholz et al., 2022). Moreover, Taleb and Jutten (1999) identify the latent sources when the mixing process is a component-wise nonlinear function added to a linear mixture. These methods do not rely on conditional independence given the auxiliary variable and thus achieve the identifiability in a fully unsupervised setting. At the same time, the requirement of above-mentioned classes of the mixing function, such as conformal maps and post-nonlinear models, restricts the applicability of the results in another way. For instance, according to Liouville's theorem (Monge, 1850), conformal maps in Euclidean spaces of dimensions higher than two are Möbius transformations, which appear to be overly restrictive for most data-generating processes. As an alternative, Zheng et al. (2022) prove that, under the assumption of Structural Sparsity, the true sources can be identified up to trivial indeterminacies. Since the proposed condition is on the connective structure from sources to observed variables, i.e., the support of the Jacobian matrix of the mixing function, it does not require the mixing function to be of any specific algebraic form. Thus, Structural Sparsity may serve as one of the first general principles for the identifiability of nonlinear ICA in a fully unsupervised setting.
While being a potential solution to the identifiability of nonlinear ICA without side information, the assumption of Structural Sparsity has its limitations from a pragmatic viewpoint. The most obvious one arises from the fact that it may fail in a number of situations where the generating processes are heavily entangled. Although the principle of simplicity may be a general rule in nature, it is intuitively possible that Structural Sparsity does not apply to at least a subset of sources, such as one or a few speakers in a crowded room. Unfortunately, Zheng et al. (2022) require Structural Sparsity to hold for all sources in order to provide any identifiability guarantee. Therefore, it would be desirable in practice to provide weaker notions of identifiability, such as the ability to identify a subset of sources to a trivial degree of uncertainty, in cases of partial sparsity.
In addition to partial sparsity, identifiability with Structural Sparsity also fails with the undercompleteness (more observed variables than sources) and/or partial source dependence (potential dependence among some hidden sources). These limitations are not unique to the sparsity assumption, but rather a result of the traditional setting of ICA, where the numbers of the sources and observed variables must be equal and dependencies among sources are not allowed. However, both situations are quite common in practice. One may easily have millions of pixels (observed variables) but only dozens of hidden concepts (sources) in a picture, constituting an undercomplete case that cannot be handled by previous results. Meanwhile, dependencies among some variables are also prevalent in tasks such as computational biology (Cardoso, 1998; Theis, 2006). The alternative assumption of conditional independence given auxiliary variables may still be overly restrictive if applied universally to all sources. For the identifiability of nonlinear ICA to truly benefit scientific discovery in a wider range of scenarios, these methodological limitations should be properly addressed.
Aiming to generalize nonlinear ICA with Structural Sparsity, we first present a set of new identifiability results to address these fundamental challenges of undercompleteness, partial sparsity, and source dependence. We show that, under the assumption of Structural Sparsity and without auxiliary variables, latent sources can be identified from their nonlinear mixtures up to a component-wise invertible transformation and a permutation, even when there are more observed variables than sources (Thm. 3.1). Moreover, if the assumption of sparsity and/or source independence does not hold for some changing sources, we provide partial identifiability results, showing that the remaining sources can still be identified up to the same trivial indeterminacy (Thm. 4.1, Thm. 4.2). Furthermore, in the cases with flexible grouping structures (e.g., part of the sources can be grouped into irreducible independent subgroupings with various sizes, such as mixtures of signals with various dimensions), certain types of identifiability are also guaranteed with auxiliary variables (Thm. 4.3, Thm. 4.4). Therefore, we establish, to the best of our knowledge, one of the first general frameworks for uncovering latent variables with appropriate identifiability guarantees in a principled manner. The theoretical claims are validated empirically through our experiments and many previous works involving disentanglement.
The data-generating process of nonlinear ICA is as follows:
$$
p _ {\mathbf {s}} (\mathbf {s}) = \prod_ {i = 1} ^ {n} p _ {\mathbf {s} _ {i}} (\mathbf {s} _ {i}), \tag {1}
$$
$$
\mathbf {x} = \mathbf {f} (\mathbf {s}), \tag {2}
$$
where  $\mathbf{s} = (\mathbf{s}_1, \ldots, \mathbf{s}_n) \in S \subseteq \mathbb{R}^n$  is a latent vector representing the independent sources, and  $\mathbf{x} = (\mathbf{x}_1, \ldots, \mathbf{x}_m) \in \mathcal{X} \subseteq \mathbb{R}^m$  denotes the observed random vector. The mixing function  $\mathbf{f}$  is assumed to be smooth in the sense that its second-order derivatives exist. The primary objective of ICA is to establish identifiable models, i.e., the sources  $\mathbf{s}$  are identifiable (recoverable) up to certain indeterminacies by learning an estimated mixing function  $\hat{\mathbf{f}}: \hat{S} \to \mathcal{X}$  with assumptions identical to the generating process (Comon, 1994). Different from most ICA results where  $m = n$  and  $\mathbf{f}: S \to \mathcal{X}$  must be linear, we allow  $m > n$  (i.e., undercompleteness) and  $\mathbf{f}$  to be a general nonlinear function, therefore extending the previous setting. Thus, we relax the previous assumption on the invertibility of  $\mathbf{f}$ , only necessitating it to be injective and its Jacobian to be of full column rank. Furthermore, we denote  $p_{s_i}$  as the marginal probability density function (PDF) of the  $i$ -th source  $s_i$  and  $p_{\mathbf{s}}$  as the joint PDF of the random vector  $\mathbf{s}$ . Moreover, we introduce some additional technical notations as follows:
Definition 2.1. Given a subset  $\mathcal{A} \subseteq \{1, \dots, n\}$ , the subspace  $\mathbb{R}_{\mathcal{A}}^{n}$  is defined as
$$
\mathbb {R} _ {\mathcal {A}} ^ {n} := \left\{z \in \mathbb {R} ^ {n} \mid i \notin \mathcal {A} \Longrightarrow z _ {i} = 0 \right\},
$$
where  $z_{i}$  is the  $i$ -th element of the vector  $z$ .
That is,  $\mathbb{R}_A^n$  denotes the subspace of  $\mathbb{R}^n$  specified by an index set  $\mathcal{A}$ . Furthermore, we define the support of a matrix as follows:
Definition 2.2. The support of a matrix  $\mathbf{M} \in \mathbb{R}^{m \times n}$  is defined as
$$
\operatorname {s u p p} (\mathbf {M}) := \left\{(i, j) \mid \mathbf {M} _ {i, j} \neq 0 \right\}.
$$
With a slight abuse of notation, we reuse  $\mathrm{supp}(\cdot)$  to denote the support of a matrix-valued function:
Definition 2.3. The support of a function  $\mathbf{M}:\Theta \to \mathbb{R}^{m\times n}$  is defined as
$$
\operatorname {s u p p} (\mathbf {M} (\Theta)) := \left\{(i, j) \mid \exists \theta \in \Theta , \mathbf {M} (\theta) _ {i, j} \neq 0 \right\}.
$$
For brevity, we denote  $\mathcal{F}$  and  $\hat{\mathcal{F}}$  as the support of the Jacobian  $\mathbf{J}_{\mathbf{f}}(\mathbf{s})$  and  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}})$ , respectively. Additionally,  $\mathcal{T}$  refers to a set of matrices with the same support of  $\mathbf{T}(\mathbf{s})$  in  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}}) = \mathbf{J}_{\mathbf{f}}(\mathbf{s})\mathbf{T}(\mathbf{s})$ , where  $\mathbf{T}(\mathbf{s})$  is a matrix-valued function. Throughout this work, for any matrix  $\mathbf{M}$ , we use  $\mathbf{M}_{i,:}$  to denote its  $i$ -th row, and  $\mathbf{M}_{i,:j}$  to denote its  $j$ -th column. For any set of indices  $\mathcal{B} \subset \{1,\dots,m\} \times \{1,\dots,n\}$ , analogously, we have  $\mathcal{B}_{i,:} := \{j \mid (i,j) \in \mathcal{B}\}$  and  $\mathcal{B}_{i,:j} := \{i \mid (i,j) \in \mathcal{B}\}$ .
3 Identifiability with undercompleteness
We first present the result on removing one of the major assumptions in ICA, i.e., the number of observed variables  $m$  must be equal to that of hidden sources  $n$ . We prove that, in the undercomplete case  $(m > n)$ , sources can be identified up to a trivial indeterminacy under Structural Sparsity.
Theorem 3.1. Let the observed data be a large enough sample generated by an undercomplete nonlinear ICA model as defined in Eqs. (1) and (2). Suppose the following assumptions hold:
i. For each  $i \in \{1, \ldots, n\}$ , there exist  $\{\mathbf{s}^{(\ell)}\}_{\ell=1}^{\left|\mathcal{F}_{i,:}\right|}$  and a matrix  $T \in \mathcal{T}$  s.t.  $\operatorname{span}\{\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})_{i,:}\}_{\ell=1}^{\left|\mathcal{F}_{i,:}\right|} = \mathbb{R}_{\mathcal{F}_{i,:}}^{n}$  and  $\left[\mathbf{J}_{\mathbf{f}}(\mathbf{s}^{(\ell)})\mathbf{T}\right]_{i,:} \in \mathbb{R}_{\hat{\mathcal{F}}_{i,:}}^{n}$ .
ii. (Structural Sparsity) For each  $k \in \{1, \dots, n\}$ , there exists  $\mathcal{C}_k$  s.t.  $\bigcap_{i \in \mathcal{C}_k} \mathcal{F}_{i,:} = \{k\}$ .
Then s is identifiable up to an element-wise invertible transformation and a permutation.
The proof is included in Appx. A.1, of which part of the conditions and techniques are based on (Zheng et al., 2022). It is noteworthy that, same as previous work, we also need to add a sparsity
regularization on the learned Jacobian during the estimation so that  $|\hat{\mathcal{F}}| \leq |\mathcal{F}|$ , which is required for all sparsity-based identifications throughout the paper and we only emphasize here for brevity.
Assumption i avoids some pathological conditions (e.g., samples are from very limited subpopulations that only span a degenerate subspace) and is typically satisfied asymptotically. The first part implies that there are at least  $|\mathcal{F}_{i,:}|$  observed samples spanning the support space, which is almost always satisfied asymptotically. The second part is also relatively mild. Note that  $\mathcal{T}$  refers to a set of matrices with the same support of  $\mathbf{T}(\mathbf{s})$  in  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}}) = \mathbf{J}_{\mathbf{f}}(\mathbf{s})\mathbf{T}(\mathbf{s})$  and  $\mathbf{J}_{\hat{\mathbf{f}}}(\hat{\mathbf{s}})_{i,:} \in \mathbb{R}_{\mathcal{F}_{i,:}}^{n}$ . Since we only necessitate the existence of one matrix  $\mathrm{T} \in \mathcal{T}$  in the entire space, even in rare cases where these two matrices do not share the same non-zero coordinates due to non-generic canceling between specific values of elements, there is almost always an existence of a matrix  $\mathrm{T} \in \mathcal{T}$  fulfilling the assumption.
Assumption ii, i.e., Structural Sparsity, originates from (Zheng et al., 2022). Intriguingly, compared to the original bijective setting considered by (Zheng et al., 2022), this assumption is much more likely to be satisfied in the undercomplete case. The key reason is that it only necessitates the existence of a subset of observed variables whose intersection uniquely identifies the target source variable. For instance, regarding  $\mathbf{s}_1$  in Fig. 1, there exist  $\mathbf{x}_1$  and  $\mathbf{x}_4$  s.t. the intersection of their parents is only  $\mathbf{s}_1$ . In principle, the size of this set can be quite small (e.g., one or two). Hence, it is very likely to be satisfied when there is a sufficient number of observed
Figure 1: The structural sparsity assumption in the undercomplete case, where the matrix represents  $\mathrm{supp}(\mathbf{J}_{\mathbf{f}}(\mathbf{s}))$
variables (e.g., millions of pixels for images), which has also been verified empirically in our experiments (e.g., Fig. 4 in Sec. 5). Additionally, in some tasks, we might even construct or select observations in a data-centric manner to satisfy this assumption. Without the previous constraint of bijectivity, structural sparsity can truly be applied in a much broader range of practical scenarios.
By proving the identifiability in the undercomplete case, we remove the previous assumption of bijectivity on the mixing function  $\mathbf{f}$  and thus generalizing the theory to more application scenarios. It is worth noting that, while some work has provided results without assuming bijectivity (Khemakhem et al., 2020a), they rely on extra information from many distinct values of auxiliary variables. Differently, we do not need any auxiliary variables and follow a fully unsupervised setting; Zheng et al. (2022); Kivva et al. (2022) explore the undercompleteness without any auxiliary variable. However, Zheng et al. (2022) only remove the rotational indeterminacy in the nonlinear case and Kivva et al. (2022) assume Gaussian mixture priors, while we provide the full identifiability result without distributional assumptions. At the same time, as elaborated above, the assumption of Structural Sparsity has been significantly weakened in the undercomplete case considered by our theorem. Moreover, identifiability with undercompleteness is also essential if assumptions are partially violated w.r.t. a subset of sources, of which the intuition is verified by theoretical results introduced in the following sections.
4 Identifiability with partial sparsity and source dependence
Under the condition of Structural Sparsity, we show the identifiability of undercomplete ICA with general nonlinear functions (Thm. 3.1). While this removes the restriction of bijectivity between sources and observed variables, it remains uncertain as to whether Structural Sparsity holds for all sources in a universal way. At the same time, even in the scenarios that Structural Sparsity may not be universally satisfied for all sources, it is still valuable to consider its potential to hold true for a subset of sources. This type of partial sparsity may often be the case in practical scenarios, as illustrated by our experiments (e.g., Fig. 5 in Sec. 5). However, the corresponding partial identifiability, i.e., the theoretical guarantee for the identification of the subset of sources satisfying Structural Sparsity, is not achieved by (Zheng et al., 2022). In fact, as long as one or a few sources do not meet the assumption of Structural Sparsity, the previous work is unable to provide any identifiability guarantees.
Furthermore, in addition to the universal sparsity, the statistical independence between sources is another fundamental assumption. This assumption arises from the original setting of ICA and has been adopted in most related works. However, in many real-world scenarios, requiring all sources to be mutually independent might be impractical, and there are likely to be a subset of sources that
are dependent in some way. For example, the frequency and duration of smoking, as well as the type of tobacco products used, are all interrelated factors that contribute to the development of lung cancer. Without any identifiability guarantees in the case where there exist any dependent sources, it is quite restrictive for nonlinear ICA, and even its undercomplete extension, to successfully deal with real problems in practice. Therefore, similar to the partial sparsity case, it is highly desirable that alternative theoretical results, i.e., identifiability for independent sources, can be guaranteed even with the existence of dependent sources.
To deal with these remaining challenges in the considered undercomplete case, we further relax other assumptions with additional information. To start with, we provide an identifiability result when Structural Sparsity holds true for only a subset of sources, thus alleviating the obstacle of partial sparsity. Moreover, we relax the mutual independence assumption and allow for some changing sources to be dependent on each other. To this end, we partition the sources into two parts  $\mathbf{s} = [\mathbf{s}_I,\mathbf{s}_D]$ , where variables in  $\mathbf{s}_I$  are mutually independent, but those in  $\mathbf{s}_D$  do not need to be. Let  $\mathbf{s}_I$  and  $\mathbf{s}_D$  correspond to variables in  $\mathbf{s}$  with indices  $\{1,\ldots ,n_I\}$  and  $\{n_{I + 1},\dots ,n\}$ , respectively. That is,  $\mathbf{s}_I = (s_1,\dots ,s_{n_I})\in \mathcal{S}_I\subseteq \mathbb{R}^{n_I}$  and  $\mathbf{s}_D = (s_{n_I + 1},\dots ,s_n)\in \mathcal{S}_D\subseteq \mathbb{R}^{n_D}$ . We denote the  $i$ -th scalar element in a vector, say  $\mathbf{s}$ , as  $s_i$ . For sources in  $\mathbf{s}_D$ , they do not need to be mutually independent as long as they are dependent on a variable  $\mathbf{u}$ , i.e.,
$$
p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} | \mathbf {u}) = p _ {\mathbf {s} _ {D} | \mathbf {u}} (\mathbf {s} _ {D} | \mathbf {u}) \prod_ {i = 1} ^ {n _ {i}} p _ {s _ {i}} \left(s _ {i}\right). \tag {3}
$$
It is noteworthy that we allow arbitrary relations between sources in  $\mathbf{s}_D$ . These sources might be grouped into several subspaces or actually be mutually independent, but we do not need to obtain this information as prior knowledge. This is essential since the exact dependence structures, or even the number of dependent variables, are usually unknown in practice. By keeping this type of uncertainty for both partial sparsity and/or partial source dependence, one can be more confident in applying the theoretical advancements of nonlinear ICA in various tasks. We prove the identifiability for this arguably more flexible scenario as the following theorems:
Theorem 4.1. Let the observed data be a large enough sample generated by an undercomplete nonlinear ICA model defined in Eqs. (2) and (3). Suppose the following assumptions hold:
i. There exist  $n_D + 1$  distinct values of  $\mathbf{u}$ , i.e.,  $\mathbf{u}_j$  with  $j \in \{0, 1, \dots, n_D\}$ , s.t. the  $n_D$  vectors  $\mathbf{w}(\mathbf{s}_D, \mathbf{u}, i)$  with  $i \in \{n_I + 1, \dots, n\}$  are linearly independent, where vector  $\mathbf{w}(\mathbf{s}_D, \mathbf{u}, i)$  is defined as follows:
$$
\mathbf {w} (\mathbf {s} _ {D}, \mathbf {u}, i) = \Big (\frac {\partial (\log p (\mathbf {s} _ {D} | \mathbf {u} _ {1}) - \log (p (\mathbf {s} _ {D} | \mathbf {u} _ {0}))}{\partial s _ {i}}, \ldots , \frac {\partial (\log p (\mathbf {s} _ {D} | \mathbf {u} _ {n _ {D}}) - \log (p (\mathbf {s} _ {D} | \mathbf {u} _ {0}))}{\partial s _ {i}} \Big).
$$
ii. There exist  $\mathbf{u}_1, \mathbf{u}_2 \in \mathbf{u}$ , s.t., for any set  $A_{\mathbf{s}} \subseteq S$  with non-zero probability measure and cannot be expressed as  $B_{\mathbf{s}_I} \times \mathbf{s}_D$  for any  $B_{\mathbf{s}_I} \subset S_I$ , we have
$$
\int_ {\mathbf {s} \in A _ {\mathbf {s}}} p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} \mid \mathbf {u} _ {1}) d \mathbf {s} \neq \int_ {\mathbf {s} \in A _ {\mathbf {s}}} p _ {\mathbf {s} | \mathbf {u}} (\mathbf {s} \mid \mathbf {u} _ {2}) d \mathbf {s}.
$$
Then  $\mathbf{s}_D$  is identifiable up to an subspace-wise invertible transformation.
Thm. 4.1 ensures the subspace-wise identifiability of  $\mathbf{s}_D$ , i.e., the estimated subspace  $\hat{\mathbf{s}}_D$  contains all and only information from  $\mathbf{s}_D$ . This implies that we can disentangle and extract the invariant part of the latents, beneficial for tasks like domain adaptation where recovering each individual source might not be necessary as long as the subspace that is invariant across domains can be disentangled. Furthermore, we also prove the component-wise identifiability as follows:
Theorem 4.2. In addition to assumptions in Thm. 4.1, suppose the following assumptions hold: