Diverse Dictionary Learning
Authors: Yujia Zheng
Zijian Li
Shunxing Fan
Andrew Gordon Wilson
Kun Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Dictionary Learning
Identifiability Theory
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: For general latent variable models, what remains recoverable with guarantees, and what inductive biases are universally helpful?

Abstract: Given only observational data $X = g(Z)$, where both the latent variables $Z$ and the generating process $g$ are unknown, recovering $Z$ is ill-posed without additional assumptions. Existing methods often assume linearity or rely on auxiliary supervision and functional constraints. However, such assumptions are rarely verifiable in practice, and most theoretical guarantees break down under even mild violations, leaving uncertainty about how to reliably understand the hidden world. To make identifiability *actionable* in the real-world scenarios, we take a complementary view: in the general settings where full identifiability is unattainable, *what can still be recovered with guarantees*, and *what biases could be universally adopted*? We introduce the problem of *diverse dictionary learning* to formalize this view. Specifically, we show that intersections, complements, and symmetric differences of latent variables linked to arbitrary observations, along with the latent-to-observed dependency structure, are still identifiable up to appropriate indeterminacies even without strong assumptions. These set-theoretic results can be composed using set algebra to construct structured and essential views of the hidden world, such as *genus-differentia* definitions. When sufficient structural diversity is present, they further imply full identifiability of all latent variables. Notably, all identifiability benefits follow from a simple inductive bias during estimation that can be readily integrated into most models. We validate the theory and demonstrate the benefits of the bias on both synthetic and real-world data.

Published as a conference paper at ICLR 2026
DIVERSE DICTIONARY LEARNING
Yujia Zheng' Zijian Li²
Shunxing Fan
Andrew Gordon Wilson Kun Zhang
CMU
MBZUAI NYU
ABSTRACT
Given only observational data X = ( where both the latent variables Z and the generating process 9 are unknown, recovering Z ill-posed without addi- tional assumptions. Existing methods often assume linearity or rely on auxiliary supervision and functional constraints. However, such assumptions are rarely ver- ifiable in practice, and most theoretical guarantees break down under even mild violations, leaving uncertainty about how to reliably understand the hidden world. To make identifiability actionable in the real-world scenarios, we take comple- mentary view: in the general settings where full identifiability is unattainable, what still be recovered with guarantees, and what biases could be universally adopted? We introduce the problem of diverse dictionary learning to formalize this view. Specifically, show that intersections, complements, and symmet- ric differences of latent variables linked to arbitrary observations, along with the cecentto-observee dependency structure, are still identifiable up to appropriate in- determinacies even without strong assumptions. These set-theoretic results can composed using set algebra to construct structured and essential views of the hidden world, such as gemus-differentia definitions. When sufficient structural diversity is present, they further imply full identifiability of all latent variables. Notably, all identifiability benefits follow from a simple inductive bias during es- timation that can readily integrated into most models. We validate the theory and demonstrate the benefits of the bias both synthetic and real-world data.
INTRODUCTION
Dictionary learning, in its most general form, assumes that observations X are generated by latent variables through an unknown function f. i.e., X = f(Z). The goal to recover the latent generative process from observational data, a fundamental task in both science and machine learning. The nonparametric formulation X = f(2) unifies wide range of latent variable models, including independent component analysis, factor analysis, and causal representation learning.
Identifiability, the ability to recover the true generative model from data, crucial for understanding the hidden world. Yet in general dictionary learning, the problem is fundamentally ill-posed without additional assumptions, akin to finding a needle in a haystack. To reduce this ambiguity, most prior work imposes strong parametric constraints to limit the potential solution space. This practice is so widespread that, although dictionary learning is fundamentally nonparametric, it is almost always instantiated linear model, where observations are sparse linear combinations of latent variables (Olshausen & Field, 1997; Aharon et al., 2006; Geadah et al., 2024). However, this linearity could be overly restrictive and fails to capture the complexity of many real-world generative processes. A relevant example is sparse autoencoders (SAEs), commonly used in mechanistic interpretability, especially for foundation models. Although effective in some settings, SAEs are rooted in sparse linear dictionary learning, raising concerns about their ability to represent the inherently nonlinear structure of large-scale neural representations.
Many efforts have been made to relax the linearity assumption. In nonlinear ICA, one line of work leverages auxiliary variables as weak supervision to achieve identifiability under statistical indepen- dence (Hyvärinen & Morioka, 2016; Hyvärinen et al., 2019; Yao et al., 2021; Hälvä et al., 2021; Lachapelle et al., 2022), while another constrains the mixing function itself (Taleb & Jutten, 1999; Moran et al., 2021: Kivva et al., 2022; Zheng et al., 2022; Buchholz et al., 2022). In causal rep- resentation learning, identifiability often depends on access to interventional data (von Kügelgen al., 2023; Jiang & Aragam, 2023; Jin & Syrgkanis, 2023; Zhang et al., 2024) counterfactual

---

Published as a conference paper at ICLR 2026
views (Von Kügelgen et al., 2021; Brehmer et al., 2022), which assume some control over the data- generating process to enable meaningful manipulation.
However, gap remains between theoretical guarantees and practical utility. Theoretically, while additional assumptions can yield recovery guarantees, it is rarely possible to verify whether such assumptions hold in practice. Understanding what guarantees remain valid under assumption viola- tions is therefore essential for reliably uncovering the truth in general settings. Practically, we are less concerned with identifiability under ideal conditions and more interested in which inductive bi- ases promote recovery, especially when the ground truth is unknown. Yet most existing approaches fail to offer any guarantees under even mild violations of their assumptions, making their associated biases, such as contrastive objectives or weak supervision, difficult to generalize across settings.
Therefore, in the general scenarios, two questions remain:
What aspects of the latent process can still be recovered?
What inductive biases should be introduced to guide recovery?
To answer these questions and thus achieve actionable identifiability, we focus on a new problem aiming offer meaningful guarantees across wide range of scenarios: diverse dictionary learn- ing. Rather than seeking to recover all latent variables in the system, we consider a complementary question: what aspects of the latent process remain identifiable even in the general settings with only basic assumptions? We show that, even without specific parametric constraints or auxiliary supervision, structured subsets of latent variables can still be identified through their set-theoretic relationships with observed variables. In particular, for any set of observed variables, the intersec- tion, complement, and symmetric difference of their associated latent supports are identifiable (Thm. 1). Moreover, the dependency structure between latent and observed variables is also identifiable up to standard indeterminacy of relabeling (Thm. 2). These flexible results naturally uncover many informative perspectives of the hidden world through the lens of diversity: the intersection captures the common latent factors (genus) underlying multiple objects, while the complement and symmet- ric difference allow to isolate the parts that unique or non-overlapping (differentia), providing a principled way to understand the hidden world from the classical genus-differentia definitions (Granger, 1984) (Prop. 1) or the atomic regions in the Venn diagram (Sec. 3.2).
Since this form of identifiabiliyy is defined entirely through basic set- aet-theoretic operations, it is highly flexible and applies to arbitrary subsets of observed variables based set algebra. When the full set of observed variables is considered and the dependency structure between latent and observed vari- ables is sufficiently diverse, it becomes possible to recover all latent variables, yielding a generalized structural criterion for full identifiability (Thm. 3). Notably, for estimation, these identifiability ben- efits require only a simple sparsity regularization on the dependency structure, which can be readily implemented in most models that admit Jacobian. Our theory also makes it rather universal, sup- porting meaningful recovery across a wide range of settings, from partial to full identifiabiliyy, and thus serves robust and broadly applicable regularization principle. We incorporate this universal bias into different types of generative models and observe immediate benefits from the correspond- ing identifiability guarantee in both synthetic and real-world datasets.
2 BACKGROUND AND PROBLEM SETUP
We adopt the standard perspective of latent variable models, where the observed world is generated from latent variables through a hidden process:
X x
where X = (X...... X E Rdd denotes the observed variables, and 7 = -Zi,.... Za,) E RRd denotes the latent variables. Let X' and denote the supports of X and 7, respectively.
Connection to linear dictionary learning. Our task can be viewed as nonlinear version of clas- sical dictionary learning. Both classical approaches (Olshausen & Field, 1997; Aharon et al., 2006) and more recent ones (Hu& & Huang, 2023; Sun & Huang, 2025) model observations as linear com- binations of dictionary atoms D, i.e X = DZ. Differently, we consider the nonlinear setting X (Z), where g is nonlinear function. Although arising in different contexts, linear dictionary learning provides a useful analogy for some necessary conditions to avoid ill- ioo-posed settings. In the linear case, conditions like Restricted Isometry Property had to be introduced, which ensure that
2

---

Published as a conference paper at ICLR 2026
different latent codes map distinguishable outputs, making the linear operator injective and thus no information is lost (Foucart & Rauhut, 2013; Jung et al., 2016). By analogy. the nonlinear setting also requires restrictions on g to ensure injectivity. Following the literature on nonlinear identifia- bility, is assumed to be a C² diffeomorphism onto its image (smooth and injective) (Hyvärinen & Pajunen, 1999; Lachapelle al., 2022; Hyvärinen et al., 2024; Moran & Aragam, 2025).
Connection to nonlinear identifiability results. However, simply avoiding information loss is insufficient for full latent recovery with guarantees in the nonlinear regime. Prior work (see the survey (Hyvärinen et al., 2024)) addresses this by constraining the form of (e.g.. post-nonlinear models) or by introducing auxiliary information, such as domain or time indices, or interven- tional/counterfactuull data. In contrast, we focus on general real-world settings and deliberately avoid such assumptions, aiming to understand what can be recovered from this minimal setup. Nat- urally, some basic conditions, such as invertibility and differentiability. are necessary to rule out pathological cases, but our goal to keep these as general as possible, even with the trade-off that recovering every latent variable becomes infeasible.
Remark (Extension to noisy processes). Equation (1) considers a deterministic function, but it can be naturally extended settings with additive noise using standard deconvolution (Kivva al., 2022), or to more general noise models under additional assumptions (Hu & Schennach, 2008).
Structure. The dependency structure between latent and observed variables, though hidden, captures the fundamental relationships underlying the data and is inherently nonparametric. To explore theoretical guarantees in general settings, this structure provides a natural starting point (Moran et al., 2021; Zheng et al., 2022; Kivva et al., 2022). Before diving deep into the hidden relations, we need to formalize them from the nonparametric functions. We first define the nonzero pattern of a matuix-valued function as:
Z Z2
Z3
X2
X3 3
0
Definition 1. The support of matrix-valuee function M + - m is the set of index pairs (i,j) such that the (i,j))th entry of M()) is nonzero for some input 0 E O:
su (i..j) [m] x [m] 1 E (,M(() #
Figure 1: Example
of structure.
For constant matrix, its support a special case of Defn. 1, which is the set of indices of non-zero elements. Then, we define the dependency structure as the support of the Jacobian of g:
Definition 2. The dependency structure between latent variables Z and observed variables X = g(Z) is defined as the support of the Jacobian matrix of g. Formally,
{ (1.5) E [dz] x [d.] 32 E Z, 0g.(z) + +0}
susppp(Das) 2) =
Ōzy
This structure S captures which latent variables functionally influence which observed variables through the generative map 9. might be noteworthy that, since it is defined via the Jacobian, reflects functional rather than statistical dependencies. In particular, it does not require statistical independence of and is therefore not limited to the mixing structures typically considered in ICA.
Example 1. Figure I illustrates the dependency structure of a generative process. The top panel shows the ground-truth mapping from latent variables Z = (Z,.Zz, Za) to observed variables X (XI,, X2, X3). The bottom panel shows the support of the Jacobian D Dz9(Z) (Z), where non-zero entries are marked with "*" Notably, the Jacobian structure also captures dependencies between latent variables, such as the interaction between Zi and Zz.
THEORY
In this section, we develop the identifiability theory of diverse dictionary learning. Our theory begins with generalized notion of identifiability based on set-theoretic indeterminacy (Sec. 3.1), capturing what remains recoverable under minimal assumptions. We then illustrate its practical implications, such as disentanglement and atomic region recovery, through concrete examples (Sec. 3.2). These insights motivate the formal guarantees in Thms. and (Sec. 3.3). Finally, we show how the same framework extends naturally to element-wise identifiability under generalized structural condition (Thm. 3, Sec. 3.4). All proofs are provided in Appx. A.
3

---

Published as a conference paper at ICLR 2026
3.1 CHARACTERIZATION GENERALIZED IDENTIFIABILITY
As previously discussed, identifying all latent variables is fundamentally ill-posed without additional information, such as restricted functional classes or multiple distributions. In general scenarios where such constraints are absent, natural question arises: what aspects of the latent process remain recoverable? Before presenting our identifiability results, we first formalize this goal, which has not been addressed in the existing literature.
We begin by defining when two models are observationally indistinguishable from the perspective of the observed data, which is the goal of estimation based on observation.
Definition 3 (Observational equivalence). we say there is an observational equivalence between two models 0 (g.Pz) and 0 (..P2), denoted O mobs O, if and only if.
p(z: 0) = p(2; 0), Vx E X.
Given that estimation yields an observationally equivalent model, our goal is to determine whether the latent variables recovered by this model correspond meaningfully model. Since we avoid placing restrictive assumptions on the entire system, we adopt a localized perspective: instead of analyzing global correspondence, we examine the relationship between latent components at the level of specific observed variables. Inspired by set theory, we introduce new notion of indeterminacy that formalizes ambiguity through basic set-theoretic operations.
those in the ground-truth
Definition (Latent index set). For any set of observed variables Xs, its latent index set Is S [d2]] is defined as
Is = E [d=] Le., the sel of indices of latent variables Zig that influence Xs- Definition 5 (Set-theoretic indeterminacy). There is a set-theoretic indeterminacy between two models = (g.Pz) and = (g. P2), denoted O wea 0, if and only if. for any two sets of observed variables X k and Xv, and their latent index sets IK and Tv, there exists a permutation over [dg)
0xs oz: + #0},
such that Z₃ is not a function of 2n(s) for all (i,3) satisfying at least one of the following:
(i) (Intersection) i Ix O Iv. IRAIV;
(ii) (Symmetric difference") i E Ix J In nlv
(iii) (Complement) i E TK] \ Tv. j E Tv \ Jx, or Tv VR. Tv.
Intuitively, set-theoretic indeterminacy guarantees that certain com- ponents of the latent variables defined by basic set-theoretic opera- tions are disentangled from the rest, with an example as:
Zz,
Zte
Ix n Iv
Ix W
Example Figure 2 illustrates latent variables indexed by I and Iv, which influence the observed variable sets Xx and Xv. The intersection I n Iv contains shared latent factors, while the sym- metric difference T Aly consists of those unique one but not the other. According to set-theoretic indeterminacy, latent variables in the intersection Igy cannot be expressed as functions of those in the symmetric difference 1 Aly, ensuring that shared compo- nents remain disentangled from exclusive ones. Similarly, variables the symmetric difference Ig Ayy cannot be entangled with Ix O Iv, preserving directional separability. Finally, the com- plement condition prohibits mutual entanglement between the exclusive parts Ix Iv and Iv AI, guaranteeing that what is unique to observed group cannot explain what is unique to another.
Xv
Xx
Figure 2: Example of Defn. 5.
Because these operations form the foundation of set algebra, they can be flexibly composed to derive a variety of meaningful perspectives on the hidden variables, which we will detail later. We are now ready to define what it means for model to have generalized identifiability.
'Given the standard invertibility assumption, the reverse also holds because the block-diagonal Jacobian, which applies similarly in related definitions. "The symmetric difference Tx ATy denotes elements in Jx Ty but not in both, i.c., (Tx \ UT) JR).
4

---

Published as a conference paper at ICLR 2026
Definition 6 (Generalized identifiability). For model 9 (g. have generalized identifi- ability, if and only if. for any other model 0 (G.P2).
0 mbs θ ses 8.
3.2 IMPLICATIONS OF GENERALIZATION
In the previous section, we introduced new characterization of identifiability suited general, unconstrained settings. Built from basic set-theoretic operations, this formulation appears flexible and composable. Yet it remains unclear how general it truly is, and more importantly, why that generality matters. To answer this, we examine its implications through concrete example in Fig. 3, highlighting both its expressive power and practical utility. We begin with several interesting implications of the generalization:
Proposition 1 (Implications of generalized identifiability). For any two models 0 (g.Pz) and i (I.P2), if ~sst 0, then for any two sets of observed variables XK and Xv. and their corresponding latent index sets Ig and Iv, Ze is not a function of Zm(s) for all (i.j) satisfying at least one of the following, where nt is permutation:
(i) (Object-centricc) i E II, j E Iv ov. Av;
(ii) (Individual-centric) i E (IK (T), Iv. or m E (Iv (R). Tx;
(iii) (Shared-centric) E TK Iv, TRAlV.
Example 3. In Fig. 3, Prop. I implies that, if we consider two groups of observed variables, such as X1 and (X2. X3). regions like I (I)UIa) illustrate individual-centric disentanglement, where la- tents unique to group must be disentangled from the rest. Regions such I1 Iz. Ig represent object-centric disentanglement, where latents relevant to a single object must remain disentangled from the rest. The shared part can also be disentangled in similar manner.
Why does matter in the real world? These impli- cation types correspond to meaningful structures in real- world tasks. Object-centric disentanglement aligns with modularity in object-centric learning, where cach object should have its own latent representation. Individual- centric disentanglement supports domain adaptation by isolating domain-speciff factors. Shared-centric disen- tanglement captures common factors across domains entities, which is essential for transferability and gen- eralization. These patterns emerge naturally from set- theoretic indeterminacy and offer a principled way to de- sign models that reflect the genus- -iffeeriffee structure.
h I (7)
(1 n 4)44
40404
4 4))
4s AAGuA I3
I
Uo UOLAAA
Atomic regions in the Venn diagram. If the union of the latent index sets covers the full latent space, the general- ized identifiability guarantees in Defn. 5 extend to every atomic region in the corresponding Venn diagram.
Figure 3: Running example.
Example 4 (Identifying atomic regions). Let L1, 12, and 13 be the latent index sets associated with three observed variables in Fig. 3. Consider the atomic region (L1 J33 To disentangle from the rest, we first take Xx X1, Xy X2, so IK = In Iy I,. Then i E (I1 n I2) \ I3 S Ix O Iv, and j Ig AIv difference. Second, take Xx = X1 U X2, Xv =X3, so I =
(I1 I2) (Iz I1), ensuring Z; not function of Zz the symmetric In and Iv = Iz. Then i (T) \ Ia C Ix \ Iv and j I3. Iv. ensuring Ze is also disentangled from latents of Xa- Together, these guarantee that the alomic region (T1 T2) \ 13 is disentangled from the rest. Other cases follow similarty and are in Appx. B. Therefore, each atomic region is disentangled from all other variables, and thus is region-wise (block-wise) identifiable' under the invertibility condition.
An atomic region in the Venn diagram is empty of the form Nion Bi. where each B; E {1i, [L.][da]] In the example, these correspond to all 7 distinct regions.
T.) for finite collection of sets ( [T,-... . Jn
A model is block-wirs identifiable (Von Kügelgen et al., 2021) if the mapping between the estimated and ground-truth latent variables is composition of block-wise invertible functions and permutations. Intuitively, variables be entangled within the same block (set) but across di fifferent blacks (sets).
5

---

Published as a conference paper at ICLR 2026
Remark 2 (Connection to block-identffabiiityy.. By leveraging basic set-theoretic operations, can construct a Venn diagram over the latent supports and identify all atomic regions, each de- fined minimal, non-overlapping region closed under finite intersections and complements. This perspective is conceptually related to block-wise identifiaility (Von Kügelgen el al., 2021; ; Li et al., 2023; Yao et al., 2024b), but differs fundamentally in its assumptions and goals. Prior work achieves block identifiability by exploiting additional information such as multiple views or domains (Von Kügelgen et al., 2021; Yao et al., 2024b; Li et al., 2023). In contrast, our approach requires no such weak supervision. Notably, Yao et al. (2024b) also proposes an identifiability algebra, but in the opposite direction: they show that the intersection of latent groups can be identified after the groups themselves ure recovered using mulli-view signals. Our formulatiinn instead starts from basic assumptions without any additional information, and directly targets the identifubility of intersec- tions and complements without relying external (weak) supervision. Of course, since the goals and setups are fundamentally different, our results do not supersede existing block- boock-ddeifffaiiiity results, but rather offer complementary perspective on recovering local structures.
3.3 GENERALIZED IDENTIFIABILITY
Having established the characterization and implications of generalized identifiability, we now turn to its formal proof. Let Il denote matrix sharing the support of the matrix-valued function h in the identity g2g(2) h=2,, 8) = D ((). We begin by introducing the following assumption, which ensures sufficient nonlinearity in the system. Some arguments are omitted for brevity.
Assumption (Sufficient nonlinearity). For each E [dæ], there exists a set S, of (l(2g)) points such that the corresponding vectors for a model (2.p2):
0X;; ax, 0x;
kk S,,
02, d2," oZa. 2=2())
are linearly independent, where (b) denotes sample with index k and supp((D2g(= S supp((Dzg)..)
The assumption ensures the connection between the structure and the nonlinear function. In the asymptotic cases, we can usually find several samples in which the corresponding Jacobian vectors are linearly independent, i.e., span the support space. The assumption of non- exceeding support at these points generally rules out global cancellation of genuine dependency across all samples, which is intuitively similar to violation of faithfulness.
Interpretation.
Connection to the literature. The sufficient nonlinearity assumption makes it feasible to draw the connection between the Jacobian and the structure. It has been widely used in the literature (Lachapelle et al., 2022: Zheng et al., 2022; Kong et al., 2023; Yan et al., 2023), and aligns with the sufficient variability assumption (Hyvärinen & Morioka, 2016; Khemakhem et al., 2020; Sorrenson et al., 2020; Lachapelle et al., 2022; Zhang et al., 2024; Lachapelle et al., 2024). While most prior works often focus variability across environments, sufficient nonlinearity imposes variability in the Jacobians across multiple samples to span the support space, following the spirit in (Lachapelle al., 2022: Zheng al., 2022: Lachapelle al., 2024).
Then we are ready to present our main theorem for the generalized identifiability:
Theorem (Generalized identifiability. Consider any two models = (g. pz) and ê = (g. P2) following the process in Sec. 2. Suppose Assum. I holds and:
The probability density of Z is positive in Rd=;
ii (Sparsity regularization') 2
9 lo < Dzzllo
Then i ross 0, we have generalized identifiability (Defn. 6). i.e., 0 rest 0.
We further show that the dependency structure is identifiable up to a standard relabeling indetermi- nacy, providing structural insight when the underlying connections are of interest.
Theorem 2 (Structure identifiability. Consider any two models = (g.Pz) and θ = (g. P2) following the process in Sec. 2. Suppose assumptions in Thm. hold. If 0 ahs O, the support of the Jacobian matrix D29 is identical to that of Dz9, up to a permutation of column indices.
Notably, this is regularization during estimation, instead of an assumption restricting the data.

---

Published as a conference paper at ICLR 2026
Universal inductive bias. The first additional condition of positive density is standard and appears in nearly all previous identifiability results. We therefore focus on the sparsity regularization. Note that this is not assumption the data-geneg process itself, but practical inductive bias applied only during estimation. Thus, the ground-truth process does not need to be sparse at all.
This dependency sparsity reflects an inductive bias toward the simplicity of the hidden world. Among the many interpretations of Occam's 's razor, our approach aligns with the connectionist view, which prefers always shave away unnecessary relations. This principle is fundamental and has been extensively studied in several fields. For example, in structural causal models, fully connected graphs are always Markovian to the observed distribution, but principles such as faithfulness, fru- gality, and minimality are used to eliminate spurious or redundant edges, revealing the true causal structure (Zhang, 2013). These simplicity criteria have been validated both theoretically and empir- ically over decades, supporting the use of sparsity as reasonable inductive bias during regulariza- tion. Moreover, this regularization highly practical: it can be integrated into most differentiable models, as long gradients of the mappings with respect to latent variables are accessible.
3.4 FROM SETS TO ELEMENTS
Having established generalized identifiability through set-theoretic indeterminacy, which provides meaningful guarantees when full recovery is out of reach, natural question arises: stronger results, such as element identifiability for all latent variables, as targeted by most prior work, be obtained by imposing additional constraints?
Definition 7 (Element-wise indeterminacy). We say there is an element-wise indeterminacy be- tween two models 0 = Pz) and (g.P%), denoted f eeem 0, if and only if
= P=+((7),
where = (p Fd. (Za. )) : = 2 element-wise diffeomorphism and Pa is permutation matrix corresponding a d perrmurttttion
Definition (Element identifiability (Hyvärinen & Pajunen, 1999)). For a model 0 = (gg. we have element identifiabilty, if and only if. for any other model = (0.P2).
0 obs 0
elem 0.
Connection to generalized identifiability. Generalized identifiability (Defn. 6) focuses recover- ing partial information from subsets of observed variables, while permutation identifiability seeks to recover all latent variables up glement-wise indeterminacy, which is strictly stronger. As result, achieving permutation identifiability naturally requires stronger assumptions.
Interestingly, this can be natural consequence of set-thcoretic indeterminacy. As discussed in Sec. 3.2, generalized identifiability guarantees the recovery of atomic regions the Venn diagram. Therefore, if the Venn diagram is sufficiently rich, meaning that each latent variable corresponds to its own atomic region, we obtain element-wise identifiability directly. Since the Venn diagram is simply a representation of the dependency structure, we now formalize the corresponding structural condition as follows. For each X, E A, let I, be the index set of latent variables connected to Xj- Assumption 2 (Sufficient diversity). For each latent variable Zi (i E [dz]), there exists set of observed variables A such that at least one of the following three conditions holds:
1. There exists X E A such that Ux,ea 1 = [dz], la AxxEAM(XX) 15
2. There exists X₃ such that Ux, L [dz), (Nx,EA)(XA)) 13) \ \/4 =i.
3. (Zheng et al., 2022) The intersection of supports satisfies NxçEA 1;= =i
Theorem 3 (Element identifiability. Consider any two models following the process in Sec. 2. Suppose assumptions in Thm. and Assum. 2 hold. Then have identifiability up element-wist indeterminacy, i.e., dbs θ 0 etem Q.
(0) and θ (g. P2)
Generalized structural condition. The sufficient diversity serves as generalized condition for element identifiability in fully unsupervised settings. The most closely related work is Zheng et al. (2022), which also derives nonparametric identifiability results based purely on structural assump- tions, without relying on auxiliary variables, interventions, restrictive functional forms. However,

---

Published as a conference paper at ICLR 2026
their structural sparsity condition aligns exactly with the third clause of sufficient diversity, making strictly stronger. In contrast, our formulation introduces two additional conditions as alterna- rives, expanding the class of admissible structures and offering greater flexibility. We conjecture that sufficient diversity may even be necessary when no distributional or functional form constraints imposed, arises naturally from the structure of atomic regions in the Venn diagram. Since any dependency structure admits such a representation, and atomic regions serve as its minimal elements, our condition may capture the essential structural requirement for element-level recovery.
Diversity is not sparsity. It is worth emphasizing that our diversity condition is fundamentally different from sparsity assumptions. Diversity does not require the structure to be sparse: remains valid even in nearly fully connected settings, as long as there is some variation (e.g., even a single differing edge) the connectivity patterns across variables. By contrast, sparsity-based assumptions strictly enforce sparse structures. For example, the well-known anchor feature assumption (Arora al., 2012; Moran et al., 2021) requires each latent variable to have least two observed variables that are unique to it, thereby excluding dense structures.
EXPERIMENT
In this section, we provide empirical support for results in both synthetic and real-world settings.
Due to page limits, additional experimental results are deferred to Appendix C, including (1) comparisons of more Jacobian/Hessian penalties (e.g., (Wei al., 2021; Peebles al., 2020)), (2)
analyses of regularization weights, and (3) further visual results on synthetic and real data.
4.1 SYNTHETIC EXPERIMENTS
Setup. We follow the data generation process in Sec. 2. We employ the variational autoencoder as our backbone model with dependency sparsity regularization in the objective function as:
1.00
Symde
0.75
Comp
Comp
Hkd
0.50
C. = (n(2)x)nnp(XX) 7
(D (d(Z)X))ll(Z)) +o Dagllo 0.25
Evidence Lower Bound
0.00
where DKL is the Kullback-Leibler divergence, q( x) the variational posterior, p(2) the prior, p(X (2) the like-
Number of variables
Figure 4: R in simulation.
lihood, and a, 3 regularization weights. We use 10, 000
samples and set implemented by MLPs with Leaky ReLU.
3 0.05 for all experiments, and all generation processes nonlinear,
Generalized Identifiability. We begin by evaluating gen- eralized identifiability across groups of observed vari- ables. We generate datasets with dimensionality in (3, 4) and split the observed variables into two groups, Xx and Xv. For each dataset, compute the R² score, lower means more disentangled, between: (1) Int, Inn and 1 Alv; (2) SymDiff. IK AIv and IK Iy: and (3) Comp A and Comp B. both directions between 1x A1 and Tv Tg. We also include Ref, the R² between Z and 2, as a baseline indicating the expected level of R² for entangled variables. As shown in Fig. 4, all disentan- glement conditions implied by set-theoretic indeterminacy (Defn. 5) are satisfied: the Rº between structurally disjoint components is consistently much lower than Ref. supporting the validity of gen- eralized identifiability.
1.0
0.8
0.6
0.4
0.2
Base
0.0
6 Number of variables
9
Figure MCC in simulation.
Element Identifiability. We evaluate whether comparing multiple variable pairs enables recovery of latent variables up to element-wise indeterminacy. We construct datasets with varying dimen- sions and structures that either satisfy Sufficient Diversity (Assum. 2) (Ours) violate through fully dense dependencies (Base). Following prior work (Hyvärinen et al., 2024), we use the mean correlation coefficient (MCC) between estimated and ground-truth latent variables as the evaluation metric. As shown in Fig. 5, only datasets satisfying the structural condition achieve high MCC, confirming that element- wise identifiability holds under our assumptions.
8

---

Published as a conference paper at ICLR 2026
Method
Shapes3D
Cars3D
MPI3D
FactorVAE
DCI 1
FactorVAE
DCTT
FactorVAE
DCI
VAE-based
FactorVA (Kim Mrih, 2018)
0,833
0.025
0.484
0.120
0.708
+ 0.026
0.135
+ 0,030
0.599
0.064
0.345
0.047
FactorVAE + Latent Sparsity
0,837
0.069
0.477 0.152
0.501
0,434
0.113
1 0,069
0.440
0.065
0.325
0.028
FactorVAE Dependency Sparsity
0.871
0.053
0.575
0.032
0.752
0.040
0.144
0.053
0.639
0.084
0.384
0.031
Diffusion-bared
EncDilf (Yang et al., 2024) EncDiff + Latent Sparsity
0.9999 1 0.0001
0.901
1 0.050
0.779 1 0.060
0.250
0.020
0.868
0.033
0.676
0.018
0.967
0.042
0.891
+ 0.057
0.729 - 0.008
0.241
+ 0.016
0,79
0.015
0.684
0.020
EncDiff + Dependency Sparsity
1,0000
0.0000
0.947
+ 0.005
0.756 + 0.041
0.256
0.911
0.881
0.024
0.667
0.047
GAN-buzed
DisCo (Ren et al, 2021) DisCo Latent Spansity
0,852
0.037
0.710
0.727 106
0.319
0.031
0.396
0.306
0.079
0.007
0.707
0.761
0.308
0.031
0.050
DisCo Dependency Spansity
0.868
0.017
0.712
0.018
0.789
0.029
0.320 0.003
0.410
0.122
0.059
Table I: Comparison of disentanglement on FactorVAE score and DCI (mean=ssd, higher is better). Bold numbers denote the best value within each model family for a given dataset/metric.
4.2 VISUAL EXPERIMENTS
Setup. Following the literature, we evaluate identification in more complex settings by learning latent variables as generative factors. Specifically, we follow the setting of (Yang et al., 2024) and use three standard benchmark datasets of disentangled representation learning: Cars3D (Reed al., 2015), Shapes3D (Kim & Mnih, 2018). and MP[3D (Gondal et al., 2019), which are benchmark datasets with known generative factors such as object color, shape, scale, orientation, and viewpoint, ranging from synthetic renderings to real-world images.
To evaluate the effectiveness of the proposed sparsity loss, we incorporate it into three powerful disentangled representation learning methods based on mainstream generative models: Variational Autoencoders (VAE), Generative Adversarial Networks (GAN), and Diffusion Models. These meth- ods correspond to FactorVAE (Kim & Mnih, 2018), DisCo (Ren et al., 2021). and EncDiff (Yang et al., 2024), respectively. We consider two types of baselines: 1) the original methods, i.c., Factor- VAE, DisCo, and EncDiff, and 2) versions of these methods that incorporate LI regularization on Z (latent sparsity). In contrast, our approach applies an L1 regularization on Jacobian (dependency sparsity). Following standard practice, we use FactorVAE score (Kim & Mnih, 2018) and the DCI Disentanglement score (Eastwood & Williams, 2018) as evaluation metrics, We repeat each method over three random seeds. Please refer Appx. C for more details on setups.
Dependency sparsity in the literature. Notably, dependency sparsity has been widely used as a simple and standard regularization across diverse settings, from disentanglement LLMs (Rhodes & Lee, 2021; Zheng et al., 2022: Farnik et al., 2025), although a general identifiability theory is still lacking. Thus, its empirical effectiveness is already well established, and our experiments aim to provide further supporting evidence.
Latent or dependency sparsity? Table shows that across most datasets and backbone methods, introducing the proposed dependency sparsity consistently helps the understanding of the hidden world. Notably, these generative models often benefit more from dependency sparsity than from latent sparsity. This is particularly interesting given the widespread use of sparse latent regular- ization in mechanistic interpretability (c.g.. sparse autoencoders (Cunningham et al., 2023)). Our results highlight not only the advantage of dependency sparsity, but also lend insight to recent con- cerns about the limitations of latent sparsity raised in the interpretability literature, such as feature absorption, linear constraints, and high dimensionality (Sharkey et al., 2025).
5 CONCLUSION
We introduce diverse dictionary learning to investigate which aspects of the hidden world can be recovered under basic conditions, and which inductive biases may be universally beneficial during estimation. Our guarantees, grounded in set algebra, offer complementary local view to prior re- sults based on global assumptions, and also unify existing structural conditions for full identifiability. For future work, it is worth exploring generalized identifiability in foundation models. Current mod- els are largely driven by empirical insights, and inductive biases inspired by identifiability, which have been overlooked, may offer fresh directions for breakthroughs. With massive data and com- putation available, asymptotic guarantees are becoming increasingly relevant, making identifiability practically significant. A deeper investigation along this line remains open limitation of our work.
9

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers, AC, Lucas Jr, and Alan Amin for helpful comments and suggestions. The authors would also like acknowledge the support from NSF Award No. 2229881, AI Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R01HL159805, and grants from Quris AI, Florin Court Capital, MBZUAI-WIS Joint Program, and the Al Deira Causal Education project.
REFERENCES
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing over- complete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11); 4311-4322, 2006.
Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topie models-going beyond svd. 2012 IEEE 53rd annual symposium on foundations of computer science, pp. 1-10. IEEE, 2012.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal repre- sentation learning. Advances in Neural Information Processing Systems, 35:38319-38333 2022.
Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable non-
linear independent component analysis. arXiv preprint arXiv:2208.06400 2022.
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning.
Guangyi Chen, Yifan Shen, Zhenhao Chen, Xiangchen Song, Yuewen Sun, Weiran Yao, Xiao Liu, and Kun Zhang. Caring: Learning temporal causal representation under non-invertible generation process. In International Conference on Machine Learning, pp. 7236-7259. PMLR, 2024.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0860 2023.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation disen- tangled representations. In 6th International Conference on Learning Representations, 2018.
Lucy Farnik, Tim Lawson, Conor Houghton, and Laurence Aitchison. Jacobian sparse autoencoders: just Sparsify computations, not just activations. In Forty-second International Conference on Machine Learning. 2025.
Simon Foucart and Holger Rauhut. Restricted isometry property. In Mathematical Introduction to Compressive Sensing, pp. 133-174. Springer, 2013.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Charles J Garfinkle and Christopher Hillar. On the uniqueness and stability of dictionaries for sparse representation of noisy signals. IEEE Transactions Signal Processing, 67(2))55884- 5892, 2019.
Victor Geadah, Gabriel Barello, Daniel Greenidge, Adam S Charles, and Jonathan W Pillow. Sparse- coding variational autoencoders. Neural computation, 36(12):2511-2611 2024.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
Edgar Herbert Granger. Aristotle on genus and differentia. Journal of the History of Philosophy, 22 (1):1-23, 1984. doi: 10. 1353/hph. 1981.0001.
10

---

Published as a conference paper at ICLR 2026
Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvärinen. Disentangling identifiable features from noisy data with structured nonlinear ICA. Advances in Neural Information Processing Systems, 34, 2021.
Jingzhou Hu and Kejun Huang. Global identifiability of E--based dictionary learning via matrix volume optimization. Advances in Neural Information Processing Systems, 36:36165-36 2023.
Yingyao Hu and Susanne M Schennach. Instrumental variable treatment of nonelassical measure- ment error models. Econometrica, 76(1):195-25 216, 2008.
Aapo Hyvärinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. Advances in Neural Information Processing Systems, 29:3765-3773, 2016.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-439, 1999.
Aapo Hyvärinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of vatent-variable and dinuttral-equatinn models: from linear to nonlinear. Annals of the Institute of Statistical Mathe- matics, 76(1): 2024.
Yibo Jiang and Bryon Aragam. Learning nonparametric latent causal graphs with unknown inter- ventions. Advances in Neural Information Processing Systems,
2023.
Jikai Jin and Vasilis Syrgkanis. Learning causal representations from general environments: Identi- fiability and intrinsic ambiguity, arXiv preprint arXiv:2311.12267 2023.
Alexander Jung, Yonina C Eldar, and Norbert Görtz. On the minimax risk of dictionary learning. IEEE Transactions on Information Theory, 1501-1555, 2016.
llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvärinen. Variational autoen- coders and nonlinear ICA: A unifying framework. In International Conference Arlificial In- telligence and Statistics, pp. 2207-2217 PMLR, 2020.
Hyunjik Kim and Andriy Mnib. Disentangling by factorising. In International conference on ma- chine learning, pp. 2649-2658. PMLR, 2018
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687-15701. 2022.
Lingjing Kong. Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akin- wande, and Kun Zhang. Partial identifiability for domain adaptation. In International Conference on Machine Learning, pp. 11455-11472. PMLR, 2022.
Lingjing Kong, Biwei Huang, Feng Xic, Eric Xing, Yuejie Chi, and Kun Zhang. Identification of nonlinear latent hierarchical models. Advances in Neural Information Processing Systems, 36: 2010-2032, 2023.
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentangiement via mechanism sparsity regularization: A new principle for nonlinear ICA. Conference on Causal Learning and Reasoning, 2022.
Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive de- coders for latent variables identification and cartesian- procucct extrapolation. Advances Neural Information Processing Systems, 36, 2024.
Zijian Li, Ruichu Cai, Guangyi Chen, Boyang Sun, Zhifeng Hao, and Kun Zhang. Subspace identi- fication for multi-source domain adaptation. Advances in Neural Information Processing Systems, 36:34504-3455 34518, 2023.
11

---

Published as a conference paper at ICLR 2026
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence, pp. 1273-1273. PMLR, 2023.
Francesco Locatello. Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 14244... PMLR, 2019.
Gemma E. Moran and Bryon Aragam. Towards interpretable deep generative models via causal representation learning. arXiv preprint arXiv:2504. /609, 2025.
Gemma Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. Identifiable variational autoen- coders via sparse decoding. arXiv preprint arXiv:2/ 10. 10804, 2021.
Hiroshi Morioka and Aapo Hyvarinen. Causal representation learning made identifiable by grouping of observational variables. In International Conference Machine Learning, pp. 36249-36293. PMLR, 2024.
Bruno A Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision research, (1123):3311-3325 1997.
William Peebles, John Peebles, Jun- Yan Zhu, Alexei Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In European conference on computer vision, pp. 581-597. Springer, 2020.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lec. Deep visual analogy-making. Advances in neural information processing systems, 28, 2015.
Olav Reiersøl. Identifiability of a linear relation between variables which are subject to error. Econo- metrica: Journal of the Econometric Society, pp. 375-389, 1950.
Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Learning disentangled representa- tion by exploiting pretrained generative models: contrastive learning view. arXiv preprint arXiv:2102.1054. 2021.
Travers Rhodes and Daniel Lee. Local disentanglement in variational auto-encoders using jacobian 4 regularization. Advances in Neural Information Processing Systems, 34:22708-223 22719, 2021.
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill. Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint MiXi::5501.16400 2025.
Peter Sorrenson, Carsten Rother, and Ulirich Köthe. Disentanglement by nonlinear ICA with general incompressible-flow networks (GIN). arXiv preprint urXiv:2001.04877 2020.
Yuchen Sun and Kejun Huang. Global identifiability of overcomplete dictionary learning via I1 and volume minimization. The Thirteenth International Conference on Learning Representations, 2025.
Yuewen Sun, Lingjing Kong, Guangyi Chen, Loka Li, Gongxu Luo, Zijian Li, Yixuan Zhang, Yujia Zheng, Mengyue Yang, Petar Stojanov, et al. Causal representation learning from multimodal biomedical observations. In The Thirleenth International Conference on Learning Representa- tions, 2024.
Anisse Taleb and Christian Jutten. Source separation in post-nonlines mixtures. IEEE Transactions signal Processing, (7(10):2877-282) 1999.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:1645: 16451-16467, 2021.
12

---

Published as a conference paper at ICLR 2026
Julius von Kügelgen, Michel Besserve, Liang Wendong. Luigi Gresele, Armin Kekić, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal represen- tations from unknown interventions. Advances in Neural Information Processing Systems, 36: 48603-48638, 2023.
Yuxiang Wei, Yupeng Shi, Xiao Liu, Zhilong Ji, Yuan Gao, Zhongqin Wu, and Wangmeng Zuo. Orthogonal jacobian regularization for unsupervised disentanglement in image generation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6721-6730, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 1.Xiv:170..0700 2017.
Shaoan Xie, Lingjing Kong, Mingming Gong, and Kun Zhang. Multi-domain image generation and translation with identifiability guarantees. In The Eleventh international conference on learning representations, 2023.
Shaoan Xie, Lingjing Kong, Yujia Zheng, Zeyu Tang, Eric P Xing, Guangyi Chen, and Kun Zhang. Learning vision and language concepts for controllable image generation. In Forty-second Inter- national Conference on Machine Learning, 2025a.
Shaoan Xie, Lingjing Lingjing, Yujia Zheng, Yu Yao, Zeyu Tang, Eric P Xing, Guangyi Chen, and Kun Zhang. Smartclip: Modular vision-language alignment with identification guarantees. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19780-29790) 2025b.
Hanqi Yan, Lingjing Kong, Lin Gui, Yuejie Chi, Eric Xing, Yulan Hc, and Kun Zhang. Counter- factual generation with identifiability guarantees. Advances in Neural Information Processing Systems, 36:56256-5666 56277,, 2023.
Tao Yang, Cuiling Lan, Yan Lu, and Nanning Zheng. Diffusion model with cross attention as in- ductive bias for disentanglement. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Dingling Yao, Caroline Muller, and Francesco Locatello. Marrying causal representation learning with dynamical systems for science. Advances in Neural Information Processing Systems, 37: 71705-71736, 2024a.
Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. International Conference on Learning Representations, 2024b.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2110.05422. 2021.
Jiji Zhang. comparison of three occam's razors for markovian causal models. The British journal for the philosophy of science, 2013.
Kun Zhang, Shaoan Xie, Ignavier Ng, and Yujia Zheng. Causal representation learning from multi- ple distributions: general setting. In Forty-first International Conference on Machine Learning, 2024.
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022.
Yujia Zheng, Yang Liu, Jiaxiong Yao, Yingyao Hu, and Kun Zhang. Nonparametric factor analysis and beyond. In International Conference on Artificial Intelligence and Statisties, pp. 424-432. PMLR, 2025.
13

---

Published as a conference paper at ICLR 2026
Diverse Dictionary Learning Supplementary Material
Table of Contents
A Proofs
15
A.1 Proof of Proposition 1
15
A.2 Proof of Theorem
16
A.3 Proof of Theorem
19
Proof Theorem
21
B
Additional Discussion
25
B.1 The Venn diagram....
................................
25
B.2
Further discussion on the connection of conditions.
................................
26
B.3
Further discussion on the connection with SAEs.
26
B.4
Further discussion on sufficient nonlienarity Dependency sparsity regularization on large models
27
B.5
me..............................
27
General noise and invertibiliity
28
B.7 Further discussion potential impact.
28
C Additional Experiments
30
C.1 Additional synthetic experiments
30
C.2 Additional visual experiments
31
Disclosure Statement
14

---

Published as a conference paper at ICLR 2026
Symbol
Description
X (X1, ..... Xdx) IR
Observed variables (data space)
Z (Zz,... Za,) Rda
Latent variables (hidden space)
0: R R
Generative map, diffeomorphism onto its image Support of a matrix-valle valued function M θ - mg
supp(M;;) 0)
5 = supp Da9; 2)
Dependency structure: support of Jacobian between Z and X
0 = (g. pz)
Model consisting generative map and latent distribution
0 obs 0
Observational equivalence (same induced distribution on X)
0 Meet 0
Sel-theoretic indeterminacy (intersection, symmetric difference, com- plement disentangled)
Is S [dz)
Latent index set associated with observed variable set X
1s Olv
Intersection of latent supports (shared factors)
Ig IgAlv
Symmetric difference of latent supports (unique factors)
Ix lv, Iv \ IK
Complements (exclusive latent components)
Atomic region
Minimal block in Venn diagram defined by intersections and comple- ments of latent supports
8 elem
Element-wise indeterminacy (permutation invertible reparametriza- tion)
Table 2: Notation used throughout the paper.
A PROOFS
A.1 PROOF OF PROPOSITION
Proposition 1 (Implications of generalized identifiability). For any two models 0 = (g.p2) and i (2.22). if 0 ~st Ó, then for any two sets of observed variables X and Xv, and their corresponding latent index sets Ig and Iv, Zi is a function of Zmii) for all (i.5) satisfying at least one of the following, where πI is permutation:
(i) (Object-centric) i E Ig. E ly \ lv. Av;
(i) (Individual-centric) i (IK) \ [v),, Iv, or (Iv \ (R). Ik:
(iii) (Shared-centric) E Ig O Tv, j Tx AIv.
Proof. For 0 = (g.P2) and = (g.P2). since 0, for any two sets of observed variables XK and Xv. and their corresponding latent index sets IK and Iv. there exists a permutation over f1....,d, d such that Z; is not function of Zx for any (i.j) satisfying at least one of the following conditions:
(i) (Intersection) i IK Iv.j IRAIv
(ii) (Symmetric difference) i Ar E Ik Ily;
iiii)
(Complement) i E 1K \ [v.j ly Ik. ori ly \ Ak.j IK (v.
Our goal is to prove that, the same holds for all (i,j) satisfying at least one of the following condi- tions:
(i) (Object-centric disentanglement) i E IR.j E Iy \ In orie I Iv;
(ii) (Inividuallcentric disentanglement) i E TK \: Tv.j e ov, Ty \ VR, E Tx:
(iii (Shared-centric disentanglement) TK Ty. AgAly.
15

---

Published as a conference paper at ICLR 2026
Let us start with the first case. If i E IK, it is either the case E IK ly or Ix \ Iv. For i Ix O Iv, according to the case Intersection in set-theoretic indeterminacy, we have
a2. 0,
a2.(j)
for any j E Ik Alv. Similarly, for IK \ lv. we also have Eq. (2) for j Iv VL.. Combining these together, Eq. (2) must holds for any i E Ix and j Tv Ix. The similar derivation holds for any E Iv and JK Iv. Thus, the first case holds.
Then we consider the second case. If i E IK Iv. then according to the case of symmetric difference set-theoretic indeterminacy, we have Eq. (2) holds for TK O Iv.
Moreover, according to the case of complement in set-theoretic indeterminacy. we have Eq. (2) holds for E Iv Ix. Note that there
(Iv
Thus, for any E IK! Iv. have Eq. (2) holds for any Iv. The similar derivation holds for any E Iv \ Ix and E Ix- Thus, the second case holds.
The third case is identical to the case of intersection in set-theoretic indeterminacy. Thus, for 0 (g. (2) and 0 P2), -ee i implies our goals.
A.2 PROOF THEOREM 1
Theorem (Generalized identifiability). Consider any two models 0 = (g. pz) and = (g. Pz) following the process in Sec. 2. Suppose Assum. holds and:
The probability density of Z is positive in Rd::
ii (Sparsity regularization') [D 2 S DDzzllo-
Then 0 ross we have generalized identifiability (Defn. 6), i.e, 0 set Ô.
Proof. Since 0 ross 0, by the change-of-variable formula there must be
2 = o o = =6(()) ()
(4)
where Φ g-1 1 g an invertible function and thus Φ- exists. Therefore, according the chain rule, we have
Dę = D
(5)
For each i [dæ], consider set S; of [[(Dz)),, llo distinct points and the corresponding Jacobians as follows
0x, 0x, 0x;
k E S.. According to Assumption all vectors in Eq. (6) are linearly independent.
(6)
07, 07,"
87d, (+)=(=(1)
Let us construct a matrix Mg. Since all vectors in Eq. (6) are linearly independent, for any j E supp((D29)..). have
MØs.- Σ
RES:
where Se, Vk St denote coefficients, and MA denotes matrix.
wish to construct a constant matrix Ms satisfying
Σ 3u (Dzg(z Ms span(e, ;j supp((D29):.),
(8)
SES.
for each [dz], while ensuring that
supp MM) supp(D2Φ01 D220~1,
(9)
'Notably, this is regularization during estimation, instead of assumption restricting the data.
16

---

Published as a conference paper at ICLR 2026
According to Assumption 1, we have
Mupp(22g(3())M), _ {upppD220((4))) Ak St.
(10)
Therefore, there must be
Dzs(z()))) Ma E span(ey j supp((D2))).))
(11)
which implies
Σ 3u (Dzg(= Ma E spantes = E supp((D2g))
(12)
KES.
Equivalently, we have
Moj. E spanfex k supp((D22),.)): supp((Dzg)..)
(13)
Define bipartite graph G (R, E) where = C (1, ..... d=) and edge exists between j E R and h E C if and only if D2 40.
1 invertible, its rows are linearly independent, so for every subset S R, the corre- sponding rows have nonzero determinant, implying that
Since 2
1 E C 1 S D2 + 0} 2 (S].
(14)
By Hall's marriage theorem, there exists a perfect matching between and C. This matching corresponds to a permutation E S. such that
(D2Ø ()0())))))))))))))) #0, (1,2,..., nn. In particular, for every j supp((Dzg).) S (1. n). we have
(15)
(D2Φ );x },x(,) + O.
(16)
Because supp(Mp) = supp( (D 2 - 1), this implies
M Maja(() # O, Vj E supp((Dzg):.)
(17)
Further incorporating Eq. (13), it follows that
(s) span k E supp((D 2)),, )}. Vj e supp((D2g)..)
(18)
Therefore, for any non-zero element Dzg, there always exists a corresponding non-zero element in Dag. with the relations their indices as follows
(D.g)., # (Da))i, # 0.
(19)
Furthermore, because of the assumption that
D s DDzgllo:
(20)
Eq. (19) can be further restricted an equivalence between the sparsity patterns, i.c.,
(Dag).. # - (D=))i, #
(21)
We then consider the following two cases for the set-theoretic indeterminacy. Specifically, for any two sets of observed variables Xxx and Xy and the index sets of their latent variables, Ix and Iv. K + VV, we consider the following cases:
(i) (Intersection) i E Ix n Iv.j E AgAV;
(ii) (Symmetric difference) i Ax IR
(iii) (Complement) i E TK \ [v. E Ty \ Jx, ori Ty \ NR.j Ix \ lv.
17

---

Published as a conference paper at ICLR 2026
Let us start from the first case, where t IK O Iv, IgAIv. Denote the index sets of Xx and Xy as JK and Jy. Then, there exists E JR such that
e supp(Dzg)a
(22)
This further implies the following relation based on Eq. (13)
Mòr. E spanfex : k' E supp((())))) (2))(())))..)))
(23)
Similarly, there exists e E Jv such that
t E supp(D29)..>
(24)
which further implies
Mar, E spanfee k supp((D2))..)
(25)
For r E AxAly, suppose
M)r,(() * 0.
(26)
According to Eqs. (23) and (25), there must be
(r)) E supp D p2)),.. k.
(27)
(rr) supp D D22),..
(28)
Together with Eq. (21), these further imply
supp(Dzg)
(29)
supp(Dz9)...
(30)
This leads to
E 1K O lv, which contradict E LK Aly. Therefore, there must be
(31)
M 0.
(32)
Since My is the support of D this implies that, for t E Ix n Iv and r E IgAlv, we have
az. = 0.
(33)
02m(())
Then we consider the case where E Tk O Iv and r E I \ (IK U Tv). Suppose
M #
(34)
According to Eq. (23), there must be
(r) E supp(D2ái
(35)
Together with Eq. (21), these further imply
supp(
(36)
This leads
E lk: which contradict E 7 T (TK U Tv). Therefore, there must be
(37)
M = 0,
(38)
where t IN n Iy and E I ([KKKU
Therefore, according to Eqs. (33) and (38) and the invertibility of Φ. for E In n Iv, Z, has to depend only on (x(e) and not other variables. Therefore, there exists an invertible function s.t. 7, h(2(1))-
18

---

Published as a conference paper at ICLR 2026
Further consider the setting where IK Aly. Since t IK Iv and (Ix nlv) 0, Z, is independent of Ze (x))) Therefore, Z, does not depend (x) and thus
a2, 0, 02x(1))
(39)
which is the second case.
Then we consider the third case where t E Ix \ ly and r E Iv \ Ig. Iff E 1 \ Iv, there exists
k E JK such that
t E supp((2z))..
(40)
Then there is
Mrr. spanfew k E suppl(( (D))),
(41)
For E Iv \ IR. suppose
Moix(r) + 0. T(r) E supp(D22)
(42)
Then have
(43)
which follows
supp(Dz9)e.
(44)
This is a contradiction since r E Iv \ 1k. Thus, we can also prove that, for the third case, where t ly \ Lx and r E IK Lv, there must be
oze
(45)
07
This concludes the proof.
A.3 PROOF OF THEOREM 2
Theorem 2 (Structure identifiability). Consider any two models = (g.PZ) and Ô = (g. pz) following the process in Sec. 2. Suppose assumptions in Thm. hold. If abs O, the support the Jacobian matrix Dzg is identical to that of D2g. up to a permutation of column indices.
Proof.
Since 0 ros 0, by considering @ =g-1, g and the change-of-variable formula, we have
(())
(46)
where o is an invertible function and thus o exists. Therefore, according the chain rule, have
Dº = DzgDzΦ For each i E [dz], consider a set S; of [[(D2g,, llo distinct points and the corresponding Jacobians as follows
(47)
ax. ax; ax, Oz
& E S..
(48)
az,
aZa
(=)=(=(1)
According to Assumption 1, all vectors in Eq. (48) are linearly independent.
Let us construct a matrix Mg. j E supp((D29)..). we have
Since all vectors Eq. (48) are linearly independent, for any
MØs.-
Σ Bc(D2g(2")))
(49)
RES.
where Be, vk S, denote coefficients.
We wish to construct a constant matrix Ms satisfying
Σ
3x (Dzg(z(())) Ms span(e, :j supp((D20):)),
(50)
k.S.
19

---

Published as a conference paper at ICLR 2026
for each [dz], while ensuring that
supp(Mp) = supp(D2ø0'),
(51)
According to Assumption 1, we have Therefore, there must be
()pp) S (supp()2(((())), Vk S..
(52)
D2g(z!8))) Ms E spanfey i E supp((D2g)..)h
(53)
which implies
Σ
By (Dzg(z Ma E spanfez i supp((D2)):.)}-
(54)
RES,
Equivalently, we have
Møj. E span(ex & supp((D20)..)).v supp(())))..)
(55)
Define bipartite graph G (R, C, E) where R= C (1. d2) and an edge exists between j E R and C if and only 220 #0.
Since Dz is invertible, its rows are linearly independent, so for every subset S C R, the corre- sponding rows have nonzero determinant, implying that
1( | E S, D2 0)} 2 (S|.
(56)
By Hall's marriage theorem, there exists a perfect matching between R and C. This matching corresponds to a permutation и E Sn such that
) #0, (1,2,...,n,) n}. In particular, for every j supp((Dzz).; S. 21.
(57)
.......... n). we have
# O.
(58)
Because supp(Mp) = supp (D 2 ) this implies
Maj,()) + 0, Vj E supp((D29)..).
(59)
Further incorporating Eq. (55), it follows that
(5) spanfe: & E (Vpp((D20)..)) Vj e supp((D29)..).
(60)
Therefore, for any non-zero element in Dzg, there always exists a corresponding non-zero element in Dag. with the relations their indices as follows
(Dag).. +0 0 (D=9)i,z)) # 0.
(61)
Furthermore, because of the assumption that
D2 I [Dzzllo:
(62)
Eq. (61) can be further restricted to an equivalence between the sparsity patterns, i.e.,
(Dag)i. +0 + (Dağ)a.z)) # O.
(63)
Therefore, there must be
supp(D..g) supp((D:))P),
(64)
where P denotes a permutation matrix. Thus, the support of the Jacobian matrix Dzg is identical to that of Dig, up to a permutation of column indices.
20

---

Published as a conference paper at ICLR 2026
A.4 PROOF OF THEOREM 3
Theorem 3 (Element identifiability) Consider any two models - (z) and = ( pz) following the process in Sec. 2. Suppose assumptions in Thm. and Assum 2 hold. Then have identifiability up to element-wise indeterminacy, i.e., mos 0 = eeem O.
Proof. Since all assumptions in Thm. are satisfied, for these two models 0 (g. pz) and 0 = P2) following the process in Sec. 2, can follow the same steps in Sec. A.2 to derive Eq. (21), i.e.,
(D=g)i, =0 (D= # 0.
(65)
Then, for any latent varible Zi E Z, let us consider all conditions in Assum. 2. We begin with the first condition: there exists set of observed variables A and an element X A such that
U X₁EΛ
1= [d,), and Is
U X,EA((X))
5=(0)
(66)
Our want to show that, for any other + i, we have
aZ, 02,(()
0.
(67)
We consider cases:
(UX,EAX)) 1 TE:
(Ux,EAX))))))) 4
Suppose E (U,,EM(X.)) 4)) Lp. Let us denote JAYa as the index set of A \ {Xa}. Since
Ie \ UXGEAXIX) {i}, for any E JAYE, there must be
i & supp(D=g))...
(68)
We then suppose for contradiction that
Møc. E spanfer I supp((D20)...)
(69)
In the proof of Theorem 1, we have proved that
Mai # 0.
(70)
Then we have According to Eq, (65), this implies
T(i)= (upp(D2g),..
(71)
i E (zg)(D2g)...
(72)
This contradicts
$ supp(D2g)e..
(73)
Thus, there must be
Mas.- & span E supp((D20)..))
(74)
We further suppose by contradiction that
M M # 0,
(75)
for (Ux,ea)xx) (3) \ J.. Then, according Eq. (74), there must be
T(r) & supp(Dz9))...
(76)
which implies
supp(Dzg), D229).. This is, again, contradiction E (Ux I5) Ig. As result, there must be M 0.
(77)
(78)
21

---

Published as a conference paper at ICLR 2026
We then consider the other case, where assume (UX,EA)1X+) ( Ig. Then there exists q E JAY& S.t.
i supp(Dzg)u.
(79)
which further implies
Møi. E spanfeg d supp((D20)g.
(80)
Since we also have
4 supp(D29)g
(81)
We suppose for contradiction that
Møi. E span :1 supp((D20)4.))
(82)
Since there is follows that
M Moia(() # O.
(83)
T(1) Supp(D29)g.
(84)
According to Eq. (65), it implies
i E Supp((Dzg),
(85)
This contradicts the case that & supp(Dzg) and thus there must be
Ma & spanfe: : I E supp((D22)g.))
(86)
For E (Ux,EAR(X.) 4 n Is. suppose
Mas,((r) + 0.
(87)
Given Eqs. (80) and (86), we have
(() E
(88)
(r) $ supp D g).)
(89)
Because of Eq. (65), these further imply
r E supp(Dzg).. r (uppppzzg))....
(90)
(91)
This leads to
re \
U I0
(92)
X)EA(1X+
which contradicts
rE
SUUU X)EA((X()
U
11 n
(93)
Therefore, there must be
Mai,(()) =
(94)
Since Ma is the support of D2 this implies that, for any other r + i, we have
az. 0.
(95)
02m(1)
it Because ó is invertible, each row of D2 1 must at least have one non-zero element. Therefore,
follows that
07, 0. a²x()
(96)
Thus, with the first condition in Assum. 2, we have identifiability up element-wise indeterminacy.
22

---

Published as a conference paper at ICLR 2026
Next, consider the second condition in Assum. 2. By applying the case of Intersection in Defn. 5 for all pairs of observed variables in X, there is
07 Nz SEA 0,
(97)
do 2 (x,x
where a denotes the transformation for the permutation pi. Then for Ux, EAV(X)) Ly and Ix. by the individual-centrr disentanglement in Prop. 1, there is
o0 (UX 4JVe 0.
(98)
(
Note that
(Znx,arisa)))) n 2 (UKSEA 1) 2 (NxgEAvENNN by Considering both Eqs. (97) and (99), we have
(99)
oz (NKJEANIN BD)
(100)
do (2))
Considering both Eqs. (98) and (99), we have
oz
(Nxxsaxixe) 11
(101)
(Z)
Note that
o ( (x)AxxX)) (0(
(102)
((
\|d=|\
1,
(103)
(x,EA)XX))
=[d=] \ i.
(104)
Further given the invertibility of d. each row of D20 must at least have one non-zero element. Therefore, it follows that
07
# 0.
(105)
a2x(0)
Lastly, consider the third condition in Assum. 2. That part of proof directly follows from
(Lachapelle et al., 2022; Zheng et al., 2022). Suppose for each row in Ma. there are more than one non-zero element. Then
3ý1 + f2, Ma n Ma #0.
(106)
Then consider ju [d.] such that
T (js) E M Ma
(107)
Since ji # js. it is either js # ji or ja + ja. Without loss of generality, we assume ja # ji.
Since we have
n 1-
(108)
there must exists X1, E AG, such that ja # It. Because ji E Igg, we have
(ts.1) E supp(Dzg),
(109)
which further implies
Man Espan(e' k' E supp((D) (Dz)))..)))
(110)
23

---

Published as a conference paper at ICLR 2026
Given Eq. (107), it implies
((is) E supp(D20)ss
(111)
This, again, implies
js E Supp(Dzg)is)
(112)
which contradicts js + l/s. Therefore, for each row in Mg. there are no more than one non-zero element. Because Ms is invertible. each row must at least have one non-zero element. Thus, there must be exactly one non-zero element each row, which is
07,
+ 0. 0Zz(1)
(113)
Thus, have proved goal with all three conditions.
24

---

Published as a conference paper at ICLR 2026
I
(1(UI)
(1, (1 4
40404
4 (440)
Ia AANGUA
I
(42 (4) \4
I3
Figure 6: The Venn diagram example (Fig. 3).
B ADDITIONAL DISCUSSION
B. THE VENN DIAGRAM.
Here we provide the full deriviation of the Venn diagram example.
Example (Identifying all atomic regions). Let J, Iz, and Ta be the latent index sets of X1, X2. and X3 in Fig. 3. For each atamic region A we pick that every i E A satisfies one of the three conditions in Defn. with every j & A. This guarantees that the latents in A are disentangled from all others, establishing block-wise identifiability.
sets of observed variables (X K, Xv) so
(i) 4 \ (I2 13) Step 1: Take XK = X1. Xy X2. Then E IK \ Iv and every j E In lies Tv \ TR, so case (iii) applies. Step Take Xx X1, Xv X3. Now E Tx Ty and every j Ia is in Tv Ix, again case (üi). All indices outside A belong to Iz or I3 (or both), so A is disentangled.
(ii) 12 (11 U1) Symmetric to (i) with the roles of (1,2) and (2, 1) swapped: use (XK, (X2, X1) and (X2,X)).
(iii) \ (2)
Symmetric (i): use (X Xv) (X3, and (X3, X2).
(iv) (11 0 12) 13 This is the worked example already given; we recap for completeness. Step 1: (X1, yields E 1K O Iv. j E LL (case (ii)). Step 2: (X1 X2, X3) yields i 1K Iv,
j E ly (case (iii)).
(v) (1 l1) \ I2 Step Xx X1, Xv = X3: i In O Iv, j E Ik (case (ūi)). Step 2: Xx =x=x11 X1 X3. Xv =x==:::0 iz T Iv. Tv (case (ii)).
(vi) (12 12) 1 Step 1: Xx = X2, Xv Xa: i In n lv, j E AR (case (ii)). Step 2: Xx X3 Xy Xiiie Ix Iv, Iv (case (tii)).
(vii) J 0 T3 Step 1: X = X1, Xy = Xx: i J 0 Tv. any j that differs only by presence in In or Iz lies in Tk AIy (case (ii)). Step 2: X X1 UX, X = Xx: i E Tx TTy while every remaining j either (a) appears in exactly one of L,, 12, 13 and so is in Agy (case (ii)), or (b) lies solely in la and Iv \ Ix (case (ii)).
In every case the chosen pairs cover all j & A, so each atomic region is disentangled from the rest and hence block-wise identifiable under the invertibility assumption.
25

---

Published as a conference paper at ICLR 2026
B.2 FURTHER DISCUSSION ON THE CONNECTION OF CONDITIONS.
The spirit of many of our conditions stems from the classical literature of latent variable models. Here we discuss the connections in more details:
Connection between diversity and sparsity assumptions. The structural diversity condition is firstly introduced in our work for the nonlinear case, but other conditions on the structure appear in related field, e.g., the sparsity condition. For instance, the anchor feature assumption (Arora et al., 2012; Moran et al.. 2021), structural sparsity Zheng et al. (2022), sparse dictionary (Garfinkle & Hillar, 2019), and many others. Although both diversity and sparsity concern latent structure, they are fundamentally different. Diversity focuses on variation in the dependency between latent and ob- served variables, and does not require sparsity all. remains valid even in nearly fully connected graphs, as long as there is some variation (c.g., even single differing edge) in the connectivity patterns across variables. In contrast, sparsity conditions do not imply diversity and always enforce sparse connectivities.
Connection between injectivity and RIP assumptions. Injectivity is a standard requirement to en- sure that latent information is not lost through the generative map. In the linear setting, this reduces to classical Restricted Isometry Property (RIP) conditions (Foucart Rauhut, 2013; Jung et al., 2016), which are known to be necessary for linear dictionary learning. Our injectivity condition is the natural nonlinear analogue: it prevents degenerate mappings that collapse latent variation, stan- dard in almost all previous work nonparametric identifiability (Hyvärinen al., 2024; Moran & Aragam, 2025).
Connection between dependency sparsity and latent sparsity regularizations. Beyond assump- tions on the data generating process, there are also connections in terms of regularization during estimation. Most prior work on sparse dictionary learning imposes sparsity directly on the recov- ered latent variables Z. The most prominent example is the Sparse Autoencoder (SAE), which is based on sparse dictionary learning and widely used in mechanistic interpretability. However, as highlighted by recent reviews (Cunninghamn et al., 2023), latent sparsity causes issues such as fea- ture absorption and extremely high latent dimensionality (e.g.. millions of variables). In contrast, we regularize sparsity on the dependency structure (Jacobian sparsity) rather than the latents them- selves, which avoids these issues and has shown benefits both in our own experiments (Sec. 4.2) and in recent work on large language models, such as the Jacobian Sparse Autoencoder (Farnik et al., 2025).
B.3 FURTHER DISCUSSION ON THE CONNECTION WITH SAES.
Discussion. Mechanistic interpretability often seeks to uncover the underlying concepts that drive the behavior of large language models, whether understand sources of hallucination to explain model responses. Sparse Autoencoders (SAEs) have been widely used for this purpose, but, as noted in the community [3], SAES rely on sparse dictionary learning and therefore inherit its limitations:
First, SAEs fundamentally assume a linear generative function, since sparse dictionary learning lacks nonlinear identifiability. As you mentioned, the Linear Representation Hy- pothesis (LRH) is reasonable working assumption in many contexts, but LRH concerns the linear relation within the latent space of Z rather than the linearity of the generative map g in X g()). In reality, most models have highly nonlinear generative functions (for example, due to nonlincar activations such as ReLU or GeLU), Assuming lincarity at the generative level simplifies implementation but inevitably introduces bias and prevents full recovery of the true latent factors in these nonlinear settings.
Different from the theoretical foundation of SAEs (i.c., sparse dictionary learning). our diverse dictionary learning provides theoretical guarantees for nonlinear latent variable models, offering a provably rigorous tool for mechanistic interpretability in almost all real- world scenarios, covering the previously unsupported nonlinear cases.
Second, SAEs impose sparsity directly on the latent variables. This often forces extremely digh--imnnsional latent spaces (e.g., millions of units) represent real-world concepts. More, latent sparsity can cause feature splitting and absorption, where important concepts are lost due to the encouragement of sparse latents. For instance, if we only maximize the
26

---

Published as a conference paper at ICLR 2026
latent sparsity, each feature will only corresponds several samples, capture concepts that are very specific (c.g., persian cats) versus more general concepts (c.g., cats).
In contrast, diverse dictionary learning encourages dependency sparsity (Jacobian sparsity) rather than latent sparsity, and is designed for nonlinear generative models with identifia- bility guarantees. This provides a principled solution to both limitations of SAEs and has meaningful implications for mechanistic interpretability.
Empirical Evidence. The recent Jacobian Sparse Autoencoder (ISAE) work (Farnik et al., 2025) provides extensive empirical support for the same dependency sparsity regularization required by our diverse dictionary learning theory. Their results suggest that replacing traditional SAE losses with dependency sparsity-base losses can improve both interpretability and efficiency. We therefore refer readers in mechanistic interpretability to their thorough empirical study.
At the same time, there one perspective that (Farik et al., 2025) has not evaluated against other SAE base- lines, i.e., the number of dead features. To make the empirical evedeinces even ducted new experiments on the WeenWettext with GPT2- Small, comparing JSAE with Top-K SAE (Gao al., 2025) and Batch Top-K K SAE (Bussmann et al.,, with the latent dimension as 12, 288. The results are in Table
Table 3: Number of dead features under different methods
comprehensive,
con-
Method
# of Dead Features
T SAE Batch Top-K JSAE
439
207
62
These results, together with the comprehensive experiments in (Farnik et al., 2025), further sup- port the advantages of dependency sparsity, demonstrating that it not only yields more interpretable representations but also preserves active and meaningful latent features more effectively.
B.4 FURTHER DISCUSSION ON SUFFICIENT NONLIENARITY
The sufficient nonlinearity assumption is meant to ensure that the Jacobian varies enough across samples so that its Jacobian vectors span the relevant support. Although this condition may seem demanding at first glance, it is usually quite mild in practice. For each observed variable X,, the requirement involves only (Dzg),..lo samples, which is simply the number of latent variables that influence X,, and this number is typically far smaller than the available sample size.
When g is smooth and the latent distribution has a continuous density, Jacobian evaluations in- dependently drawn samples form continuous random vectors. Such vectors are in general position with probability one, which means that small number of random samples already produces Jaco- bian rows that span the required support. For example, if X; depends on five latent coordinates, then roughly five random samples usually sufficient, even in much higher dimensional systems.
The second part (upp((D2g(=(*))))) pendencies being diminished across all samples during estimation. We have (Dzū(²)). = (D2g(=)),.hh(zz 2), which already lies inside nupp((D2)):.) The assumption requires the istence of several samples in the whole space where that relation holds for a matrix of the same support as h. The key nuance is that the condition only require that matrix exists for several samples of 2 in the whole continuous space, but identifiability is defined as an asymptotic property over infi- nite samples. Conceptually, this condition implies that the estimation should have denser Jacobian compared to the ground truth. It rules out the case where true dependency is globally eliminated during estimation. practice, we start with a neural network, which typically has a fully dense Jacobian when initialized, and then applying sparsity regularization to shave away redundant de- pendencies. A violation of this assumption would therefore correspond to global cancellation of genuine dependency across all samples, which is intuitively similar to a violation of faithfulness. As a result, alternative conditions may also be applied to achieve similar goals.
S supp((D2))..) avoids the cases where true de-
B.5 DEPENDENCY SPARSITY REGULARIZATION ON LARGE MODELS
Computing the full Jacobian of large model can be expensive. Fortunately, recent work (Famik et al., 2025) shows that dependency sparsity regularization remains practical even at scale up to common LLMSs when combined with two standard strategies.
27

---

Published as a conference paper at ICLR 2026
Apply latent sparsity first. Large models often have high dimensional latent spaces, but for any given input, many latent coordinates are inactive or irrelevant. A common approach is to first identify the active coordinates and compute the Jacobian only with respect to this subset. Since the active block is often tiny compared to the full latent space, this reduces both computation and memory by several orders of magnitude in typical transformer archi- tectures [1]. In practice, the Jacobian is rarely formed as a dense d2 x dg matrix, but only as restricted slice that corresponds the active latent directions for that specific input.
Use efficient closed-form expressions. For several widely used architectures, the Jaco- bian with respect to selected latent directions has efficient factorizations. As shown in [1], models with residual attention and feedforward structure admit closed form expressions for the relevant Jacobian blocks that require only few matrix multiplications and inexpensive elementwise operations. This avoids repeated backward passes through the full model and keeps the cost manageable even when the underlying network is large.
With these strategies, training a large model with our dependency sparsity regularization is reported to be only about twice as slow as training with standard & regularization on the latent variables Z (Farnik et al., 2025).
B.6 GENERAL NOISE AND NON-INVERTIBILITY
In our paper, we follow the standard setup the identifiability literature regarding noise and invert- ibility because violations of it usually require much stronger assumptions to compensate. That said, we feel that more discussion on how to handle these cases is helpful in certain scenarios.
Handling general noises. Most identifiability results for latent variable models focus on additive independent noise or noiseless settings. This holds for both classical linear models (Reiersøl, 1950) and recent nonparametric work (Hyvärinen et al., 2024). The main difficulty is separating noise from latent variables, since in the general form they can be entangled in complex ways. Recent work (Zheng et al., 2025), based on the Hu-Schennach theorem (Hu & Schennach, 2008), shows that general noise be separated under additional assumptions the generative function f and on conditional independence across groups of observed variables. As noted in Remark 1, under the same conditions, our results extend to settings with general noise.
Handling partial non-invertibility. Invertibility and its variants among the most common assumptions in the literature. Without additional assumptions, it can even be necessary, since in- formation that is lost during generation cannot be recovered in principle. In scenarios where partial non-invertibility must be addressed, one may consider incorporating temporal information together with sufficient changes in the nonstationary transition (Chen al., 2024), which can provide the extra information needed for recovery. Since our theory focuses on the general setting without any auxiliary information, we adopt the standard assumption of invertibility.
B.7 FURTHER DISCUSSION ON POTENTIAL IMPACT.
In a nutshell, recovering the ground-truth data generative process is important for both predictive and non-predictive tasks. Machine learning is fundamentally a balance between inductive bias and data. By uncovering the hidden world underlying the data, we obtain principled, domain-agnostic inductive biases grounded in fundamental understanding rather than heuristics. These insights can be embedded directly into model architectures, training objectives, and evaluation protocols, yielding systems that more robust, generalizable, and data-efficient. Importantly, these inductive biases are not merely theoretical; they have already produced measurable improvements across diverse real-world domains.
We first highlight two overarching benefits:
Truthfulness. It is well known that, given any pair of observationally equivalent models (e.g., models perfectly trained by MLE), without additional assumptions, there is guarantee all that these models actually recover the underlying data generative processes. This has led to the critical issue of non-identifiability of the nonlinear latent variable model (e.g., the classical work the
28

---

Published as a conference paper at ICLR 2026
non-uniqueness of nonlinear ICA (Hyvärinen & Pajunen, 1999)), and has been widely validated by large-scale empirical studies on unsupervised learning (e.g., (Locatello et al., 2019)). This raises significant challenges for tasks (e.g,, transfer learning, controllable generation, compositional gen- eralization, and mechanistic interpretability) where we need to understand the true process, rather than purely fitting the observed distribution. Therefore, understanding what aspects can still be covered, and what inductive biases should be introduced to guide recovery, provide both theoretical guarantees and practical guidance on ensuring the truthfulness of the learning. These are exactly what our paper aims to address.
Efficiency. Recovering the underlying true generative factors provides a principled solution to improve the efficiency, since we only need to model those essential representation capturing what we are interested, instead of the whole high-dimensional space that also include numerous irrelevant information or arbitrary noises, For instance, if we can make sure that the recovered latents will not be entangled with other latent variables, we can precisely only on those that relevant our tasks (e.g., content for transfer learning, or specific latent concepts we need to modify) instead of the whole latent spaces, not only reducing the cost but enforcing the model to focus only on those relevant, improve the efficiency in a principled way.
We have conducted experiments on visual disentanglement to illustrate one of the potential appli- cation scenarios. Across datasets and backbone models, the results consistently show that adding the identifiabiltt-guudke regularization (dependency sparsity) enables the model to recover the un- derlying generative factors. This not only improves disentanglement performance, which is itself practically important, but also directly benefits the following application scenarios.
Mechanistic Interpretabilityy In many scenarios, we aim to study the underlying concepts that generate the responses of LLMs, for many reasons such as investigating the root of hallucination or explaining the mechanisms for specific responses or behaviors. This is related to the field of Mech- anistic Interpretability, where Sparse Autoencoder (SAE) has been popular. However, as known by the community (Sharkey et al., 2025), SAE is based on the theory of Sparse Dictionary Learning, and there are some open problems:
First, SAE can only deal with linear data due to the lack of nonlinear identifibality of sparse dic- tioanry learning. However, the real-world is full of nonlinearity. Assuming everything is linear simplicity the procedure and will unavoidably bring bias and cannot fully recover the truth.
Moreover, SAE encourages sparsity over the latent variables. As a result, it needs an extremely high-dimensional vector capture real-world concepts (e.g., millions of dimensions), and some features are easy to be absorbed due to its encourage on latent sparsity. In contrast, our Diverse Dictionary Learning encourage the sparsity on the Jacobian (i.e.. dependency sparsity) instead of latent diversity, and can handle nonlinear generative processes with identiability guarantees. This provides principled solution to both open problems of SAE, which highly significant for the whole filed of mechanistic interpretability.
Transfer Learning. In transfer learning, it is important disentangle the invariant and changing part. such as disentangling invariant content from changing styles. Identifiability has been widely leveraged in the previous literature on guarantecing reliable and efficient domain adaptation (c.g., (Von Kügelgen et al., 2021; Kong et al., 2022; Li et al., 2023)). Given two observed variables, our generalized identifiability results guarantees the recovery of the shared and private parts of their la- tent variables, under a simple regularization of dependency sparsity. This provides a flexible frame- work for transfer learning that can really capture the essential part across domains.
Controllable Generation. A perfect prediction machine can only master at mining correlations, while leveraging the true causation relies on identifiability. For instance, if we want to add eyeglasses on a kid's face, due to the overwhelming correlation between wearing eyeglasses and the relatively higher age, models sometimes will also increase the age the kid, even for large foundational models. This negligence of the true underlying process is one of the fundamental reasons why large models, although extensively trained on web-scale data, can still provide many extra changes that are out of our control. Previous work has already shown that, by incorporating inductive bias guided via identifiability, models can achieve precise control on the generated images (Xie et al., 2025a; 2023).
29

---

Published as a conference paper at ICLR 2026
Dim
Ours
OroJAR
Hessian Penalty
3
0.8258
0.0085
0.7288 + 0.0280
0.8257 + 0.0240
4
0.8449
0.0043
0.6301 + 0.0810
0.8352 0.0396
5
0.8048 0.0080
0,5119 i 0.1482
0.7789 0.0174
Table 4: MCC under different regularization penalties across dimensions (mean + std, higher is better).
Dim
Ours w/o noise
Ours w/ noise
Base
3
0.8258 + 0.0085
0.8210 + 0.0088
0.3814 + 0.0369
4
0.8449
0.0043
0.8381 1 0.0093
0.5467 1 0.0326
5
0.8048 = 0.0080
0.7944 = 0.0134
0,4576 + 0.1075
Table 5: MCC across dimensions with and without noise (mean higher is better).
Multi-modal Alignment. Similarly, one may consider multiple modalities as multiple sets of ob- served variables, where the semantically meaningful concepts are those that shared by multiple modalities, and the modality-speciffc concepts (e.g., texture for images, volumns for audio) are those private latent variables. By formalizing the process general dictionary learning problem, many previous work have already shown the practical implication of identifiability, such as address- ing information misalignment (Xie et al., 2025b) and capturing complex interactions across different modalities (Sun et al., 2024).
Scientific Discovery. Understanding the hidden truth from observation is always of the main tasks of scientific discovery. Newton observed the falling apple (observation X), then he got curi- ous, studied much, and found out it was actually gravity (latent variables Z) causing items to fall. Therefore, the fundamental task of scientific discovery may be sumamrized into the simple equation of fZ). where we aim recover based solely on X, which is exactly our task of general dictionary learning. Of course, the guarantee that the recovered latent variables Ž correspond to the ground-truth latents Z in meaningful way is what matters the most, and this is exactly the focus of our generalized identifiability theory. There have been numerous successful applications of identifiable representation leraning in the literature, such dynamic systems like climate change (Yao et al., 2024a), robotics (Lippe et al., 2023), neuralimaging (Hyvärinen & Morioka, 2016) and genomics (Morioka & Hyvarinen, 2024).
Of course, the list non-exclusive, and will only grow faster given the development of large-scale models. The reason is simple: even with infinite data and computation, without identifiability, mod- els may achieve perfect predictions but cannot be guaranteed to recover the underlying truth. Thus, the closer we are that boundary, the more efforts we should put into going beyond correlation.
C ADDITIONAL EXPERIMENTS
In this section, we present further experiments both synthetic and real-world data.
C.1 ADDITIONAL SYNTHETIC EXPERIMENTS
We begin with series of additional experiments in the synthetic setting.
Additional baselines. We first compare dependency sparsity alternative regularizers. Table reports MCC for d E (3, 5) against two Jacobian/Hessian penalties: OroJAR (Wei et al., 2021) and the Hessian Penalty (Peebles al., 2020). Neither provides identifiability guarantees in the non- parametric setting. Empirically, both underperform our method, with widening gap as d increases. This indicates that penalizing the dependency map in a structural way improves recovery the true latent factors.
Noise robustness. Remark states that our framework naturally extends to generative processes with additive noise. Table 5 confirms this: MCC remains essentially unchanged compared to the
30

---

Published as a conference paper at ICLR 2026
Dimensionality
A
3
5
0
0.6789 + 0.0364
0.7317 + 0.1092
0.6989 + 0.0133
0,001
0.7313 1 0.0242
0.729411..014.
0.7513 1 0.0007
0.005
0.7765 + 0.0363
0.7681 = 0.0455
0.01
0.8145
+ 0.0221
0.8032
0.0327
0.7979 + 0.0421
0.03
0,8268
1 0.0268
0.8232 + 0.0187
0.8101
= 0.0112
0.05
0.8256 + 0.0088
0.8420 + 0.0401
0.8099 = 0.0296
Table 6: MCC across different À values (sparsity regularization weight) and dimensionalities tst, higher better).
Method
FactorVAE
DCI
Factor
0.708 + 0.026
0.135 + 0,030
Latent Sparsity
0.501 + 0.434
0.113 + 0.069
Dependeney Sparsity
0.752 + 0.040
0.144 - 0.053
Dependency Sparsity (128)
0.723 +: 0.023
0.141 + 0.004
Table 7: Quantitative comparison on Cars3D dataset (mean=std, higher is better).
noiseless case, with only minor drops, while the base model degrades sharply. This supports the claim that dependency sparsity stabilizes latent recovery under noise. Extending identifiability to arbitrary noise remains more challenging in the nonparametric setting, since invertibility can break down and stronger assumptions are typically required.
Regularization weight. To examine the effect of regularization strength, we vary the sparsity weight À in Table 6. MCC increases steadily from À O and plateaus around À [0.03, 0.05]. showing that the method stable and not overly sensitive once past the under-regularized regime. Importantly, sparsity here serves only assume the data-generating process itself is sparse. Instead, relies on structural diversity, which can hold even in dense settings. Moreover, the set-theoretic framework is robust to partial violations of assumptions and still enables meaningful recovery when full identifiability is unattainable.
inductive bias during estimation. Our theory does not
C.2 ADDITIONAL VISUAL EXPERIMENTS
We next evaluate more on images, providing both quantitative comparisons and qualitative analyses.
To assess scalability, we upsampled Cars3D to 128 x 128 and re-ran Factor with dependency sparsity. As shown Table (last row), performance remains consistent with 64 x 64 setting. This suggests that the observed improvements stem from leveraging structural regularization rather than resolution, and that the method scales robustly with image size.
Scalability.
Quantitative evaluation. Table 7 evaluates FactorVAB score and DCI on Cars3D. Adding de- pendency sparsity improves FactorVAE from 0.708 to 0.752 and DCI from 0.135 to 0.144. Latent sparsity often underperforms. Table extends MP33D and includes OroJAR and the Hessian
Cars3D
MP13D
Method
Factor VAE t
DCI f
FactorVAF
DCT 1
VactorVAF
0.708 0.026
0.135 + 0.030
0.599 + 0.064
0.345 0.047
Latent Sparsity
0.501 1 0.434
0.113 1 0.069
0.440 1 0.065
0.325 1 0.028
OroJAR
0.165 + 0.235
0.030
= 0.007
0.499 + 0.090
0.272 = 0.054
Hessian Penalty
0.321 + 0.455
0.082
= 0.077
0.506 - 0.056
0.254
- 0.067
Dependency Sparsity
0.752 + 0.040
0.144
0.053
0.639 + 0.084
0.384 + 0.031
Table 8: Quantitative comparison Cars3D and MP13D datasets (sta higher is better).
31

---

Published as a conference paper at ICLR 2026
Figure 7: Latent variable visualization on Fashion with Flow Dependency Sparsity. From top to bottom, the latent variables correspond gender, heel height, and upper width.
Figure 8: Latent variable visualization on Shapes3D with EncDiff + Dependency Sparsity. From top to bottom, the latent variables correspond to wall angle, wall color, object shape, and object color.
Penalty. Dependency sparsity gives the best results on both datasets, improving Factor and DCI while maintaining backbone training stability.
Qualitative evaluation. A key goal of these experiments is to test whether dependency sparsity leads to more interpretable and disentangled latent representations in visual domains. Figure 8 shows latent traversals on Fashion (Xiao et al., 2017) with Flow. Individual latent coordinates correspond cleanly to gender, heel height, and upper width, with minimal interference across factors. The Shapes3D traversals in Figure (EncDiff) show similarly sharp control, disentangling wall angle, wall color, object shape, and object color. These traversals illustrate that dependency sparsity yields latent axes that align with semantic attributes and preserve orthogonality among factors.
Figures 9, 10, and 1I further evaluate controllability via latent swapping. On Shapes3D, swapping single factor cleanly transfers floor or wall color while leaving other factors intact. On Cars3D, EncDiff isolates azimuth and color. On MPI3D, rotation and background are controlled indepen- dently. These results highlight that dependency sparsity encourages localized and non-overlapping influences, enabling intuitive editing operations without unintended side effects. At the same time, the generative quality of the backbone (diffusion in this case) is preserved, with realistic outputs.
Together, these traversals and swaps reinforce the quantitative results: dependency sparsity not only improves disentanglement scores but also enhances interpretability and practical usability of the leamned latents. By aligning latent dimensions with distinct semantic factors, it enables robust single- attribute manipulation and semantically meaningful latent arithmetic. These benefits are precisely what identifiability is meant guarantee, providing further empirical validation of our theory.
Additional results on controllable generation. Moreover, we conduct further experiments on the benefit of recovering task-relevant representation for controllable generation. We consider the problems of
D DISCLOSURE STATEMENT
Grammar checks were performed using LLMs; no significant edits were made.
32

---

Published as a conference paper at ICLR 2026
SRC
TGT
WALL FLOOR
Figure 9: Latent variable visualization on Shapes3D with EncDiff + Dependency Sparsity. Top row: source. Second row: target. Each subsequent row modifies the source by swapping a single latent factor (floor color wall color) from the target.
SRC
TGT
Color Azimuth
Figure 10: Latent variable visualization on Cars3D with EncDiff + Dependency Sparsity. Top row: source. Second target. Each subsequent row modifies the source by swapping single latent factor (azimuth and color) from the target.
BG ROT TGT SRC
Figure 11: Latent variable visualization on MPI3D with EncDiff Dependency Sparsity. Top row: source. Second row: target. Each subsequent row modifies the source by swapping a single latent factor (rotation and background) from the target.
33


Reviews:
[{"review_id": "EXQreiY9Tk", "reviewer": "Reviewer_xMy9", "summary": "This paper studies a set theoretic notion of dictionary learning, where the properties of the collection of atoms and the model are defined at the full set theoretic level of abstraction, and the generative model admits a Jacobian. The authors define a series of strenghtenings of identifiability with respect to how accurate to “ground truth” the estimated values are. They then study the effect of sparsity. Finally, they define a diversity condition that allows for element-wise recovery. They evaluate their theoretical results empirically.", "questions": "In addition to the questions above:\n\nMaybe I’m missing something, but why is Theorem 1, condition (ii) called “regularization”? To me regularization is algorithmic, whereas this is structural.\n\nCan you please clarify why your experiments are about structural properties of dictionary learning, rather than the specific methods you have implemented?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The main strength of the paper is in abstracting sufficient conditions for recovery of various consistent dictionaries from examples. It is powerful to reason about these properties in a general class of generative models using only set theoretic terms.", "weaknesses": "Some of the theoretical claims are imprecise. In particular, I am really puzzled by the interplay between Definition 6 and Theorem 1. Is Theorem 1 a result about methods that satisfy the regularization condition? Or should I read it as a result about the nature of identifiable models, i.e., they are ones that have higher sparsity than all others that are observationally equivalent? This requires clarification. Also, I am a bit confused about Theorem 2: if all the conditions of Theorem 1 are met, you get both generalized and structure identifiability? There are no additional conditions?\n\nRelated to all of this: in section 3.3, what is the quantifier over $\\hat{z}, \\hat{g}$? I thought it was the estimated value. I don’t see where it shows up in Assumption 1 until the last part, and in Theorem 1, the hat versions are specified but somehow you are invoking Defn 6, in which the quantifier is “any” theta? \n\nOn the necessity side: Are there any necessity conditions for set-theoretic indeterminacy or generic identifiability?\n\nFinally, it would be helpful in your exposition to map your abstracted properties to properties in general linear dictionary learning, non-linear dictionary learning, and combinatorial dictionary learning. (e.g., diversity = RIP = well-structuredness).\n\nI can consider increasing my evaluation of the paper if the theoretical claims are made more precise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pk0GKI5LFX", "reviewer": "Reviewer_KUfT", "summary": "The authors introduce a new, weaker form of identifiability based on \"set-theoretic indeterminacy\". The main theoretical result is the Set-Theoretic Guarantees (Theorem 1): that even when full recovery is impossible, structural relationships between latent variables are identifiable. Specifically, the intersection ($I_K \\cap I_V$, shared factors), complement ($I_K \\setminus I_V$, exclusive factors), and symmetric difference ($I_K \\Delta I_V$, unique factors) of latent variable sets are shown to be disentangled.\n\nThese identifiability guarantees (Theorems 1 & 2) are achieved by introducing a \"simple inductive bias\" during estimation: a sparsity regularization on the dependency structure (i.e., the Jacobian $D_{\\hat{Z}}\\hat{g}$). This is a practical regularization rather than a strict assumption about the data-generating process itself.\n\nThe authors show that their set-theoretic guarantees naturally extend to full, element-wise identifiability (the traditional goal) if a \"Sufficient Diversity\" condition (Assumption 2) is met. This condition, which is shown to be a weaker and more general structural requirement than in prior work, essentially ensures the dependency structure is rich enough for each latent variable to be isolated in its own \"atomic region\" of the set-theoretic Venn diagram.\n\nFinally, the authors conduct experiments on synthetic and real-world image datasets to validate their theory and demonstrate the practical benefits of the proposed dependency sparsity bias.", "questions": "- The conclusion ends suggesting to explore identifiability in foundation models. How would one practically implement and optimize this dependency sparsity bias in a large-scale transformer as the full Jacobian of a transformer's activations is computationally intractable. \n- How does the proposed diverse dictionary learning compare to traditional SAE methods for *interpreting* model architectures rather than solely looking at traditional datasets? Would the result be identical to that of Farnik et al. 2025 (Jacobian Sparse Autoencoders)?\n- I am not sure I fully understand how the terminology and figures would map onto a real world example. In a complex domain like images, what would these \"atomic regions\" practically represent? For example, if $I_1$ = \"dog\" latents and $I_2$ = \"cat\" latents, could the intersection $I_1 \\cap I_2$ be the latent for \"furry,\" and the complement $I_1 \\setminus I_2$ the latent for \"barks\"? How does this set-based view compare to traditional feature representations?\n\nGenerally, I do not see many issues with the paper, and if my confusions are addressed during the rebuttal I would be happy to raise my score accordingly!", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The research area of expanding traditional dictionary learning inductive priors to learn true latents of the data is interesting and a valuable area of research. \n- The paper appears theoretically sounds. The authors provide rigorous definitions for all new theoretical concepts and include detailed proofs in the appendix. I took a brief look at the proofs and did not find any issues, although I was not able to follow everything and there is a good chance I could have missed an issue. \n- The writing is very clean and easy to understand (although the math is dense) with associated well made figures backing up the intuition with toy examples which really helped my understanding of the paper.", "weaknesses": "- The paper discusses SAEs in the introduction but never again throughout the paper. I was expecting some experiments or discussion about whether Diverse Dictionary Learning could be used in-place of traditional SAE losses. This shouldn't be difficult to do as there are many small models + SAE libraries available and would be really interesting to compare. Indeed, SAEs assume the linear representation hypothesis, but there is some decent evidence that the LRH is at least somewhat true in many scenarios - how does the proposed method do when examining model internals?\n- The main theory relies heavily on Assumption 1 (Sufficient Nonlinearity). I checked the referenced papers in Lines 303 and found that some of them contained similar statements but it was hard to determine if they were identical. I think the paper would be much stronger if the authors provided a bit more intuition as to why Assumption 1 is reasonable to make. Even better, if there were some small experiments to show that this generally holds empirically, that would strengthen the paper a lot. However, if this really is a very standard assumption, then this evidence is not wholly necessary, but I would appreciate the authors highlighting exactly which equations in which papers make the same assumption so I can more readily compare the two. (For example, I am guessing Assumption 1 in Lachapelle et al., 2022 makes the same argument?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5XGBKowrM", "reviewer": "Reviewer_U1DX", "summary": "The paper studies nonlinear dictionary learning under a general generative model $X = g(Z)$, where both the latent variables $Z$ and the generator $g$ are unknown. It introduces a set-theoretic notion of identifiability, focusing on recoverability of set operations: intersections, complements, and symmetric differences. Under assumptions of positive density and dependency sparsity, the authors show that these structural dependencies are identifiable, and that element-wise identifiability follows when the system exhibits “sufficient diversity.”", "questions": "Do you have any results or insights on how accurately the support of $D_z g$ can be recovered with finite samples or under noise?\nCan the diversity condition be checked empirically?\nWould it be possible to report direct recovery metrics?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper provides a practical and general inductive bias that improves disentanglement without model-specific changes, May be useful for interpretability and modular representation learning.\n2) Strong theoretical results: generalized and structural identifiability (Theorems 1–3) under mild assumptions.\n3) The “sufficient diversity” condition unifies and relaxes previous sparsity-based identifiability assumptions.\n4) Introduces a novel set-theoretic identifiability framework that generalizes traditional recovery notions through operations on supports of $D_z g$.", "weaknesses": "Below I list a few weaknesses of the paper: \n\n1. The “sufficient nonlinearity” and independance conditions on $D_z g$ are elegant but may be hard to verify or hold only asymptotically. Some discussion of finite-sample behavior or diagnostics would help.\n\n2. Theoretical results assume $\\ell_0$ regularization on $|D_z \\hat g|_0$, but experiments use an $\\ell_1$ proxy. A clearer justification or sensitivity analysis for this relaxation would strengthen the bridge between theory and practice.\n\n3. Penalizing full Jacobians can be expensive for large decoders or diffusion models. Runtime or memory comparisons against latent-sparsity or Hessian-based baselines are missing.\n\n4. Theoretical claims focus on structural identifiability, but evaluations (e.g., DCI, FactorVAE) measure disentanglement indirectly. A metric targeting recovery of the Jacobian’s support would better reflect the paper’s main claims.\n\n5. Noise and model misspecification (e.g., non-additive noise, partial non-invertibility) are briefly mentioned but not deeply explored; results may not generalize to such cases.\n\n**On another note, I am not sure how good of  a fit is this theoretical endeavor suitable for ICLR**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QWhOlv0kGE", "reviewer": "Reviewer_P3Ud", "summary": "This paper introduces the problem of diverse dictionary learning to make identifiability actionable in the real-world scenario. In particular, they study: in the general settings where full identifiability is unattainable, what can still be recovered with guarantees, and what biases could be universally adopted?\n\nSection 3 introduces the main technical results.\nThe current presentation is hard to see it's impact in real problems. \nThere is a paragraph started with \"Why does it matter in the real world?\" By reading that paragraph, it's still not clear how the research problem in this paper relevant to any objective that matters in reality, such as improving the performance of a predictive model or dimension reduction. \n\nThe current simulations don't demonstrate the potential usefulness of the work. The synthetic data part focuses on Generalized Identifiability and Element Identifiability. Without knowing the possible impact of quantifying them, it is hard to see the potential value. \n\nThe section 4.2 is written in the way for readers who are familiar with the references. It does not provide sufficient explanation of the experimental results and their potential impact in practice. \n\nOverall, it is hard to see the potential impact of this work. A better justification is needed.", "questions": "See in the Summary; especially in the later part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is free of grammatical errors.", "weaknesses": "See in the Summary; the later part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)