Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability
Authors: Hoang-Son Nguyen
Xiao Fu
Venue: NeurIPS 2025 poster ()
Keywords: identifiability
Jacobian
component analysis
nonlinear mixture
Primary area: probabilistic_methods
TL;DR: DICA identifies latent components from nonlinear mixtures by maximizing Jacobian volume, bringing identifiability without source independence, Jacobian sparsity, or auxiliary signals.

Abstract: Latent component identification from unknown *nonlinear* mixtures is a foundational challenge in machine learning, with applications in tasks such as self-supervised learning and causal representation learning. Prior work in *nonlinear independent component analysis* (nICA) has shown that auxiliary signals---such as weak supervision---can support *identifiability* of conditionally independent latent components. More recent approaches explore structural assumptions, like sparsity in the Jacobian of the mixing function, to relax such requirements. In this work, we introduce *Diverse Influence Component Analysis* (DICA), a framework that exploits the convex geometry of the mixing function’s Jacobian. We propose a *Jacobian Volume Maximization* (J-VolMax) criterion, which enables latent component identification by encouraging diversity in their influence on the observed variables. Under suitable conditions, this approach achieves identifiability without relying on auxiliary information, latent component independence, or Jacobian sparsity assumptions. These results extend the scope of identifiability analysis and offer a complementary perspective to existing methods.

Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability
Hoang-Son Nguyen and Xiao Fu
School Electrical Engineering and Computer Science
Oregon State University
{nguyhoa3, xi www..oorrggonttttt.ccommm edu
Abstract
Latent component identification from unknown nonlinear mixtures is a foun- dational challenge in machine learning, with applications in tasks such self- supervised learning and causal representation learning. Prior work in nonlinear independent component analysis (nICA) has shown that auxiliary signals-sun such as canoon--nn weak supervision- can support identifiaility of conditionally independent latent components. More recent approaches explore structural assumptions, e.g., sparsity in the Jacobian of the mixing function, to relax such requirements. In this work, we introduce Diverse Influence Component Analysis (DICA), a framework that exploits the convex geometry of the mixing function's 's Jacobian. We propose a Jaco- bian Volume Maximization (J- VolMax) criterion, which enables latent component identification by encouraging diversity in their influence on the observed variables. Under reasonable conditions, this approach achieves identifiability without relying auxiliary information, latent component independence. Jacobian sparsity assumptions. These results extend the scope of identifiability analysis and offer a complementary perspective to existing methods.
Introduction
Nonlinear mixture model idenrification (NMMI) seeks to uncover latent components transformed by unknown nonlinear functions. typical data model of interest in the context of NMMI is as follows:
x f()), 8 E Rd æ E R""
(I)
where ($1...., sa) p(s) is a random vector following certain distribution p(s) with support S. f Rdd + I is unknown, nonlinear mixing function, observed data, and 8; for [d] and x, for j E [m]] represent the ith latent component and the jth observed feature, respectively. The function is modeled a diffeomorphism that maps latent variable to a d-dimensional Riemannian manifold X' embedded in R., where m 2 d [ Given the observations (samples) of æ. NMMI amounts to recovering and f up certain acceptable or inconsequential ambiguities. NMMI and variants play fundamental role understanding many machine learning tasks, e.g., latent disentanglement [I 4. causal representation learning [5] 6
(Z1...., Tm E R is the
object-centric learning (2). and sulfvvuee learning (7.
The NMMI task is clearly ill- -o--edeed-in general, infinite number of different (f. s) could be found from observations of under (1). Hence, establishing identifiability of f and s becomes a central topic in NMMI. This identifiability challenge was extensively studied under the umbrella of nonlinear independent component analysis (nICA) [9H12]. A key take-home point that nICA poses a much more challenging identification problem relative to ICA (where f is linear system). That is, even if $1....; Sd are statistically independent, the model in not identifiable
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In recent years, much progress has been made in understanding identifiability of (T. The line of work [ showed that if Sq...., St are conditionally independent given certain auxiliary variable u (which can be understood as additional side information), then f and can be recovered to reasonable extents. Another line of work tackles identifiability by assuming that the mixing function f is structured other than completely unknown. For example, the works [13 make an explicit structural assumption that f is a post-nonlinear mixing function, the work [17] assumes that f is conformal, and the more recent work (18) assumes that f is piecewise linear. Using these structures, the auxiliary information can be circumvented for establishing identifiability.
More recent advances propose to exploit implicir structures (other than explicit structures like post- nonlinearity) of f. Notably, the works [3] 19 20] utilize structures defined over the Jacobian of f for identifying the model (T). In particular, it was shown that as long as the Jacobian of f follows certain sparsity patterns, then the model (T) is identifiable [2 2T). Imposing structural constraints on the Jacobian of f is arguably less restrictive compared to assuming explicit parameterization of f. Structures of Jacobian reflect how the latent variables 81,...,. 8d affect the observed features Ti,.... ..... Zm, making the related assumptions admit intuitive physical meaning, Using structured Jaco- bian to model such influences is also advocated by several causal representation learning paradigms (see, e.g., independent mechanism analysis (IMA) 20 under the independent causal mechanism (ICM) principle [22), ) and other frameworks, e.g. object-centric representation learning 2 23 24)
Open Question. The advancements have been encouraging. yet understanding to NMMI identifi- ability remains be deepened. In particular, the Jacobian sparsity- -based upproaches, e.g.. 2 3
19 21). provided intriguing ways of establishing identifiability of the model in [ without using auxiliary variables, latent component independence, or relatively restrictive structural constraints of
f. However, the Jacobian sparsity assumptions in these works essentially assume that the observed features are only generated from a subset of .i...., Sd. which sometimes may not hold. It is tempting to circumvent using such strict sparsity- -aaeed assumptions in NMMI.
Contributions. In this work, we propose to make use of an alternative condition of f's Jacobian for NMML We leverage the fact that, as long as the influences of imposed on each 2, for j = l..... .
are sufficiently diverse, f's Jacobian exhibits an interesting convex geometry. This geometry is similar to the classical "sufficiently scattered condition (SSC)" in the structured matrix faatiixaactoizzttion (SMF)
literature 25 29
and, particularly, the more recent developments polytopic matrix factoriztiion 30 (also see (31. As a consequence, taking intuition from regula-re SMF (25
(PMF)
30
36 we show that fitting the data model in ① together with maximizing the learned f's Jacobian
volume provably recovers I and s up to acceptable ambiguities. The proposed Jacobian volume maximization (J- M-lMaX approach does not rely on auxiliary variables or statistical independence. More notably, an f with a dense Jacobian can still be provably identified under our formulation. These advantages make our identifiability results applicable to wide range of scenarios that are not covered by the existing literature, substantially expanding the understanding of model identifiability under (I). We tested the proposed approach on synthetic data and in a single-cell transcriptomics application. The results corroborate with our NMMI theory.
Notation. Please refer to Appendix A. I for details.
2 Background
From ICA to nICA. ICA is arguably one of the most influential latent component identification approaches. Classic ICA (37 38 assumes that x As with a nonsingular A and that
p(s) I Pc(s))
where at most one 8; is a Gaussian variable. Then, the identifiability of (A, 8) up to permutation and scaling ambiguities can be established by finding linear filter to output independent estimates. Unfortunately, the nICA model, i.c.,
( with condition (2) is in general non-identifiable
nICA with Auxiliary Information. More recent breakthroughs propose to use the following conditional independence model, i.e..
p( (1)
II p1();
to establish fientifiability of (T. The side information u. can be time frame labels TO
21
observation group indices 41 view indices (42) The takeaway from this line of work that, using

---

the variations of u. one can fend against negative effects (e.g., the existence of measure- preserving automorphism (MPA) 66 (3)) leading to con-identifiability under (T. The results are elegant and inspiring. Nonetheless, both u and conditional independence might not always be available.
IICA with Structured f. Another route to establish identifiability is to exploit prior structural information of f. For example, the works ( 18 23 44 used conformal, local isometry, close-to-linear, post-nonlinear, piecewise affine structures, and additive structures of f. respectively. also showed that some of these structures can be used for dependent component identification. Nonetheless, such explicit structures of f. e.g., post-nonlinear mixtures, only make sense when they are used for suitable applications (e.g., hyperspectral imaging [15] and audio separation (16)), yet most problems in generative model learning and representation learning may not have such structures.
16
Recent works, e.g., [14]
15
18
23 48),
Exploiting Jacobian Structures. Instead of imposing explicit structures directly f. it is also plausible to exploit the structures of the Jacobian of f. Note that [J { = 02//0s characterizes how T; influenced by the change of 5j-
The aforementioned IMA approach [20], inspired by the principle of ICM [2 assumes orthogonal columns of J (() at each point E S. But these developments lack comprehensive identifiability
characterizations (see [44). On the other hand, the works (2 3 19 49 assume that J y exhibits
certain type of sparsity pattern. These works formulate the NMMI problem as Jacobian sparsity-
regularized data fitting problems. For example, the work (2) proposed the following:
min E. (I)(((()) f.g
x1 + (J
(4)
where the first term finds a diffeomorphism f and its "inverse" g (in which g ( supposed to
recover s), and the second term (-p(-) promotes sparsity of ر see (2 3 19 49 50 for their respective ways of sparsity promotion. The most notable feature of this line of work lies in its relatively relaxed conditions on for establishing identifiability of the model. For instance, the work 2 showed that identifiability can be established without using auxiliary variables or statistical independence of s. On the other hand, J being sparse means that the observed features in x are only generated by subsets of s. This assumption is justifiable in some applications, e.g., object-centric image generation 22 23 24 but can be violated in other settings.
3 Proposed Approach: Diverse Influence Component Analysis
We present an alternative way to establish identifiability of the nonlinear mixture model in ( without relying on statistical assumptions of s. sparsity of J f. or auxiliary information. Our approach is built upon convex geometry of J y (*) and an underlying connection between NMMI and SMF models (5 29
particularly the recent advancements in (30] and variants in (3).
Preliminaries of Convex Geometry. Appendix A.2 we give brief introduction to the notions (e.g., convex hull, polar set, maximal volume inscribed ellipsoid (MVIE) that we use in our context.
3.1 Sufficiently Diverse Influence
Motivation. We are interested in understanding how the influences of x affect the identifiability of (I). beginning with a close examination of existing Jacobian assumptions. First, the sparsity assumptions on J in [2] 3 19 49 embody key principles in generative modeling and causal representation learning. For instance, the ICM principle [22] promotes generative models where latent
variables exert distinct influences on observed features 22) 3. Sparse Jacobian models share this view when the sparsity patterns of [J yla, for = 1,..., m (or [J f for j = I,..., d) differ sufficiently. However, sparsity is arguably relatively stringent assumption the hard constraint that each a; depends only on subset of $1....,5 Sa may not always hold in practice. Second, the IMA method [52) reflects the ICM principle differently: by enforcing column orthogonality in J. it assumes that the influences of and 8j on the change of æ are mutually perpendicular. This circumvents sparsity-based constraints and permits all x; to depend on all latent components. Nonetheless, IMA only guarantees local identifiability of [1] (see (44). while global identifiability remains unresolved. These observations motivate us to develop an alternative framework that flexibly models diverse interactions between and æ, while ensuring global identifiability of nonlinear mixtures.
3

---

(()
E1
81
22
v fute
mom (x)
82
(1)
81
Vf.(s)
T m
Figure 1: [Left] and V f((s) E B for d 2. Vi E [re]]: line thickness indicates the magnitude of influence of 8, on z,. [Middle] Condition I in Assumption 3.1 for d = 2: axes represent 0f./Os: and 0 I: the pink region is conv(Vfi(s),
fm (s)), the dashed purple ellipse is (( and the solid black diamond is B)o(e) [Right] Condition 2 in Assumption 3.1 shaded region shows conv VV fi(s), (m))))) dashed cllipse shows (B and solid rectangle shows (B) (=))))
Diverse Influence. To formally underpin the intuitive notion of "diverse influence" to an exact definition, we use convex geometry to characterize (s) Sm (m(()) (i.e., the rows of J f). Note that when the rows of J are sufficiently distinct, it means that s's changes render different variations of Æ1,.... Tmm In particular, we will exploit the cases where some observed dimensions x; = f(()) for E [m]] are affected by the changes of $1, .... Sd in a sufficiently different way.
Assumption 3.1 (Sufficiently Diverse Influence (SDI).. At S, there exists an s-dependent weighted L1 norm ball B1 (3 such that ff()), fm) E BT((()) In addition, the following two conditions hold:
((BT(()) S conv (V Vmm(s)) _ B| and
2. conv convt (V)i (8))** (d(E(()((*)))) extr(B
where conv {-} denotes the convex hull of set of vectors, and for given polytope P, E(P) is its MVIE, P stands for its polar set, extr(P) contains its extreme points, and bd(P) is the polytope's 's boundary.
The SDI condition describes the scattering geometry of (s), (m, (m).
which originates from the sufficiently scattered condition (SSC) in the matrix factorization literature. The SSC first appeared in identifiability analysis of nonnegative matrix matrix factorization (NMF) (26) and has various forms in the SMF literature; see [25] 27 36) Here, SDI takes the form of SSC from polytopic matrix factorization (PMF) 330; also 31 33
The key difference between SDI and SSC is that the former specifies the row scatternng pattern of a Jacobian J (8) at every 8 over a continuous manifold S, yet SSC only characterizes the latent factors of data matrix (e.g., W. H in X W H) that do not involve nonlinear functions or derivatives. But their geometries are essentially the same.
Geometry of SDL. Fig. [Middle] illustrates Condition I of Assumption 3.] for d = 2. One can see that SDI is about the spread of {V V. (8)} in an s-dependent weighted -nor ball B70(s) We note that (zo(() always exists, as long as [ (s), (s), fm [m)} are finite. Geometrically, Condition 1 requires that the gradient vectors' convex hull contains the MVIE of BBw(() Condition
2 is imposed on the polar sets conv CV (is),, ..., (m)) ((B))(())) and B1/ Bl/w(s) (note that [/w()))) weighted by (s., 1/ (s),,,,
is the polar of B)u(a)). The condition implies that conv (s)(s),..., ..., Vm (s))* touches the boundary of E(B))()))+ at the vertices of B1/ ll/w(e) which is vector of the form [=) ..... =wa(s) E Rd.. In the original domain, this means that the ellipsoid E {1370())) must touch the convex hull conv V f (s), V (s)} at the facets of gro(s) Fig. [Right] shows the polar set view of Fig. | [Middle]. Readers are referred to (30) for visualization of SSC under PMF, which further clarifies the connection between SSC and SDI.
Physical Meaning of SDI. SDI reflects how diversely affects Fı, .1...., I'- Under SDI, there exist some observed features positively influenced by 8j (i.e., dz./0s; > 0), while also some features negatively influenced by 8j (i.e., dx, /ds < 0). This pattern likely holds in many applications with high-dimensional data (i.e. m >> d). This because each z; can only either be positively or negatively influenced by sj. As m increases there would be more features that are affected differently by sy.

---

Note that the positive and negative influences need not be symmetric (i.e., 0z,/0s; # -dxy/00 is allowed), illustrated in Fig. _ [Middle].
In addition, SDI assumes that each latent component 85 dominantly influences at least two observed features, Fe and : here, "+" and "_" indicate that the features are positively and negatively affected by 81, respectively. other words, features satisfying the following should exist:
ix /0sk >> 0zz/Osj, Vj + ix and ix,,/0ss < dzi jōsj, Vj サ in.
(5)
Discussion. Beyond the basic geometric and physical interpretations, some additional remarks on other aspects of SDI
follows:
(i) Dependent 8 and Dense Jacobian Can Satisfy SDI. Notably, SDI does not impose statistical (c.g.. conditionally independent given auxiliary variables like in [10) In addition, SDI does not rely on the sparsity of y as in 19I. In fact, J y can be completely dense and at the same time satisfies SDI (see Fig. [Middle]]
assumptions
(ii) SDI Favors m > Cases. The SDI condition is arguably easier to satisfy when m >> d (which is justified in many applications, such as image/video generation [2] 40
53 It is evident that SDI fm are located at :.i (8)e1, ...., twa (8)e.. To see why SDI is in favor of larger m's, consider a case where the rows of J (s). i.e. (s). are drawn from a certain continuous distribution supported on BTo((), Then, large means that more realizations of (s('s's s are available. Therefore, for a fixed d, leads to ( fm (s)} increasingly covering more of BT((+) and eventually becoming 13((()) which ensures that SDI condition is met. A related note is that a matrix factor W E RR with larger m under fixed would have higher probabilities to satisfy SSC, which was formally studied in [54). using exactly the same insight.
requires at least m 2d, which corresponds to the case where J
oo
(iil) SDI Encodes Influence Variations. Lastly, the SDI condition allows the gradient pattern to vary from point to point on S: at different & E S, both the ball B7((() and the position pattern of V (s), v (s) in Bur(s) can be different, long the gradients are sufficiently spread out in different directions as in Assumption 3.1
3.2 Proposed Learning Criterion
the goal of Ceentiffabllityyguaratteee learning in this work is to find an invertible function (over the Ie manifold) f such that ~ = (x:) f fs) recovers & to a reasonable extent. practice, we use a neural network (supposedly universal function representer) Se as our learning function. To ensure the invertibility of fe I2 + m over
Similar to the NMMI literature, e.g., 3 12 19).
the manifold S C Rd, we use another neural network 96 R" + Rd and enforce
x = f= Yx X.
(6)
Note that if the above holds, both fo and 9 are invertible over the data- tatta-genrratig d-dimensional manifold. Eq. (6) nothing but stacked autoencoder, which by itself does not ensure identifiability of the model (1). To utilize SDI for establishing identifiability, we propose the following learning criterion, namely, Jacobian volume maximization (o-lo))
(J- VolMax)
maximize det (g Jo 1 (g (x))) 0.4 subject to: ر| (2)))) SC, Vi 1..... m,
del
(7a)
(7b)
x= f= Vx X
(7c)
The term log det (g) Js (x)) represents the squared volume of the convex hull spanned by the columns of J fo-
Using this criterion, the partial derivatives contained in J (3) (where = g (x)) are encouraged to scatter in space-reflecting the belief that the influences of 8 on different x, 's are diverse.
Identifiability Result. Under the model in (I) and Assumption 3.1
we show our main result:
5

---

Theorem 3.2 (Identifiability of J-VolMax), Denote any optimal solution of Problem 7 as (0, Φ). Assume f = fo and - universal function representers. Suppose the model in and Assumption 3.7 hold for every 8 E S. Then, we have 8 (ax) = 9 f(8) where
[ [ (3) pi(s)())) Vi E [d],
in which is a permulation of (1, ( dl and pt(.) R R is an invertible function.
Notice that we used the constraint || ( (x))).: 1..111 < C in where C > 0 is unknown. Un-
der SDI, the ground-truth Jacobian is bounded by (s)x Bl((s) with an unknown Bro(s) Nonetheless, using an arbitrary -nomm ball with radius C' 7 does not affect identifiability of the J- J-VolMax criterion-the learned & will have a scaling ambiguity anyway.
The proof of the theorem consists of three major steps. First, we recast the nonlinear identifiability problem J-VolMax into its first-order derivative domain as matrix- finding problem at each s E S. which is closely related to intermediate steps matrix factor identification under the PMF model [30]. Second, consequently, we utilize SDI and algebraic properties from volume maximization- to underpin identifiability under J-VolMax at each s up to an s-dependent permutation ambiguity. Third, we invoke continuity of f and its domain S to unify permutation ambiguity over the entire S. The details can be found in Appendix B
PMF
Finite-Sample SDI and Identifiability. In Theorem 3.2 an assumption is that every 8 in the continuous domain S has satisfy SDI, which could be relatively restrictive. To proceed, we show that, as f and the learned functions satisfy certain conditions, having finite number of & satisfying SDI suffices to establish identifiability up to bounded errors.
To see how we approach this, consider a finite set SN := {s(1), sf [N) with X' == fxx E X : fs). Vs E } such that Assumption 3.1 is satisfied at each of the N points Sx. Note that at each a ((m) the optimal encoder 9 recovers s(m) from the observation (n) up to permutation (s))
and an invertible dlement-wise map ((s(m)), i.e.
g(x (n) fī( ((s(()) (a.(") E XN=
which is direct result when using the J- JoVa learning criterion; see Lemma C.2 The following result shows that under certain regularity conditions, if the set SN contains samples that locate densely recover the ground-truth s up to the same ambiguities in Theorem 3.2] with a bounded error that decays as N grows.
enough in space, the learned encoder
Theorem 3.3 (Identifiability under Finite-sample SDI). Assume that there is a finite set SN := (s!1).... s(N) } with Xy := {x E X x = f()), Vs E &NN such that the Assumption 3.1 is satisfied at each of the N points in Sx- Let 9 E G be the optimal encoder by J-VolMax criterion 77. and O contains the parameters of the learned encoder and decoder. Further assume that the following regularity conditions hold:
1. The functions g = f and 96 are from classes G' and G, respectively, where g' S g.
2. The functions f 99 P are Lipschitz continuous with constants L,, Lg. La > 0.
3. There is a > 0 such that for any permutation matrix II T Pa and II + f((s(1) (from 99).
[g (m)) II 2 7, Vn E [N].
4. For N(m) = {s E S ||s s(n) ||| r(0) with p()) 2(T.y1g+pp)) the union of the neighborhoods, N = U.Y N(a is connected subset of S and V(s; O) is optimal for
any 8 E N.
5. The points s!
(N) E SN densely locate in S such that P(s P(sEN)/P(( ESSNN > Ginaz/ Gainn > 1,
(10)
where Gmin, Gmax are bi-Lipschitz constants of the Jacobian volume surrogate: for any parameters O₁ O₂ in (F,9) 9).
Gmin || O2l12 - (s: 11) - (ss; 02)l < Gmax |0 - O2l12, V S.
(11)

---

Then, g(x!")) (ppss()), Vn E for constant permutation matrix II E Pa. Furthermore, with probability at least 1 5.
Bx-p(()) [g(x) [Ip()))lal o
( ( +
(12)
where RN (9) is the empirical Rademacher complexity of the encoder class.
The theorem implies that if the number of {8 [g(") Jn=I satisfying SDI is sufficiently large, the identifi- cation error defined in (2) can be made arbitrarily small. Note that when G is not a universal function class that is, its capacity is controlled, e.g., through bounded norms or Lipschitz constants- empirical Rademacher complexity satisfies (g) = O(/VX) (55]. Consequently, with probability at least 1 -, the overall error rate is ((Vln118++////) = (P1//N). The detailed proof is provided in Appendix
In Theorem 3.3 Condition I means that the learning function class can faithfully represent g- Condition 2 requires that and the learned functions have bounded continuity constants, which is mild assumption as the learning functions are often represented using continuous function classes (e.g., neural networks). Condition assumes that at each point ((n) E SN. there is a unique optimal permutation between s(n) and ((x)))). That is, y = 0 holds for the optimal permutation, but an error of y > O occurs for all other permutations. Such condition naturally holds under sufficient continuity of the functions. Condition 4 translates to the fact that the points s ""......"** (NN) E Sy locate densely to each other so that their neighborhoods. A/(x) (with radius y(()) overlap. Intuitively, if each A/(n) has a unified permutation for all 8 within A7r),, such overlapping leads to the same permutation for the union of "s Condition 5 means that there are enough points g(1), ,.., s(n) and they are densely positioned in S, such that N covers sufficiently large fraction of S. This required fraction depends on the bi-L.isschtz constants with respect to θ of Jacobian volume surrogate V(s; O).
4 Related Works
IMA, Sparse Jacobian, and Object-Centric Learning. As mentioned in "Motivation" (see Sec. 3.1
IMA [20) and sparse Jacobian-bas -ased methods (c.g.. [2] 19 49 50) are conceptually related to
work. Additional remarks on connections between the SDI condition and IMA, Jacobian sparsity,
anchor features [49]. and object-centric methods
23 24 can be found in Appendix D
SSC and Matrix Factorization. Volume-regulaarzzaton approaches are popular in SMF models ii... X = W H with structural constraints (e.g., boundedness and nonnegativity) on W and/or H (27 33) where SSC-like conditions are imposed on the matrix factors ensure identifiability of W and H. The work (25) was the first to employ SSC defined over the nonnegative orthant (originated from (26) to show that latent-factor volume minimization identifies the corresponding SMF model. There, SSC requires that the columns/rows of factor matrix widely spread in the nonnegative orthant. This line of work later was generalized to SSC in different norm balls [30] 31]. The SDI condition can be viewed an extension of SSC in [30] (also see similar SSC in [31 33]
into the Jacobian domain over a continuous manifold. Although NMMI and SMF contexts differ substantially, some mathematical properties resulted from SSC (coupled with a allmmmmmximmiing criterion) in 30 provide key stepping stones for identifiability analysis in this work.
Maximum Likelihood Estimation (MLE) and InfoMax for ICA. Both InfoMax (56) and MLE (57 handle ICA via optimizing the Jacobian volume in their formulations. Nonetheless, the Jacobian volume arises in their learning criteria result of information-theoree based derivation (e.g., being surrogate of entropy). Model identifiability under nonlinear mixture models has not been established under these frameworks yet.
or statistical estimation-
Numerical Results
Implementation 1 Given L realizations of æ, i.c., (1) x2) (2)), use multi-layer percep-
trons (MLPs) to parametrize fo and Jg- We implement a regularized version of J-VoMMax in (7) and train for T iterations via warm-up heuristic with the number of warm-up epochs is T, < T. Our
Our implementation code is available here: https//gitthbbcomm//sggyyyaaaaa
7

---

loss function L, at the ith epoch is
min L. : : (11x(1) Sol--olllloo (e(9 -vaal() x Cval + Anorm x Crorm (0).
(13)
0.4
In the above, Cvol is defined as
Cvol := log det (J (g6 J 3 (g
(14)
and (t) := Aval min(t. (t, Tw): }; i.e., (t) (t) linearly increases from to a chosen Avol > 0 the Te
end of warm-up phase. i.e., Tw. Note that the explicit form of (14) might impose computational challenges when m and d are large. Hence, for high-dimensional datasets, one can use trace-based
surrogate of Cvul, reducing the per-iteration flops from (mm)) to O(m)): see Appendix E.
The term Crorm (t) implements the norm constraint in
7b via
Caorm (t) = 5 (0
if S Tw [Sotplus (g C} ift Tw
(15)
with Anorm > O. The element-wise function Softplus(z) In(1 + e*) R is used as smooth approximation of the hinge loss (see (58). This term serves as soft version of the norm constraint (7b) in J-VoMa that is, it penalizes large Lj norm values but does not put hard constraint on their upper bound; see (59].
After the warm-up period, C' set to the average of f (1 in the last 10 epochs, to avoid that the Cuorm term disproportionally skews the optimization process. More discussions on implementation can be found in Appendix E.1
1)l
Evaluation. Following standard practice in NMMI correlation coefficient (MC) and the mean coefficient determination R² over pairs of latent components, after fixing the permutation ambiguity via the Hungarian algorithm [60]. While MCC can only measure linear correlation, the R² score can measure nonlinear dependence. A perfect R² score means there is a mapping between the estimated latent component and the ground truth.
12 we calculate both the mean Pearson
Baselines. We the unregularized autoencoder (Base), autoencoder with the IMA contrast 20 52 (IMA), and the sparsity-regulan Xap >> 0 is used to promote Jacobian sparsity, in [19 (0). To present the best performance of all the baselines under the considered settings, we tune their hyperparameters in separately generated validation dataset with known ground-truth latent components. The hyperparameters Avol, Anorm. and Aap are chosen by grid search from {10-2, 10-,, 10-4 10-5]
loss (Sparse) as in (4). The L. -noom regularization with weight
on the same validation set.
5.1 Synthetic Data Experiments
Data Generation. We generate three types of mixtures test method. The generating process and ground-truth latent variables are unknown to our algorithms and the baselines. We use such controlled generation for constructing SDI-satisfying f's. In all simulations, neural networks with one hidden layer of 64 neurons to represent fo and 9s.
two fully-connected ReLU
Mixture A: We use linear mixture as the first checkpoint of our theory. Here, the latent components 8 are sampled from a normal distribution p(s) = A (0, E), where the covariance Σ drawn from a Wishart distribution W, (I, d). This way, the components of are dependent. The observed mixtures created by generating matrix A xd to form As. The matrix A can be generated to approximately satisfy the SDI condition by sampling m points in Rd randomly from an inflated Le
ball, and then projecting those points onto the chosen weighted norm ball B (*)( to create the m rows of A; see 30 for a similar way to generate matrices that approximate SSC.
Mixture B: On top of x = As as in Mixture A, we also apply an element-wise nonlinear distortion to create nonlinear mixtures, i.c., x = f(z) and [f() = a cos(2)) + 2;, where a UO0.5, 1.0). Notice that (s) = Diag(1 - sin( (A)) A under this construction. Therefore, with A approx- imately satisfying SDI and the nonlinear distortion weight a being sufficiently small, J y (s) also approximately satisfies the SDI.
Mixture C: This mixture is generated by æ f(s), where U(-1,1) and the nonlinear mixing ..., fm(()) E R.. The functions fx 1Rd + R, k 1,... m are MLPs function is f(-) (..,,)...,
8

---

Table 1: Nonlinear I² and MCC scores (mean 1 std., over 10 random trials).
Mixture A
Mixture
Mixture C
Model
m))
Rta
MCC
n²²
MCC
n²²
MCC
(2, 90)
0.16
0.93
0.97 + 0.06
0.03
0.13
0.97
(3, 40)
0.92 + 0.11
0.91 0.08
0.99 = 0.02
0.99 = 0.01
0.90
1:00=00 0.12
0.95 = 0.07
(4, 50)
0.98 0.06
0.98 = 0.04
0.92 = 0.10
0.95 = 0.07
0.12 0.87
0.92 = 0.07
(5, 60)
0.92 1: 0.10
0.93
0.09
0.89
1 0.13
0.93
0.08
0.87
0.11
0.93 0.06
(2, 90)
0.19
0.83 0.12
0.88
0.12
0.12
0.92 0.07
(3,40) 40)
0.71 + 0.15
0.78 0.13
0.72 = 0.11
0.80 = 0.09
00.11
0.83 0.07
Sparse
(4, 50)
0.65 0.13
0.72
0.71 + 0.10
0.81 = 0.06
0.12
0.79 0.07
(5,
0.63 :: 0.10
0.71 0.00
0.67 + 0.07
0.78 0.05
0.60 + 0.09
0.76 0.06
(2, 30)
0.88
0.11
0.81 0.17
0.88 0.11
0.77 0.12
0.87 0.08
(3,40)
0.60
0.07
0.68 + 0.12
0.80 0.09
0.14
0.77 0.09
(4, 50)
0.12
0.09
0.63 0.15
0.76
0.48
0.68 = 0.06
(5, 60)
0.42
0.53 0.07
0.54 0.06
0.04
0.47
0.66 0.04
(2, 30)
0.14
0.81 0.10
0.14 0.07
0.91 0.07
(3, 40)
0.70
0.83
0.69 + 0.13
0.09
0.67 107=000 0.07
0.81 0.04
INA
(4, 50)
0.10
0.07
0.08 + 0.09
0.81
0.56
0.74 0.07
(5.
0.66 + 0.05
0.80
0.63 = 0.10
0.79 = 0.06
0.53 0.08
0.72 0.05
Table 2: Nonlinear R² scores for different ratios m/d /d (mean + std., over 10 random trials).
(d, ml
DICA
IMA
Sparse
Base
(3, 40)
0.90 1 0.10
0.63
0.15
0.80 1 0.13
0.63
0.10
(3,30)
0.89 + 0.07
0.63
0.12
0.77 1 0.12
0.63
0.14
(3, 20)
0.82 + 0.17
0.60
- 0.12
0.74 + 0.13
0.60 = 0.10
10)
0.17
0.75
(3,5)
0.64
1 0.09
0.66 1 0.16
0.78
1 0.16
0.61 0.08
(3.3)
0.55
4: 0.07
0.60
0.09
0.58 1 0.16
0.09
whose layer weights are randomly generated. In addition, we pick fi, f2, J1m/21 and modify them as follows: a) randomly choose d elements from [d]; and b) add an input layer that down- scales the chosen d - 1 latent component Sj by 8, a BySBy where a, ~ U (0.0011, 0.002) and P(3, = 1) (3 = -) = 1/2. This way, the first half of Vf((s)'s create subset of gradients that are dominated by one latent component (cf. "Physical Meaning of SDI" in Section 3.1
We should mention that, unlike Mixtures A and B, we have less control generating SDI-satisfying mixtures under Mixture C. Hence, this type of nonlinear mixture can be used to test the model robustness of approach.
Results. Table | shows both the MCC and the nonlinear R scores of all four methods, namely, DICA. IMA, Sparse, and Base, Mixtures A, B, C. We observe that for Mixtures A and B, both scores attained by DICA are mostly above 0.92 (except for R² under (d, m) = (5, 60)). contrast, IMA, Sparse, and Base do not perform well. Importantly, the performance of DICA does not significantly degrade as m and d grows, but the baselines appear to struggle under larger m's and d's. For the more challenging Mixture C, DICA's performance scores still have substantial margin over the baselines, and the scores remain largely the same when (d. m) varies.
Ablation on m/d. We examine the performance of DICA when varying the ratio m/d. Per our discussion (cf. See. 3. SDI favors m > d cases. As the ratio m m/d approaches 2, SDI less likely to be satisfied. In Table 2 we use Mixture C to test the performance of DICA where we fix d = 3 and vary m. Consistent with our analysis, the performance of DICA deteriorates as m approaches 2d. For 2d, the performance of DICA is similar to vanilla autoencoder showiin that the effect of volume regularization vanishes in these cases. This result confirms that DICA is suitable for learning low-dimensional latent components from high-dimensional data, e.g., image representation learning.
5.2 Inferring Transcription Factor Activities from Single-cell Gene Expressions
In this experiment, we apply J- VolMxx to inferring transcription factors (TFs) activities in single- cell transcriptomics analysis 61). Specifically, we employ J- J-VMa to infer the ground-truth mRNA concentrations of TFs from gene expression data. In this context, our single-cell gene expressions are from a bio-realistic generator, namely, SERGIO ; see the use of SERGIO the NMMI literature 41 63). The mRNA concentration levels R of the TFs are governed from chemical Langevin

---

R2 Motras x ITPs vs Latentsi
R2 Hottik (TFS Laterie:
R2 Hotite (TPs Laterts:
3:45
0.00
0:319
0.36
1ifedd
0,306
04
0.0011
8211
641
S:1
E27
646
2:11
6871
Libk
Bear
B:4
5:26
2014
6/6
Q44
02
B:2
0.11
222
60657
9:27
0014
8800
E9330
8327
00044
0.06
C33
0:035
0.
134
000
Slork SY600
Litk Elmitakra
Liwis dinsakara
(a) DICA (mean R² 1)
(b) Sparse (mean R² 0.4544)
(c) Base (mean R x 0.3381)
Figure 2: Heatmap of R² scores between estimated components and ground-truth mRNA concentrations of TFs.
equation, and the gene expressions (i.e., samples of æ E I.' are generated by a gene regulatory
mechanism f _ + R applied onto
We use the TRRUST dataset of TF-gene pairs of mouse 64). It is manually curated dataset that includes high-confidence interactions between TFs and their target genes. These interactions are either activating or repressing. All entries are supported by experimental evidence from multiple published studies. We sample sub-network from the large gene regulatory network provided by TRRUST, via which m = 178 genes are governed by d master regulator TFs. More details of the data generation process using SERGIO is in Appendix E.3
Fig. 2 shows average R²2 scores (from 10 trials) for each $1,..., SG up best permutation, as obtained by DICA, Sparse, and Base. One can see that DICA successfully infers the ground-truth mRNA concentration level ,1,,.. ..., 85 of each TF (up to a permutation and invertible mapping), but Sparse and Base are not able to attain reasonable R²2 scores. Furthermore, the proposed method can disentangle the influences of different TFs. Specifically, each learned latent component only matched a unique ground- truth TF with R² score x 1, yet the R² score is approximately O when matching with other TFs. In contrast, Base could not disentangle the TFs all. Similarly, Sparse could only mildly disentangle two TFs (i.e., the 3th and the 5th TFs). Additionally, the MCC score of DI CA whereas Sparse and Base output MCC 0.6635 and MCC 0.5721, respectively.
reaches xs
Some remarks regarding this experiment is as follows. Note that the data generation process follows biology-driven mechanisms in SERGIO, and thus we do have way to enforce SDL, However, as the TFs are believed to have diverse influences on the gene expressions (e.g.. due to the activating/repressing effects on genes by the TFs), it is reasonable assume that SDI approximately holds in this application-wwe which perhaps explains why DICA works well. On the other hand, some TFs may influence many gene expressions ((65), and many coos-interacttios between TFs and genes exist ([66)) Hence, the assumption that the Jacobian of f is strictly sparse might be stringent. Therefore, the sparse Jacobian approach may not be a good fit, as its performance indicates.
which
6 Conclusion
We introduced DICA, a new paradigm for nonlinear mixture learning with identifiability guarantees. Via the insight of convex geometry, we showed that, as long as the latent components have sufficiently different influences on the observed features so that geometric condition, namely, the SDI condition, is satisfied, the mixture model is identifiable via fitting the generative model and maximizing the Jacobian of the learning function simultaneously. This J-VolMax criterion ensures identifiability with- out relying on many assumptions in the NMMI literature, e.g., availability of auxiliary information, independence of latent components, or the sparsity of the mixing function's 's Jacobian, thus offering a valuable alternative existing NMMI methods. The experiments supported our theory.
Limitations and Future Works. We noticed number of limitations. First, the DICA framework has a log det term to compute, which poses challenging optimization problem. This is particularly hard as this term has to be evaluated at every realization of s. How to optimize the J-Vola criterion efficiently is an interesting yet challenging direction to consider. Second, the identifiability analysis was done under ideal conditions where noise absent, one has access to unlimited samples, and the learner class is assumed to perfectly include the target function. While Mox is empirically shown to be quite robust in experiments, a more thorough identifiability analysis under practical conditions would close the gap. e.g., by analyzing sample complexity under noise and model mismatches.
10

---

Acknowledgment: This work is supported in part by the National Science Foundation (NSF) CAREER Award ECCS-2144889. The authors would like to thank Sagar Shrestha for insightful discussions on the identifiability analysis, and the anonymous reviewers for constructive feedbacks.
References
[1] Ilyes Khemakhem et al. "Variational Autoencoders and Nonlinear ICA: A Unifying Frame- work". In: Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics. Vol. 108. Proceedings of Machine Learning Research. PMLR, 2020, pp. 2207-
2217.
[2] Jack Brady et al. "Provably Learning Object-Centric Representations" In: Proceedings of the 40th International Conference on Machine Learning. Vol. 202. Proceedings of Machine
Learning Research. PMLR, 2023, pp. 3038-3062.
[3] Yujia Zheng and Kun Zhang. "Generalizing Nonlinear ICA Beyond Structural Sparsity". In: Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 13326-13355.
[4] Irina Higgins et "beta- VAE: Learning basic visual concepts with constrained variational framework." In: International Conference on Learning Representations (2017).
[5] Bernhard Schölkopf et al. "Toward Causal Representation Learning". In: Proceedings of the
IEEE 109.5 (2021). pp. 612-634.
[6] Julius von Kügelgen. "Identifiable Causal Representation Learning: Unsupervised, Multi-Viee and Multi-Environmenn Environment". PhD thesis. University of Cambridge, 2023.
[7] Qi Lyu et al. "Understanding Latent Correlation- -Based Multiview Learning and Self- Supervision: An Identifiability Perspective". In: International Conference Learning Repre- sentations. 2022.
[8] Julius von Kügelgen et al. "Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style". In: Advances in Neural Information Processing Systems. Vol. 34. 2021, pp. 16451-16467.
[9] Aapo Hyvärinen and Petteri Pajunen. "Nonlinear independent component analysis: Existence and uniqueness results", In: Neural Networks 12.3 (1999), pp. 429-439.
[10] Aapo Hyvarinen and Hiroshi Morioka. "Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA". In: Advances in Neural Information Processing Systems. Vol. 29. 2016.
[11] Aapo Hyvarinen and Hiroshi Morioka. "Nonlinear ICA of Temporally Dependent Stationary Sources". In: Proceedings of the 20th International Conference on Artificial Intelligncee and Statistics. Vol. 54. Proceedings of Machine Learning Research. PMLR, 2017, pp- 460-469 660.469.
[12] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. "Nonlinear ICA Using Auxiliary Vari- ables and Generalized Contrastive Learning". In: Proceedings of the "wenty-Secont Interna- tional Conference on Artificial Intelligence and Statistics. Vol. 89. Proceedings of Machine Learning Research. PMLR, 2019, pp. 859-868.
[13] S. Achard and C. Jutten. "Identifiability of Post-Nonlinear Mixtures". In: JEEE Signal Pro- cessing Letters 12.5 (2005), pp. 423-426.
[14] Qi Lyu and Xiao Fu. "Provable Subspace Identification Under Post-Nonlinear Mixtures". In: Advances in Neural Information Processing Systems. Vol. 35. 2022, pp. 1554-1567.
[15] Qi Lyu and Xiao Fu. "Identifiability -Guaaantee Simplex-Structuue Structured Post-Nenl Mixture Learning via Autoencoder".. In: IEEE Transactions on Signal Processing 69 (2021).
[16] Andreas Ziehe et al. "Blind Separation of Post-nonlineas Mixtures using Linearizing Trans- formations and Temporal Decorrelation". In: Journal of Muchine Learning Research (2003), pp. 1319-1338.
[17] A. Hyvarinen and P. Pajunen. "On existence and uniqueness of solutions in nonlinear in- dependent component analysis". In: 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence. Vol. 2. 1998,
pp. 1350-1355.
[18] Bohdan Kivva et al. "Identifiability of deep generative models without auxiliary information". In: Advances in Neural Information Processing Systems. Vol. 35. 2022, pp. 15687-15777 15701.
11

---

[19] Yujia Zheng. Ignavier Ng, and Kun Zhang. "On the Identifiability of Nonlinear ICA: Spar- sity and Beyond"" In: Advances in Neural Information Processing Systems. Vol. 35. 2022. pp. 16411-16422.
[20] Luigi Gresele et al. "Independent mechanism analysis, new concept?" In: Advances in Neural Information Processing Systems. Vol. 34. 2021, pp. 28233-288 28248.
[21] Sebastien Lachapelle et al. "Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA". In: Proceedings of the First Conference on Causal Learning and Reasoning. Vol. 177. Proceedings of Machine Learning Research. PMLR, 2022, pp. 428-484. [22] Bernhard Schölkopf, Jonas Peters, and Dominik Janzing. Elements of Causal Inference. Adap- tive Computation and Machine Learning series. MIT Press, 2017.
[23] Sébastien Lachapelle et al. "Additive Decoders for Latent Variables Identification and Cartesian- Product Extrapolation" In: Advances in Neural Information Processing Systems. Vol. 36. 2023,
pp. 251
[24] Jack Brady et al. "Interaction Asymmetry: A General Principle for Learning Composable Abstractions". " In: The Thirteenth International Conference on Learning Representations. 2025.
[25] Xiao Fu et al. "Blind Separation of Covariance Domain". In: IEEE Transactions Signal Processing 63.9 (2015), pp. 2306-2320.
Sources: Exploiting Convex Geometry in
[26] Kejun Huang, Nicholas D. Sidiropoulos, and Ananthram Swami. "Non-Negative Matrix Factorization Revisited: Uniqueness and Algorithm for Symmetric Decomposition". In: IEEE Transactions on Signal Processing 62.1 1 (2014), pp. 211-224.
[27] Xiao Fu, Kejun Huang, and Nicholas D. Sidiropoulos. "On Identifiability of Nonnegative Matrix Factorization". In: IEEE Signal Processing Letters 25.3 (2018), pp. 328-332.
[28] Nicolas Gillis. Nonnegutive Matrix Factorization. Society for Industrial and Applied Mathe- matics, Jan. 2020. ISBN: 97816 1976410.
[29] Xiao Fu et al. "Nonnegative Matrix Factorization for Signal and Data Analytics: Identifiability. Algorithms, and Applications". In: IEEE Signal Processing Magazine 36.2 (2019), pp- 59-80. [30] Gokcan Tatli and Alper T. Erdogan. "Polytopic Matrix Factorization: Determinant Maximiza- tion Based Criterion and Identifiability.. In: IEEE Transactions on Signal Processing 69 (2021), pp- 5431-5447.
[31] Jingzhou Hu and Kejun Huang. "Global Identifiability of LI-based Dictionary Learning via Matrix Volume Optimization". In: Advances in Neural Information Processing Systems. Vol. 36. 2023, pp. 36165-3613 36186.
[32] Jingzhou Hu and Kejun Huang. "Identifiable Bounded Component Analysis Via Minimum Volume Enclosing Parallelotope". In: 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2023.
[33] Yuchen Sun and Kejun Huang. "Global Identifiability of Overcomplete Dictionary Learning via LI and Volume Minimization". In: The Thirteenth International Conference on Learning Representations. 2025.
[34] Xiao Fu et al. "Robust Volume Minimization-Basse Matrix Factorization for Remote Sens- ing and Document Clustering". In: IEEE Transactions Signal Processing 64.23 (2016), pp. 6254-6268.
[35] Lidan Miao and Hairong Qi. "Endmember extraction from highly mixed data using minimum volume constrained nonnegative matrix factorization". In: IEEE Transactions on Geoscience and Remote Sensing 45.3 (2007), pp. 765-777
[36]
Chia-Hsiang Lin et al. "Maximum Volume Inscribed Ellipsoid: A New Simplex-Structure Matrix Factorization Framework via Facet Enumeration and Convex Optimization". In: SIAM Journal on Imaging Sciences 11.2 (2018), pp. 1651-1679.
[37] Christian Jutten and Jeanny Herault. "Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture". " In: Signal Processing 24.1 (1991), pp. 1-10.
[38] Pierre Comon. "Independent component analysis, A new concept?" In: Signal Processing 36.3 (1994), pp. 287-314, 14, ISSN: 0165-1684.
[39] Séhastien Lachapelle et al. Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies. 2024. arXiv: 2401 04890
[40] David A. Klindt et al. "Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding". In: International Conference on Learning Representations. 2021.
12

---

[41] Hiroshi Morioka and Aapo Hyvarinen, "Causal Representation Learning Made Identifiable by Grouping of Observational Variables". In: Proceedings of the 41st International Conference on Machine Learning. Vol. 235. Proceedings of Machine Learning Research. PMLR, 2024,
pp. 36249-3625
[42] Luigi Gresele et al. "The Incomplete Rosetta Stone problem: Identifiability results for Multi- view Nonlinear ICA". In: Proceedings of The 35th Uncertainty in Artificial Intelligence Conference. Proceedings of Machine Learning Research. PMLR, 2020.
[43] Sagar Shrestha and Xiao Fu. "Towards Identifiable Unsupervised Domain Translation: A Diversified Distribution Matching Approach", In: The Twelfth International Conference on Learning Representations. 2024.
[44] Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. "Function Classes for Identifiable Nonlinear Independent Component Analysis". In: Advances in Neural Information Processing Systems. Vol. 35. 2022, pp. 16946-16961.
[45] Daniella Horan, Eitan Richardson, and Yair Weiss. "When Is Unsupervised Disentanglement Possible?" In: Advances in Neural Information Processing Systems. 2021, pp. 5150-5161.
[46] Kun Zhang and Laiwan Chan. "Minimal Nonlinear Distortion Principle for Nonlinear In- dependent Component Analysis". In: Journal of Machine Learning Research 9.81 (2008),
pp. 2455-2487.
[47] A. Taleb and C. Jutten. "Source Separation in Post-nonlinear Mixtures", In: IEEE Transactions on Signal Processing 47.10 (1999), pp. 2807-2820.
[48] Avinash Kori et al. "Identifiable Object- Centric Representation Learning via Probabilistic Slot Attention". In: Advances in Neural Information Processing Systems. Vol. 37. 2024, pp. 93300- 93335.
[49] Gemma Elyse Moran et al. "Identifiable Deep Generative Models via Sparse Decoding". In: Transactions on Machine Learning Research (2022). ISSN: 2835-8856.
[50] Travers Rhodes and Daniel Lee. "Local Disentanglement in Variational Auto-Encoders Using Jacobian L1 Regularization", In: Advances in Neural Information Processing Systems. Vol. 34.
2021. pp. 2770-22779..
[51] William Peebles "The Hessian Penalty: Weak Prior for Unsupervised Disentanglement".
In: Proceedings of European Conference on Computer Vision (ECCV). 2020.
[52] Shubhangi Ghosh et al. "Independent Mechanism Analysis and the Manifold Hypothesis". In: Causal Representation Learning Workshop at NeurIPS 2023. 2023.
[53] Francesco Locatello et al. "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations" In: Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR, 2019, pp. 4114-4124.
[54] Shahana Ibrahim et al. "Crowdsourcing via Pairwise Co-occurrences: Identifiability and Algorithms". In: Advances in Neural Information Processing Systems. 2019.
[55] Peter Bartlett and Shahar Mendelson. "Rademacher and gaussian complexities: Risk bounds and structural results". In: Journal of machine learning research 3.Nov (2002), pp. 463-482.
[56] Anthony J. Bell and Terrence J. Sejnowski. " "n -atitttioonMaiiizzattion Approach to Blind Separation and Blind Deconvolution". In: Neural Computation 7.6 (1995), pp. 1129-1159. ISSN: 1530-888X.
[57] J.-F. Cardoso. "Blind signal separation: statistical principles". In: Proceedings of the IEEE 86.10 (1998), pp. 2009-2025.
[58] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. "Deep Sparse Rectifier Neural Networks". In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. Vol. 15. Proceedings of Machine Learning Research. PMLR, 201 1, pp. 315-323,
[59] Jose M. Bioucas-Dira "A variable splitting augmented Lagrangian approach to linear spectral unmixing". In: 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. 2009.
[60] H. W. Kuhn. "The Hungarian method for the assignment problem". In: Naval Research Logistics Quarterly 2.1-2 (Mar. 1955), pp. 83-97.
[61] Dennis Hecker et al. "Computational tools for inferring transcription factor activity". In: Proteomics 23 (2023). ISSN: 1615-9861.
[62] Payam Dibaeinia and Saurabh Sinha. "SERGIO: A Single-Cell Expression Simulator Guided
by Gene Regulatory Networks". In: Cell Systems 11.3 (2020), 252-27 1.ell I. ISSN: 4405-4712.
13

---

[63] Hiroshi Morioka and Aapo Hyvarinen. cinnttaivvyy-contrustivee learning: Combining causal discovery and representation learning for multimodal data". In: Proceedings of The 26th International Conference on Artificial Intelligence and Statistics. Vol. 206. Proceedings of Machine Learning Research. PMLR, 2023, pp- 3399-3426.
[64] Heonjong Han et al. "TRRUST v2: an expanded reference database of human and mouse transcriptional regulatory interactions". In: Nucleic Acids Research 46.D1 (2017).
[65] Samuel A. Lambert et al. "The Human Transcription Factors". In: Cell 172.4 (2018), pp. 650-
665. ISSN; 0092-8674.
[66] Tamar Friedlander et al. "Intrinsic limits to gene regulation by global crosstalk". In: Nature Communications (2016). ISSN: 2041-1723.
[67]
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[68] Arne Brondsted. An Introduction to Convex Polytopes. Graduate Texts in Mathematics. Springer, 2012.
[69]
Loring W Tu. An Introduction to Manifolds. 2nd ed. Universitext. Springer, 2010.
[70] Shai Shalev-Shwarzz and Shai Ben-David. Understanding machine learning. Cambridge University Press, 2014.
[71] Yuxiang Wei et al. "Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation". In: Proceedings of the IEEE/CVE International Conference on Computer
Vision (ICCV). 2021. 2021.
[72] M. Berman al. "ICE: statistical approach identifying endmembers in hyperspectral images". In: IEEE Transactions on Geoscience and Remote Sensing (2004), pp. 2085-2095.
[73] Luigi Gresele et al. "Relative gradient optimization of the Jacobian term in unsupervised deep
learning". In: Advances in Neural Information Processing Systems. Vol. 33. 2020.
[74] Kevin Murphy. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.
[75] Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". In:
International Conference for Learning Representations. 2015.
[76] Kaiming He et "Delving Deep into Rectifiers: Surpassing Human-Leve Performance on ImageNet Classification". In: 2015 IEEE International Conference on Computer Vision (ICCV). 2015, pp. 1026-1034.
[77] Y. LeCun et al. "Gradient-based learning applied to document recognition". In: Proceedings of the IEEE 86.11 6. (1998), pp. 2278-2324.
14

---

Supplementary Material of "Diverse Influence Component Analysis: A Geometric Approach to Nonlinear Mixture Identifiability"
A Preliminaries
A.1 Notation
For m E N, [m] a shorthand for (1,
I, and X denotes scalar, vector, and matrix, respectively.
Xij. [ and vi,j denotes the entry ith row and jth column of X. X and X denotes the ith row and jth column of X.
For a vector, || Il denotes Lp vector norm; for a matrix, || llp denotes the entry-wise Lp matrix norm and denotes the spectral norm.
The Jacobian of function f Id R at point is matrix y of partial derivatives such that [J ( 0f./0e; Vf(s) [Pf(s//as ... af(e)/oaa] denotes the gradient of scalar function f I2d I; hence, [ Vf((s)
convf.} denotes the convex hull of set of vectors.
(R) being the unweighted no-norm ball with radius R > 0, and Bp is shorthand for the unweighted unit -n-norm ball (i.e., 1).
BBt((a) {y E Rd Ei wn (s)
yyl S 1} an Lornr ball weighted by
1/w1 1/w](s),.... (1))))..1//w(()) > 0.
E(P) is the maximum volume inscribed ellipsoid (MVIE) of an origin-centered convex polytope P.
Polar of set C C* fx Fd x S 1, by e C}.
bd(P) is the boundary of set P.
extr(P (P) is the set of extreme points (i.e., vertices) of polytope P.
Pa the set of d x permutation matrices.
e क is shorthand for the vector of parameters of parametrized encoder fø and decoder -
is shorthand for the log-determinant of Jacobian matrix of θ = 0, p) evaluated at æ f (s); this is used as the Jacobian volume surrogate in J-Vol objective (7a).
Vss;5) log JJ (gg) (g
A.2 Background on Convex Geometry
In this section, we briefly introduce the key notions in convex geometry that will be used in the paper. Convex Hull. Given set dR.d its convex hull conv(S) is the set of all convex combinations of vectors in S, i.e.,
m Σ
m
conv conv(s)
og C; S,a, 2
Σ o =1
Maximum Volume Inscribed Ellipsoid (MVIE). An MVIE (sometimes called John ellipsoid) of a bounded convex set C is the ellipsoid of maximum volume that lies inside C, which can be represented
E(C) = ( (Bu + d: ||u||2 1)
for B E dxd d E Rd, and B is symmetric positive-definite matrix (i.e., B E S. E(C) can be obtained from solving
log det. B sup Ic(Bu + d) 50,
BeSd deRd
15

---

with (-)- being the indicator function of whether point is in C. Note that every non non-empty bounded convex set has an unique MVIE. See [67, Section 8.4.2] for more details. Note that the MVIE has the following linear scaling property:
Proposition A.1. IfT IR + R is an invertible linear map and non-empty compact convex set, then the MVIE of (C) is the image under T of the MVIE of C, i.e.
C(T(C))=T(E(C)
Proof. See 67 Section 8.4.3].
In the following, we give examples of MVIEs several sets that are relevant to our context.
Proposition A.2. The followings are true about a convex set and its MVIE:
1. The MVIE of an Lı-oorm ball B, (R) in I& is {x ||0|| S va ):
2. The MVIE of Loo-nomm ball 30 (R) in TRd is (xc ||0||| R):
3. The MVIE of an L2-norm ball (R) is itself.
Polytope. Intuitively, a polytope P is bounded geometric object with flat sides (called facets). A polytope P containing the origin can be represented either by intersections of f half-spaces as
P = fx E Id a. x S 1, I.... f}.
or by convex hull of & vertices as
P conv((u Uz
For examples, Ly-norm ball B, and simplex A = f0 E Rd xi 2 0, -i 24 = 1} are polytopes.
Polar Set. The polar a set C Rd is
=== xT < 1, Vy EC.
There some interesting properties of polar sets: Proposition A.3. For two sets A, B C Rd and a > 0, the followings holds:
1. A C B implies 8 A",
(AUB)* = A" n n",
(a:A)* = 1 A"
We give examples of polar sets of several sets that are related to our context:
Proposition A.4. The followings are true about set and its polar set:
1. Polar set of a weighted Li-norm ball BT = ty E RRd Eim " lanl S 1} is the reciprocally
weighted Loo norm ball B {y I2d max:= (ww/ya S 1}, and vice versa;
Polar set of a L2-norm ball (R) with radius R > is a L2-norm ball (2()) with radius T and vice versa.
For the details of Proposition A.3 and Proposition A.4 refer the readers 68 Chapter 2].
B Proof of Theorem 3.2
The proof consists two parts. In the first part, i.e., Lemma B.2 we show that any optimal solution of J-VolMax problem (7) must identify ground-truth latent vector S, up to s-dependent permutation and invertible component-ve leverage the continuity and full-rank structure of the Jacobian matrices at different to argue that the permutation of latent components should be identical for all s E S, instead of being s-dependent as initially shown in Lemma B.2 That gives the desired identifiability result.
transformation. Then, in the second part, namely. Theorem we
Before we begin the proof, we invoke the following lemma from (30
16

---

Lemma B.1. Ler H Rdxd be a matrix satisfying
H uul2 S vd, Vu E extr(Bxa).
Then, det H < 1, with equality occurs if and only if H is a real orthogonal matrix.
Proof. See
30
Theorem 3].
Lemma B.2. Denote any optimal solution of Problem 7 as (0. 2). Assume f fa and 9% universal function representers. Suppose the model in ① and Assumption 3. hold every point eeS. S. Then, the Jacobian at the point 8 & of the function h" f will satisfy
D(s)
where ((s) is an invertible diagonal matrix with [D(s)],, dependent on i-th component s; only, and (s)
(s) is permutation matrix dependent on the point 8, almost everywhere.
Proof. Define a differentiable function h S + S as a composition h g Our ultimate goal is to show that the Jacobian (s) has permutation-like support structure, which is equivalent to the result statement.
We begin our analysis by considering data point a = f(() for s E S. Let 3 = g(z). By definition of function h,
f(8) x x= = f = fū(((s)))
(16)
Then, by chain rule,
( =
= J (8)(())
(17)
We now argue that (s) a full rank matrix. By definition of h.
(x(8) n
(18)
By definition of the ground-truth mixing process, f is an injective function, implying that J (s) is full rank. We now investigate Jag (x). Due to the data-fitting constraint = ==fa (g(9)(x)) and the chain rule, have
J (8) J9) (z)
(19)
Note that Jz (8) is full rank, due the optimality of the problem 7 Therefore, Jap. (ar) must also be full rank. As a result, (s) is full rank, and hence indeed locally invertible function at 8, by the inverse function theorem 699 Theorem 6.26).
Now, we can rewrite 17 as
(J) Jy(s)
= Js (z)
(20)
By the inverse function theorem
69 Theorem 6.26] and the chain rule, also have (s)
J Jn-1 (8), which results in
(J h (3)) J (s) J T
(21)
We note that there is an intrinsic scaling ambiguity of the Jacobian matrices here: with an diagonal matrix D (s) = diag(1/ (aag(l/w)) (s), s/W)(s)) whose entries are dependent on the unknown w(s). we have
f (8) T (8)) (.))
(22)
We use the shorthand 2; == Dw, u ( is matrix whose columns lie inside the unit L -nor ball B, i.c.
which is the i-th column of D (s) Notice that
...... E B
(23)
17

---

By combining Proposition A.1 and Assumption 3.1 we have rescaled version of the SDI condition:
B2 C conv 3m) C B1, and
(24)
convi (21, 2m (bd((B((V))) extr(B.o).
(25)
To proceed, let H (J))
D (s) To show that Jn (s) has the support structure of permutation matrix as desired, we can prove that such structure occurs in H. Now, we consider the constraint 7b
JJ s. - C, Vi E [m].
(26)
Combining 26 with the equation 22 we have
1 H2,11 S 1, Vi = 1...., m.
(27)
Observe that (27 can be further rewritten as the following set of 2d explicit inequalities over all possible signs of each entry of g Hzz
( HHHue u) S V. E {+1]d.
(28)
Note that (=1]² exor( This allows reveal from 28 that
T H u) S 1, Vu E extr (Bx).
(29)
Hence, by definition of the polar set of polytope, we have
H u E conv 3m) )", Vu (Bc))
(30)
To proceed, recall from the rescaled SDI assumption in (24) that B2( (2) C conv (2..... mm By Proposition A.3 of polar sets,
conv 3m}² C B2(vd).
(31)
Since 1 H E conv 3m and conv Zm 3m}
C B2(vd), we have H T u
e
B2(vd), i.e ! ulll V for any extr(Bxx). By Lemma B.1
that
det H S.
(32)
with equality holding if and only if 1 H is orthogonal matrix. Here, we note that the J- VolMxx criterion would lead to an H with maximal determinant, so then det H C.
Since H is an orthogonal matrix and ||u/|2 Vd for any u extr(B.o), we have j u (d(B)vv)), Also, recall C H u E conv }m) Hence, H T u E conv 2m) |*' n
bd(B2(Vd) Since
convf21, 2m 2m ('dd((BB(VV)) = extr(Boo),
(33)
we have
C
1 . T cxtr( Yu E cxor
(34)
which implies
H 1, Vi 1,....d d.
(35)
Hence, det H = if and only if both 1 H.illl = 1 and C H is an orthogonal matrix. Then, we are able conclude that
det H C
(36)
and only if & H is signed permutation matrix. Equivalently,
log det H < log C
(37)
18

---

with equality holds if and only if 1 H is signed permutation matrix. The arguments in Eqs. 226 from analytical tools first developed in the PMF work 30 Similar steps were used [3 for other models as well.
33
Suppose that there exists optimal solution (0, ()) with estimated source g such that the mapping h - f is not a composition of permutation and an element- invertible transformation with strictly positive probability, i.e., for H (Jx-188) (8)) (8) we have
P(log det H| < log C) > 0,
(38)
where the probability is over Ps. This implies
Ellog det
(3) J (3)]
(39)
Ellog det (J,
(8) J
(40)
Ellog dot 4
(8) D [y
(3))
(41)
Ellog det H (8) (
(42)
E[log(dct
det D (() J
(43)
E[2 log det H + dogget (s) JJ()(()))(())
(44)
(a)
(2 |
det H + det (o) u ds)
(45)
+
p(s)[2log det H +og J (s))
(46)
AES
(6)
p(s) log C' log ( D (s))
(47)
NEA
+
1os) C (olgoooee J D (s))
(48)
BESIAA
p(s)[21og C.
, D (s))]ds
(49)
JAES
E[2 log C do (s))
(50)
Ellog(C2 (()
J(s)
(51)
where (a) and (b) involve two sets A {s log del H < C} and S A = {s log det H = C).
Consider an invertible continuous element-wise function h S + S with h (8) E S, h()) = 8, and J i (8) Jn (s)) CDD,..s) This function h merely rescales and maps each latent component by an element-wise invertible transformation. Define
f(3) f(h(3)),
(52)
whose Jacobian
(53)
We show that f(8) is feasible solution to J-VooMaM i.c. satisfying (7b) and 7c
First, note that
|| (ICC))))))]))
(54)
Second, observe that
f(8) f(h f f(h(h (s))) f(s)
(55)
Hence, there exists feasible solution x f() VolMMax such that
Ellog f [3) _ (3)) Eloog det J ( (B)]
(56)
This contradicts the optimality of (0, )) to VolMax problem in 7 We hence conclude that any optimal solution (0*, o of would have the composite function h I satisfying
(s) = D (s)) a.c.,
(57)
where (s)) is an invertible diagonal matrix such that with the ith diagonal entries dependent on s;
due to the definition of Jacobian matrix, and (T(s) is permutation matrix that depends _
19

---

Theorem 3.2 (Identifiability of J-VolMax), Denote any optimal solution of Problem 7 as (0, Φ). Assume f = fo and - universal function representers. Suppose the model in and Assumption 3.7 hold for every E S. Then, we have 8 (æ)) = 9 f(8) where
[ g (2) pi pi(s)(()) Vi E [d],
in which is a permutation of ( d] and pe()) R R is an invertible function.
Define differentiable function h' S - composition = 9 f. By Lemma B.2 we have that the Jacobian at the point 8 E S of the function h" 9 o f will satisfy
Proof of Theorem 3.2
(s) = ((s)
(58)
where ((s) is an invertible diagonal matrix with [s)),, dependent on i-th component 8; only, and (I(s) is a permutation matrix dependent on the point almost everywhere.
First, want leverage continuity of h" to show that (n-(s) (I () actually holds everywhere on S. This is because if there exists & E such that (3) ) + D(3) then due to the full-rank property of (s), (8), there exist j E [d] such that the column [J A- ($)]. has least two non-zero elements. However, the continuity of the Jacobian J h. implies that there exists an open neighhorhood of where the column [J (s)].s has least two non-zero elements. This contradicts the fact that J (8) = ((s) ((s)II(8)) almost everywhere.
Next, it remains to show that (s)s) is constant (say II E Pa) for all s E S. The reason is that if the non-zero elements (I) switched locations, then there would exist a point in 3 where Jn (B) is singular, which contradicts the full-rank property of Jn (5). Hence, II(() II, Vs E S; see similar argument in [2 Theorem 1].
Finally, let π denote the permutation of .......d) corresponding to the permutation matrix II, i.e., let r = [I, dj then (i) = Hrr Then, Jn (s) = (D(s) II implies that Th; (s) Os, (s
03, = 0, şi ())(1)
+ and
(()())
Hence, there exist scalar functions Pı..... Pd such that
$ (8)()())
Note that Pi....: Pa are invertible functions by the invertibility of h", which was shown in Lemma B.2 In conclusion, the estimated unmixer -ø- satisfies -Φ- o f being an invertible element-wise Transformation and permutation.
C Proof of Theorem 3.3
Before diving into proving Theorem 3.3 we first show three cornerstone lemmas of the proof. These lemmas help characterize the detailed conditions under which an optimal solution of J- V-Vxl criterion can recover the latent components up to a constant permutation and invertible element-wise transformations, i.e.,
g( (n) (ip(s (s(n) SN.
(59)
The formal result is given in Lemma C.3
Intuitively. (59) holds if the points s(n) in set SN are sampled densely and closely enough from Ps with sufficiently large N, together with some regularity conditions on the ground-truth and the learnable function classes. To show (59), we employ a three-step argument, Lemma C.I Lemma C.2 and Lemma C.3
Firstly, we show that for any feasible solution of J-Vollax that attains maximal Jacobian volume at each (n) Sy, must give an estimated latent component vector that is permutation and invertible element-wise transformation of the ground truth s(m) fact, due to continuity, this holds every
point an union of neighborhoods UN -.\ -,,,, (m)) centered on each ((x), The result is stated in the following Lemma C.I. and proof is a straightforward adaptation of Lemma B.2
20

---

Lemma C.I. Denote any feasible solution of J- J-VolMax problem 7 as 6= := 0, o learned from classes of learnable encoders and decoders F,9 that include the function classes of ground-truth encoders and decoders F' G', Assume that there is an unknown finite set SN == { ..... } such that the Assumption 3.7 is satisfied at each of the N points Sx. the estimated latent components 8/ 9 (x ()) are not permutation and invertible element-wit transformation
} C S with unknown XN := {x(*)) E X x) f(g () E Sx) C X
of ground-truth 8 s(m), then there exists another feasible solution with parameters θ such that
V(s ") ,0) < ((s θ), Vgl") E Exx
(60)
Furthermore, there exists union UN = U U S S of open ball neighborhoods U(m) E ||8 8/n)|/2 < ) centered g(n) E SN such that
V(s; Vs; As UN.
(61)
Proof. For the feasible solution == fa and g JΦ. define differentiable function h S S as composition h = g f. and consider a data point (n) = f(s(m)) E M' for s s(n) E SN. Let g()) = (( Following the same argument 16 36 as in Lemma B.2 for H =
(J " (w((())
we can similarly derive that log det H < log C' at any point s E SN. with equality holds if and only if H is signed permutation matrix, i.e. when h composition of permutation and invertible element-wise transformations.
We will show in the following that if h is not a composition of a permutation and an element- wise invertible transformation, then the Jacobian volume V(s("); attained at s(x) of the feasible solution θ strictly smaller than the Jacobian volume of another feasible solution; hence, the feasible solution e does not reach maximal Jacobian volume at s(m).
Since = g o f is not a composition of permutation and component-wise invertible mappings,
log det H < log C, (s(") E Sy.
(62)
This implies
log det(( s (1) J (m))
= log det h 1 18 (n) T , (n)) T
18(
log det (3(1)) Dw ())) Dio [5 (s()()) ( (m 133(1))
= log det HD (m)) )J (8 )m)) Jy
= log[(det H)* det (s (n)) J (1) 5 (m) D (s(
< log((2 del( (m)))) J ())) D (s (o E SN..
(63)
Consider an invertible continuous element- function h S + S with ((() := h (s( S. h() (")), and J (3))) Jj (h (s("))) CDw This function h merely rescales and maps each latent component by an element- wise invertible transformation. Define
j(3(m)) ((((a())),
(64)
whose Jacobian
_y((()) = J ) ((((m))
(((0))
(65)
We show that f()(t)) ) is a feasible solution to J- VolMax 7 i.c. 7b) and (7c) hold. First, note that
Jy )..l1 IW ( 1/1 - C, Vi [m].
(66)
Second, observe that
= (n). F(8(n)) = f(h(s(m))) = 1f f(h(h-" (s(())) "((()"(s('))) = f(s(n))
(67)
21

---

Hence, there exists feasible solution a = f(s(")) to J- VolMxx such that
log det (J fo (J())) fa (3( log det (((( J y ))((m))), Va SN.
(68)
Equivalently, for Ô [0, Φb) being the parameters corresponding to f and s(m). V(s!)),) () < v(o'"); θ), Va!") E SN-
(69)
Finally, due to continuity of the involved functions, there exists neighborhood U(x) = (s E ||s - s s(x) ll < } centered at s(a) such that
V(s; < O)s As E U(m)
(70)
which holds for all (1)) U()), Hence, with UN = - U(m) V(s; < ()) Vs E UN-
(71)
Following Lemma C.1 we show that if the SDI-satisfying finite set Sx are sampled dense enough such that UN covers significant part of S, then any optimal solution of J-VolMax criterion must recover the ground-truth latent components s(x), up "deeeddddd permutation and invertible element-wise transformations. The proof leverages the result from the previous Lemma C.T to show contradiction that if a feasible solution e does not give us ground-truth latents up to aforementioned ambiguities, there must exist another feasible solution θ that is strictly better than θ in terms of J-Vol objective 77 i.e.,
B[V(s; E[V(s;
(72)
The result is formally given in the following Lemma
Ø learned from classes of learnable encoders and decoders F, that include the function classes of ground-truth encoders and decoders F'G Suppose there is unknown finite set SN (o()).., s(N)} } C. with unknown XN := {x(x) X x(n) f(s(")), Vg(n) E ENN} C X such that the Assumption 3.I satisfied at each of the N points in SN.
Lemma C.2. Denote any optimal solution of J-VolMax problem 77 as θ=
U E E --s-- s d(n)3 within which V (s; θ) is optimal, assume that Sy is sampled densely enough from with sufficiently large N such that
For the union of neighborhoods UN ,N_,,U
F(SEUN)
Guax Guni 1,
(73)
P(sES\UUx)
where Gmin, Gmax are bi-Lipschitz constants of the Jacobian volume surrogate with respect to the parameter, i.e.
Gmin I1 O2l12 (V V(s; 02) S Gmax 02l12, Vs E S,
(74)
for any parameters θ2, O₂ in F. G. Then, the optimal encoder g = 96 gives
(^) ( ii(s("))((s!))) Vg(n) SN
(75)
where f((())) is (n) depenndnt permulation mairix, and p(s(")) (M (81), vector of d invertible element-wise transformations on each component of (()) P...a)))))))T is
Proof. Suppose that there exists an optimal solution θ 0, क) of J-VolMax such that the mapping h 96 f is not composition of permutation and element wesse invertible transformations. By Lemma C. within a certain union of neighborhoods UN such that (73) is satisfied, there exists
another feasible solution é = [0, お) such that,
V(s; θ) < V(s;)), vs UN.
(76)
22

---

We now show that: since the solution θ does not reach maximal Jacobian volume at each point within UN as 76.. + cannot reach maximal expected Jacobian volume over all of S, i.e.,
(sV θ) < ([V(s: θ)],
(77)
and therefore θ is actually not an optimal solution of J-VMM,, which raises contradiction.
To begin, notice that the open neighborhood U(m) {s 8 || s(')|| < d has non-zero measure, then their union UN = - U()) also has non-zero measure. This allows us to define
CUN ( = f V(s;
(78)
JOEUN
CSyUN (0) [
V(s; ()sssds
(79)
JOESUN
as the part over UN and the part over S \ Un of the objective value ([V(s;; ()), ie., CUN (0)
-
Csvux (0) = (s; 0)). Similarly, define
CUn (U)( : S SEUN
V(s; ()p(sdd,
(80)
Csrux (6): == -SESSUM
V(s; O(s:))((s)ds
(81)
with CUp (7) + CSrUN (0) = ([V(s;00)
Now, observe that
( Cux ( = / (V(=;))
V(s: (s))
(82)
JEUUN
[V θ) - Vss + p(8)ds
(83)
DEUN
Gmin 0 (12p(s)d
(84)
xEUU
Gmin 16 - 01|
p(s)ds
(85)
DEUN
= (l
(86)
where 83 is due the fact that V(s; θ) > ((:;)), Vs UN from Lemma C.1 and 84 is obtained by applying bi-Lipschitz continuity of (); as assumed in (4). Similarly, we have
/
ICsiuw (6) - (s)uv(())
JEESUUX
(V(;;)) (8: θ (ps)ss
(87)
G Ô (1(p(ss)ds
(88)
JESSUUN
G
||
p(s)ds
(89)
BES\UN
= Gmax |
11lP((s e S UN),
(90)
where (87) is obtained via triangle inequality, and (88 IS from the bi-Lipschitz continuity of V (s;-) as in the assumption 74p.
Combining 86
990) with the assumption that Sv (8(1) (,l),,.. m... ((N) is sampled densely and closely
enough over S via ps such that
P(s E UN)
Guax Gmin
1,
(91)
P(s ESUUN
we have the following chain of inequalities:
CUN (²) CUn ()) > Gmin |
2 121((s E Un)
(92)
23

---

maax |60 + 12P(s S Un)
(93)
z CCsUUN (0) CosUN (0)
(94)
z CSAUN (0) Csiux
(95)
Therefore, the following strict inequality holds:
Cux(()) + CsAUx (9) > CUN ( + CSaUx (0).
(96)
or equivalently.
B[V(s;))] > (s; 6)
(97)
That to say, θ = 10, [b is indeed feasible solution of J- that strictly better than
- (0, Φ) in terms of expected Jacobian volume in J-VolMax objective 7a This contradicts the assumed optimality of (0, Φ) to J-VolMax problem 7
We hence conclude that any optimal solution with encoder g. the Jacobian at every point s(n) E SN of the function h ÿ must satisfy
n ()) ((s (I(ss
where (n)(n) is an invertible diagonal matrix with [D(s())),, dependent on i-th component (x) only, and IT( (x))) is permutation matrix dependent the point g(x)
In the following Lemma C.3 we show that under further regularity conditions, the permutation ordering in the result 775) of Lemma C.2
g(x (0) (n)))))))pll ((s(())) (a(i) E SN.
become a constant permutation independent of s(1), i.e. [[(s()) Ii, Va(a) Sy. That is, g(x(x) (Ip(s!"), Vs(n) SN-
Lemma C.3. Assume that there is finite set SN == {s(1), .... s with Xx == for X a = f(s), Vs Sx} such that the Assumption 3.1 is satisfied at each of the N points in Sy. Denote as the optimal encoder learned by J- J-Volxax. Suppose that
g(x( (m) (i((s!)))pl (ps)))),, (p(s")) Vgln) E SN
(98)
for a ( dependent permutation I1, and additionally these three regularity conditions hold:
1. The functions f. 9. P are Lipschitz continuous with constants L Lg. Lp 0.
2. There is constant 7 >> such that for any permutation matrix II Pa and II + ni(s(())
(n)( - -IIp(s())1l )12 2 7, vn E [N].
(99)
3. For No0) = (8 E S: ||s - 2/1) < (mi) with (n) T.Lybgtig) the union set of the neighborhoods, N == U A(t),, is connected subset of S.
Then, the permutation ordering ii(s!!)) (98) of the estimated latent components is constant, i.e., fi()(n)) = for a fixed permutation II Pa- Consequently,
(()( rip(s (m)), Vn E [N].
(100)
Proof. Define a function (s) parametrized by permutation matrix II Pa as (u(s) := [g(xz) Ipp())ll-
(101)
From the Lipschitz continuity of f.G. P, we have the corresponding Lipschitz continuity property of (nx-)): for any 8, s' e S,
(n)(s) (n)(s [l)(f())) IP((s)l22 ll'()()')) 1Ip(s')lla)
(102)
24

---

< [I@(f(s)) ITp(s) (0((()) Ipp(s
(103)
= |I@(f(s)) - + - Ipp(s)]l
(104)
< [0([(s) - - [T)(()) - -IIp
(105)
[0(f(s)) ) 110(s) -P(s']l2
(106)
I L - - -plss
(107)
= (Lghy + L - 8'112.
(108)
where applied the triangle inequality in (103) and (105), the fact that II Pa is orthogonal
matrix in (106), and the Lipschitz continuity of 9. p in (07). Then, we have the following chain
of inequalities for any E A(m);
((a (8) S ((((tty)(( + (Lgh4 + [p]l|s - s
(109)
< <"iist (s(m)) + (L + Lp)r(n)
(110)
= =L'gli + Ipp)n(n)
(111)
where we have used the Lipschitz property of l (n(att)) for 109). the defined radius of A/()) for (10
and the fact that (A(ati) (s(m)) = due to (98) for Similarly, for any E M(a) and any permutation matrix II * (((())),
(T) (T((")) (Lghe Lp)lls s 12
(112)
> (n(smm) - (Lgh. + Lp)r(n)
(113)
> (Lghg + Lpp)r(n))
(114)
where we again used the Lipschitz property of ( for 112 the defined radius of A()) for (13), and the error gap of non-optimal permutations in assumption (99 for (114. Combining [III
and 114) with the assumption p(n) < HLgLy+LoJ' we have
(i())))()) < (LgLg + Lp)r(n) < (Lgb + Lp)r (n) < (T)
(115)
This implies that for any points & in the s(n) centered neighhorhood s E A/(x), the optimal permuta- tion for (s) is still the optimal permutation (i()))) at the center point s(m); that is,
((()()) arg min П1FР
(0x()) (T)s))l,, Vs E ((())
(116)
As a result, fi(:): :NN + Pa. which maps a latent vector s E A to permutation matrix in Pa. is a locally constant mapping over the union set N. Since the locally constant II(-) maps to discrete space Pa. the map II..) is indeed continuous map. Combining the continuity of II(.). (.), map from A into a discrete space Pa, with the fact that A connected, we can conclude that IT(.) is indeed constant. Hence, (I{s) = II for any s E N, which also implies (s()))) = ÍI for any n E [N].
In conclusion, there is single permutation matrix ñ E Pa such that g(x(*)) (m)(s Vn E [N].
(117)
Lastly, we combine the results from Lemma C.I generalization bound to derive Theorem 3.3
C with Rademacher complexity-bas based
Theorem 3.3 (Identifiability under Finite-sample SDI). Assume that there a finite set Sy
(s!1),.. ..., ((V) } with Xx {x E X satisfied at each of the N points in SN. Let g E G be the optimal encoder by J-VolMax criterion 77. and O contains the parameters of the learned encoder and decoder. Further assume that the following regularity conditions hold:
f(s), Vs E SNY such that the Assumption 3.1 is
1. The functions = f and 96 are from classes G' and G, respectively, where g' S G.
2. The functions f 9, P are Lipschitz continuous with constants Lg, Lgg Lg > 0.
25

---

3. There is a > O such that for any permutation matrix II Pa and II o II(s (from 9
(0(x)()) ITp(s("))lz 2 7, Vn [N].
4. For N( = fs E S ||s - sl < r(m)} with r(n) < 2(L)L the union of the neighborhoods, N = Un N is connected subset of S and ((s; θ) is optimal for any 8 E E.
5. The points s(1)
(NN) SN densely locate in S such that P(s E EN) E EXNN) Gmax/Gmin > 1,
(10)
where Gmin, Gmax are bi-Lipschitz constants of the Jacobian volume surrogate: for any parameters 01 22 F.)). Gmin 02112 - (8; θ1) Vss; 02) Gmax |0, - 02l12, E S.
(11)
Then, ((x(m)) = ())) vn E [or constant permutation matrix fi Pa. Furthermore, with probability at least 1 5,
Bx-p(s) (x)) [ip(s))la)
(LI ( /uLL////N,
(12)
where RN(9) is the empirical Rademacher complexity of the encoder class.
Proof. Given that J-VoIMax criterion provides us with an optimal solution (F reaching maximal Jacobian volume at each s(n), such that the estimated latent components g(x(")) = identifies s (n) SN up to permutation and component-wis wise transformation, i.e.,
(a)(") (Tp(s("), Vn [N],
(118)
show unite-sample analysis on how well estimate the ground-truth latent sources
all S, up the said permutation II and the invertible element-wise mappings p0-) associated with g()) SN. To proceed, let us define loss function
{ig, (æ, 8)) = (l fip(8)
(119)
which T.g-Lipsehi,z with the constant depending on Lipschitz continuity Ø. f. as I.
LgLg + Lp (as derived in Lemma C.3 h. The defined loss function can be assumed to be upper-
bounded by a finite constant as E(g, (x, 5) S M. Note that the encoder from J- J-VolMax 9 minimizes the following empirical risk:
Z(g) = + N ((g, (((()),(())) (())))
(120)
i.c. [(g) O. We are now a position use a generalization bound that characterizes the difference between the given empirical risk L(Φ) and the population risk
C(g) E. [(( (x, s)) = Ea-p, (lgg2( (Tp(s)]l2)
(121)
Applying an (empirical) Rademacher complexity based generalization bound 770 Theorem 26.5] with the contraction lemma (70] Lemma 26.9] gives the following with probability least 1 8:
2ln(4/5)
((g) < -(g) L(g)+ 22 4M
Vig g.
(122)
N
As result, the following bound holds for any optimal solution ğ G: with probability least
L E Eg-p, [l((()) rTp(s) & 2L.
4M
(123)
N
26

---

D Additional Remarks on Related Works
IMA. The IMA framework also exploits influence diversity [20). There, the ground decoder
f is assumed to have an orthogonal Jacobian (c.g., Möbius transformations (20), which reflects
linearly uncorrelated influences from the latent components. Similar orthogonal lacobian-based
regularization have found empirical successes for disentanglement problems in computer vision 771
However, IMA does not provide identifiability (only local identifiability was shown (44)
Sparse Jacobian-based NMMI.
Our proposed SDI condition naturally subsumes some cases of
sparse Jacobian conditions employed in (2 19 49 A notable example is when m 2d, the SDI assumption boils down to a sparsity pattern in y where each row touches a corner of the weighted
L1 ball BTe((] and thus the gradients are scaled unit vectors (with d 1 zeros). Nonetheless, when m > 2d, completely dense J y can also satisfy SDI. We note that although the L1 regularization on J appears in both J-VolMax and sparse Jacobian criteria (see (3 19 50), the reasons of having this regularization are very different: the latter use the I,1 norm as a surrogate to attain Jacobian sparsity, but our method uses the regularization to confine [s))s in an In-mrrm ball (which is not proxy for sparsity).
Object-centric Representation Learning (OCRL). Many OCRL works also share the perspective of modeling sevenness-partccllaae the influences of latent variables onto objects/concepts in the observed domain. While the goal of NMMI is recover each individual latent variable, OCRL (2)
23 24 48) seeks weaker form of identifiability -ww.bbiitty-ddenffyiiinngn blocks of variables corresponding
to observed objects/concepts. When the block size becomes 1, the goal of OCRL becomes latent
variable identification as in NMMI. Notably, 2 23 24 propose specific forms of f (which is called "decoder" in the OCRL literature) for block-wiss wise identifiability. These structures often imply sparsity in Jacobian or higher-derivatives of f. For example, the compositional generator in (2) and additive decoder in 23) are associated with structured sparse J Js's. Similar to DICA, the work (2 uses a
Jacobian- based regularizer in their training loss. The work (23) imposes the additive decoder structure
at the model architecture level. Interestingly, the additive decoder structure in (23) was shown to have the same disentanglement effects as a block-diagonal Hessian penalty proposed in [51]
E Experiment Details
E.1 Further Details on Implementation and Evaluation
On Warm-up Heuristic. The use of warm-up heuristic with regularization schedulers help alleviate the numerical instability that comes with optimizing log det of Jacobian of the decoding neural network fe. which can quickly explode if not controlled. Specifically, by gradually introducing the det det term, we prioritize optimizing for data reconstruction the warm-up period, since it is a hard constraint in the VolMax formulation (7c). Moreover, by minimizing || s. (g ()) during warm-up, we prevent the Jacobian from exploding as result of maximizing Cvol(t) (t) while keeping the norm term | so (g ((("))) at a reasonable magnitude. This also encourages a smoother encoder fo with small enough Lipschitz constant; recall that this is important for identifiability of J- J-VolMax under finite SDI-satisfying samples, as pointed out by Theorem 3.3.
Efficient Computation of Jacobian Volume Regularizer. As mentioned in Section [ the computa- tional cost of the log-det volume surrogate Cvol in (14) is (m²)) This might make Cvol impractical to use in high-dimensional data setting, where the dimension of the Jacobian matrix is large. An
alternative volume surrogate is to use (34]]] 72
tr Tr((dI - 11 JJ (g
J (g
(124)
d-1
ofo (x()))
0S 0So (ar
Σ
(125)
as!")
which corresponds to the sum of Euclidean distances between all pairs of partial derivative vectors 0fo/0s, Maximizing err would force the partial derivatives to be spread out, akin to the effect from
maximizing log-det surrogate Cyol, while reducing the computational cost to O(m)). In Table 3 we test the wall-clock time needed for backpropagation per epoch of the logdet-based and trace-based regularization, using setting similar to Mixture C in synthetic experiment (with d = 3, m 40).
27

---

Table 3: Compare performance and wall-clock time for gradient computation (per epoch) of trace-based and logdet-based surrogate for Jacobian volume.
Logdet-basee DICA
Trace-based DICA
Sparse
Base
R score
0.94 + 0.08
0.91 = 0.07
0.79 = 0.12
0.63 0.04
Time (ms)
31.31 = 0.30
24.89 + 0.26
18.95 0.18
6.90 0.18
One can see that while trace-based DICA formulation approximately 25% faster than logdet-based DICA, R score only slightly decreases. Lastly, we remark that there are other methods for reducing the computational cost for optimizing Jacobian determinant of neural network, such as (73
Evaluation with R². To evaluate the R² score, we use nonlinear kernel ridge regression with radial basis function kernel [74), which is universal function approximator. We train the regression model on a train set, and use the prediction of the trained regressor on a test set to calculate the coefficient of determination. This gives us the nonlinear R score.
Compute Resources. All experiments use one NVIDIA A40 48GB GPU, hosted on a server using Intel Xeon Gold 6148 CPU @ 2.40GHz with 260GB of RAM.
E.2 Synthetic Simulations
For each of the simulation, we generate 30000 samples from the described synthetic data generation processes, of which 90% are for training and 10% are for evaluating the MCC and R² scores. We use two ReLU fully-com colly-connected neural networks with one 64-neuron hidden layer for the encoder and the decoder. The autoencoder is trained via Adam method (75) with learning rate 14-4 for 200 iterations,
among which the first 20 epochs are for warm-up. The regularization hyperparameters are chosen by
validating from (10-2 10-3, 10-4, 10-5), resulting in Aval = 10-1 Anorm = 10-4, Asp = 10-4
The neural networks are initialized via He initialization with uniform distribution 76
E.3 Single-cell Transcriptonies Analysis
Data Generation with SERGIO Simulator. We extract top 5 TFss that regulate the most number of genes in TRRUST database. Furthermore, we only keep 30% of genes with a single regulator, so as to have more challenging mixture; this gives us m 178 gene expressions.
To construct the gene regulatory mechanism f for SERGIO generation, confident interactions from TRRUST as primary regulating edges with coefficients randomly sampled from (1.5,, 1.8) if the TF is the gene's activator, and from U--1.8, -15)) if the TF is the gene's repressor. In addition, to model the potential spurious cross-talks interactions between the TF's and the genes, add secondary weak regulating edges with coefficients with random signs and their magnitudes uniformly sampled from U 1001,
the extracted high-
We simulate 20000 cells of the same cell type, and use the gene expression data of these cells the observation dataset to train with J- J-Volaax learning criterion.
Training. We use two ReLU fully-connected neural networks with one 64-neuron hidden layer for encoder and decoder. We use Adam optimizer (75) with learning rate 10-4, and train the autoencoder for 4000 epochs, with the first 1000 epochs for warm-up. The regularization hyperparameters are
chosen by validation from (10-2 10-3, 10-4, 10-5) to be Avol = 10-3 3 Annrm = 10-4 Aap 10
The neural networks are initialized using He initialization with normal distribution 776).
F Additional Experiment: Unsupervised Concept Discovery in MNIST
Setting. In this section, we explore further experiments beyond nonlinear mixture model identification tasks, to demonstrate potential applicability of DICA in disentanglement and representation learning. Specifically, we apply DICA to the MNIST dataset [77] to train an autoencoder using J-VoIMax loss function. We use convolutional neural networks (CNNs) for both encoder and decoder, with the architectures reported in Table 4 We choose d = 10 as the latent dimension, and the observation dimension is 32 x 32- 1024. The regularization hyperparameters are Avol 10 Asp 10-4, and optimizing using Adam with learning rate 10 for 100 iterations, of which the first 50
28

---

Table 4: Architectures of encoder and decoder used in MNIST experiment (all stride pad out pad
Encoder
Decoder
Input: x E R32x32x1
Input: 8 E 100x1x1
3 3 Conv, 256 ReLU
2 X 2 ConvTrans, 32 ReLU
3 Cony, 128 ReLU
3 x ConvTrans, 64 ReLU
X 3 Conv, 64 ReLU
3 X 3 ConvTrans, 128 ReLU
x 3 Conv, 32 ReLU
X 3 ConvTrans, 256 ReLU
2 2 Conv, 10
3 x 3 Conv JonvTrans, 1
epochs are for warm-up. To prevent numerical instability regarding the log-determinant of Jacobian due high-dimensional data of 1024, instead of optimizing log det(.) directly, optimize log det(- + 7T) with r > 0, so as to avoid zero determinant that leads to exploding log-determinant.
encode test image To visualize how the learned latent factors affect the observed images,
corresponding to digit to achieve a latent vector ă [$1, ..., 110 E R Then, we vary each component 8; in the range of +:4 std to achieve new latent vector 8new with certain component being increased/decreasee. The new latent vector neww are used as input of the trained decoder to obtain a new image Tnew- Therefore, the observed image &new-
examine how specific latent component 83 affect
Results. We reported four visualizations corresponding to varying 88, 89. 81, from images with digit 3, 0, 2, 5, correspondingly in Fig. 3 One can observe that as learned latent component increases/decreases, the semantic meaning (i.e., digit) of the image changes in relatively uniform manner towards a new digit. This suggests that the latent components induced by J-VolMax are somewhat correlated with semantic meanings of the images.
We that not all latent components obtained correspond to a clear semantic meaning, and the variation of latent components can sometimes significantly distort the images beyond normal hand-written images. We hypothesize that this is due the used autoencoder architecture suitable for disentanglement purposes, as well as the optimization algorithm not well-designed for this task, since we directly implement J- VolMax criterion without adaptations for image data. Nonetheless, the preliminary results are encouraging, and we speculate that with better-designed architecture and algorithm, the J-Vx criterion can significantly improve in the challenging task of disentangling meaningful latent components of image data.
being
29

---

3
3
3
3
3
3
3
3
9
9
6
2
3
3
3
3
3
3
9
9
9
D
3
3
3
3
3
3
3
5
9
9
E
3
3
3
3
8
8
8
8
6
3
3
3
3
3
5
5
5 5
6
3
3
5
3 0
6
3
5
2
2
3
3
3
3
3
33
5
3
3
3
3
3
3
3
9
6 666
(a) [Anchor digit 3] As increases, digit 3
(b) [Anchor digit 0] As 8y decreases, digit increasingly looks like digit 6.
increasingly looks like digit 9.
2
2
2
2
2
2
4
4
2
2
2
2
2
2
27
J 5
2
2
2
2
2
2
2
J 555555555
2
2
2
22
9 5
2
2.22.
5
0777
5 5
2
2
222
8
2
27777.
5 $
22
5 055555555
0
5
(c) [Anchor digit 2] As decreases, digit 2
(d) [Anchor digit 5] As 87 increases. digit increasingly looks like digit 4.
increasingly looks like digit
Figure 3: Some resulting images obtained by varying certain component s; by #4 std (increasing from left to right) from the latent vector of anchor image. Fach corresponds one of 10 different anchor images sampled from test set, and each column is the resulting image by varing from the corresponding anchor image. We can see that some latent components correlate the semantic meaning (i.c., digit) output images: as some s; increases/decreases, the semantic digit of all 10 anchor images change uniformly towards another digit.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim on model identifiability of DICA framework using J- VolMax
learning criterion is proved in Theorem 3.2 and 3.3 and the theoretical results are supported by numerical experiments Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed limitations the DICA framework, particularly the current algorithm design as well robustness analysis under noisy setting, in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [Yes]
Justification: The assumptions are provided in the statement of Theorem 3.2 and Theorem and their proofs are included in the supplementary material.
3.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: An overview of J-VolMax implementation and the datasets used in numerical experiments are provided in Section 5 more details are provided in the supplementary
materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
32

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The data and code are provided in the supplementary materials, along with relevant instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
http:/.cccpubbbcguuuueeee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The training and test details such as neural network architecture, learning rate, regularization hyperparameters, etc. are provided in the main paper well as the supplementary materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the that is necessary to appreciate the results and make sense of them.
the paper to level of detail
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiment results in Section 5. include mean and standard deviation, whereas experiment results reported in Section 5.2 are based on median over multiple trials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources used for the experiments are reported in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The theoretical and experimental works in this paper follow the NeuRIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is on the theoretical aspects of machine learning, which does not have immediate societal impacts.
Guidelines:
34

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Since this work is of theoretical nature, data and models proposed by this paper does not pose immediate risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Most the implementations in this work are created by the authors, and the external libraries, codes, and datasets used in the paper are properly credited, with license and terms respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
35

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The codes accompanying this paper are provided in the supplementary materi- als, together with relevant instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not include crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
36

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
37


Reviews:
[{"review_id": "YXjCyHgfxl", "reviewer": "Reviewer_8yNX", "summary": "The authors propose an alternative way to tackle the nonlinear mixture identifiability problem. The setting is as follows: unknown latent components are transformed by an unknown nonlinear function into observation and the challenge is to recover the unknown latent components from the observations. In general, the problem might be ill-posed, therefore most work are restricted to scenarios where identifiability can be shown. One way to do so is to make stronger assumptions on the nonlinear transformation, e.g. by imposing sparsity on the Jacobian matrix of the transformation. Instead, the authors build a geometric argument as to why it is sufficient to penalize the Jacobian log determinant. This allows them to treat functions with dense jacobians as well.", "questions": "- the proposed approach employs an unconstrained neural network and forces its inverse to exist with a loss that acts on training points. While this approach has been previously used in other contexts as well, it only requires the learnt map to be invertible locally. Especially in high-dimensional spaces, this doesn't enforce invertibility in any systematic way. This might not be visible in the experiments since a very small network is used (1 hidden layer with 64 units) in a very low-dimensional space\n- Maybe I am missing something, but have you considered defining f as invertible by construction like normalizing flows? this would also make the proposed method computationally much more efficient when it comes to computing the Jacobian determinant (from cubic to quadratic/linear)\n- If this is a viable approach, one could easily [1] even constrain the function to lie on a L1 norm ball by construction (in place of the regularization term of your approach)\n- I have a general doubt concerning the experiment section. In the synthetic experiment the nonlinearity is chosen at random so I expect its jacobian not to be sparse (which is also a benefit of the proposed approach). On the other hand, for the single-cell experiment, do we know something about the degree of sparsity of the Jacobian of the \"true\" f to be learnt?\n- If the underlying Jacobian is however sparse, should the method Sparse provide better results than the one obtained? Did you also try with higher sparsity? (I saw in the appendix you selected the best parameter via cross-validation)\n- In case both experiment have a dense/non-sparse Jacobian, how does your method compare to current approaches that assume some form of sparsity?\n\n[1] Negri et al., Injective flows for star-like manifolds, ICLR 2025", "limitations": "Limitations are clearly stated and discussed in the conclusions. The main limitation is that in practice it is really expensive to compute the Jacobian log-determinant at every iteration, which makes the method not viable in practice.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- the mathematical foundation seems solid and the solution proposed is elegant\n\nWeaknesses:\n- even though it is left for future work, the proposed approach is not viable in practice because training requires computing the Jacobian determinant, which is cubic if the function is unconstrained (see questions below)", "was_revised": true, "final_justification": "- I believe the work provides interesting theoretical insight to the mixture identifiability problem\n- experimental results are encouraging even though not particularly extensive\n- I remain a bit concerned about the true invertibility of the learnt function (which is invertible only on the training set) and about the Jacobian determinant computations, which are extremely expensive (if computed exactly) or only approximate (if an estimator is used)", "rebuttal": "## Efficient computation of Jacobian log-determinant\n\nNote that our focus was on the identifiability analysis side, and thus we did not delve deep into the implementation side. **In practice, multiple methods can be used to avoid the cubic complexity**. Here we provide two options:\n\n1. For example, one can use a trace-based formula as a surrogate [A1] (instead of using determinant) for volume optimization, **which would reduce the computational complexity to quadratic**. As a quick proof-of-concept test, we replace the log-det term with a trace-based surrogate in [A1] in our synthetic experiment similar to Mixture C with $d = 3$ and $m = 40$. We report the nonlinear R2 scores obtained from each method, as well as the average wall-clock time for gradient computation per epoch; as we can see, **the gradient computing time for this trace-based DICA is much lower than logdet-based DICA** while having similar performance in R2 score:\n\n| | Logdet-based DICA | Trace-based DICA | Sparse | Base |\n|---|---|---|---|---|\n| R2 Score | $0.94 \\pm 0.08$ | $\\boldsymbol{0.91} \\pm 0.07$ | $0.79 \\pm 0.12$ | $0.63 \\pm 0.04$ |\n| Wall-clock Time per Epoch (ms) | $31.31 \\pm 0.30$ | $\\boldsymbol{24.89} \\pm 0.26$ | $18.95 \\pm 0.18$ | $6.90 \\pm 0.18$ |\n\n2. In addition, as pointed out by Reviewer Sbtv, there are techniques in the unsupervised learning literature [A2] that can efficiently deal with Jacobian determinant optimization. **The per-iteration complexity of this method is quadratic instead of cubic**. We leave the details for future follow-up works.\n\nWe will add discussions on the computational aspects in the conclusion section.\n\n*References*\n\n[A1] Fu et al., “Robust Volume Minimization-Based Matrix Factorization for Remote Sensing and Document Clustering”, IEEE Transactions on Signal Processing, 2016.\n\n[A2] Gresele et al. \"Relative gradient optimization of the Jacobian term in unsupervised deep learning\" NeuRIPS, 2020.\n\n## On inverting neural networks via autoencoder loss\n\nIn practice, as pointed out by the reviewer, when the mixing function is not too complicated and there are enough data points, such an autoencoding loss in DICA objective can learn to approximately invert the decoder. However, when the mixing function is much more complicated or the dimensions are high, learning to invert the decoder might be much more complicated. Hence, the vanilla autoencoder presented in the manuscript is limited in this setting, which calls for more sophisticated models.\n\nNonetheless, our focus is to demonstrate a learning principle for identifiable nonlinear mixture learning, and the vanilla autoencoder merely serves as proof-of-concept. We leave the investigations for implementing DICA learning criterion under more complex settings in follow-up works.\n\n## On the use of normalizing flow in DICA framework\n\nWe thank the reviewer for suggesting normalizing flow (NF), which is invertible by construction, as a possible alternative for the vanilla autoencoder. However, as far as we know, most of the current NF models require the latent dimension $d$ being equal to the number of observations $m$, whereas our SDI condition requires $m \\geq 2d$. \n\nThat being said, NF is a promising tool for efficiently implementing DICA framework, which we leave for further investigation in the future.\n\n## On the single-cell transcriptomics experiment\n\nThe purpose of our experiment on single-cell transcriptomics data is to test the performance of DICA in a setting where we have *no control of the mixing Jacobian* (which would show the usefulness of this framework in practice). To the best of our knowledge, a full ground-truth gene regulatory mechanism is yet available, and the transcription factor - gene expression interactions are only partially available. Therefore, we don’t know exactly how sparse gene regulatory mechanisms are, as there might be crosstalks (one transcription factor interacts with many genes) and various transcription noises [A3] such that the mixing Jacobian is dense/non-sparse. DICA can be considered as an addition to the data scientist’s toolkit for studying gene regulation in the case of non-sparse regulatory systems, on top of existing sparsity-based tools.\n\n*References*\n\n[A3] Tamar Friedlander et al. “Intrinsic limits to gene regulation by global crosstalk”. Nature\nCommunications, 2016.\n\n## Performance of DICA when the mixing Jacobian is sparse\n\n*Following the reviewer’s suggestion, we tried with higher sparsity of the Jacobian*. We used a setting similar to Mixture C in the paper. The nonlinear R2 scores are shown in the table below. One can see that both DICA and Sparse work reasonably well. We hope to remark that DICA’s identifiability condition covers some cases where the Jacobian is sparse, so it is not surprising that DICA could work well under higher sparsity levels. However, DICA is not proposed to outperform sparsity-based methods when the Jacobian is sparse, but as a complement to tackle more cases where the Jacobian is not sparse.\n\n| $(d, m)$ | DICA | Sparse | Base |\n|---|---|---|---|\n| $(3, 40)$ | $0.88 \\pm 0.16$ | $0.85 \\pm 0.07$ | $0.68 \\pm 0.14$ |\n| $(4, 50)$ | $0.85 \\pm 0.80$ | $0.80 \\pm 0.12$ | $0.58 \\pm 0.07$ |\n\n## Performance of DICA vs. sparsity-based approaches, under dense/non-sparse mixing Jacobian\n\nWe refer the reviewer to our experiments (in Section 5) as well as the table in our reply to your first question. We note that both the mixtures in the synthetic simulations and the single-cell transcriptomics experiment have Jacobian matrices that are (highly likely) non-sparse and SDI-satisfying. DICA outperforms Sparse, another approach that only exploits the Jacobian sparsity structure via a Jacobian L1 norm regularizer. These experiment results confirm the advantage of our proposed method in non-sparse settings."}, {"review_id": "pnVOGcvf6f", "reviewer": "Reviewer_Sbtv", "summary": "The paper studies the problem of nonlinear mixture model identification (NMMI), i.e., how to provably recover d-dim. latent sources $s$ from m-dim. observed nonlinear mixtures $x=f(s)$ thereof. (If the sources are jointly independent, this problem is known as nonlinear ICA.) NMMI is hard in general, but has been addressed in prior work under additional assumptions such as auxiliary information or sparsity. \n\nThe paper instead proposes a different approach based on the assumption of sufficiently diverse influences (SDI). SDI is a non-trivial condition involving the convex geometry of the gradients $\\nabla f_1, ..., \\nabla f_m$. Intuitively, it states that globally, i.e., for all $s$: \n- (i) for each source $s_j$ there exists an $x_i$ that is positively influenced by $s_j$ and an $x_k$ that is negatively influenced by $s_j$\n- (ii) for each source $s_j$ there exists an $x_l$ such that $\\nabla f_l$ is dominated (in magnitude) by its $j$th component $\\partial x_l / \\partial s_j$\n\nAlgorithmically, the SDI condition suggests an approach based on maximising the volume of the Jacobian $J_f$ subject to invertibility of $f$ (enforced via an autoencoder) and a bounded norm constraint on the rows of the Jacobian, i.e., the gradients $\\nabla f_i$. \n\nTheoretically, it is shown that SDI enables identification of the latent sources up to permutation and element-wise reparametrization, without requiring statistical independence or sparsity. For a finite sample setting, it is shown that, under additional conditions, the identification error can be bounded. \n\nEmpirically, the proposed DICA approach is compared against a vanilla autoencoder and a baseline with sparsity regularization on the Jacobian. On synthetic and semi-synthetic gene expression data, DICA is shown to outperform these baselines in terms of latent variable recovery.", "questions": "### Questions\n- How did you arrive at or come up with the SDI assumption? Can it be derived or motivated \"from first principles\"? \n- Thm. 3.2 establishes that J-VolMax under Asm. 3.1 yields identifiability, but is there a more direct connection between the SDI condition and volume maximisation?\n- For fixed magnitude vectors, the volume of a parallelotope is maximal iff the vectors are orthogonal, as enforced in IMA [11]. This seems related to the J-VolMax objective, but there the norm constraint in Eq. (7b) appears to be on the gradients (i.e., rows of the Jacobian), rather than on the partial derivatives (i.e., columns of the Jacobian) as in IMA. Could you please comment further on the relationship between the two approaches? \n- is the SDI condition invariant to reparametrisation of the sources? i.e., if SDI holds for $f$ and $f$ is replaced by $f\\circ \\rho$ for an element-wise function $\\rho$, does SDI still hold? \n- what is relation between the notion of \"one source dominating the derivatives\" entailed by SDI to that of anchor features/hard sparsity [e.g., 41]? can one think of the former as a relaxation of the latter? \n- Since [4] appears to be a key reference and is used as a baseline, are the authors aware of [R1] and [R2] which relax the sparsity assumption on the Jacobian of [4]? You may want to consider these works and discuss their relation to SDI as well. \n\n### Comments and Suggestions\n- There appear to be some inconsistencies regarding $\\mathcal{B}_\\infty^{1/w(s)}$ vs. $\\mathcal{B}_\\infty^{w(s)}$ in Asm. 3.1., the caption of Fig. 1, and lines 125-126. Is it possible that there is a typo here? \n- Eq. (7a) seems to miss a closing bracket.\n- $\\hat{\\Pi}$ is undefined in eq. (9)\n- Thm 3.3 mentions \"three regularity conditions\" but four are given.\n- The claim that IMA [11] does not have identifiability underpinning appears a little too strong: while full identifiability for IMA has not been established (to the best of my knowledge), e.g., [5] proves local identifiability.\n- Since optimization of the Jacobian is highlighted as a computational challenge, the authors may be interested in [R3].\n- The authors may be interested in [R4] which remarks on some potential issues with some proofs of reference [50] in their manuscript. \n\n### References\n[R1] Lachapelle, Sébastien, et al. \"Additive decoders for latent variables identification and cartesian-product extrapolation.\" Advances in Neural Information Processing Systems 36 (2023): 25112-25150.\n\n[R2] Brady, Jack, et al. \"Interaction Asymmetry: A General Principle for Learning Composable Abstractions.\" International Conference on Learning Representations (2025).\n\n[R3] Gresele, Luigi, et al. \"Relative gradient optimization of the jacobian term in unsupervised deep learning.\" Advances in neural information processing systems 33 (2020): 16567-16578.\n\n[R4] Buchholz, Simon. \"Some remarks on identifiability of independent component analysis in restricted function classes.\" Transactions on Machine Learning Research (2023).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- the paper is well written and (apart from the mathematical complexity) easy to read\n- the proposed approach is original and novel\n- the paper provides contributions along several axes: a new principle, identifiability results, an estimation methods, and experiments\n- the finite sample analysis (which is not common in the identifiability literature) is a nice addition\n- the paper does a good job at covering related work and discussing its relations to the proposed approach\n- the paper does not just consider sythetic data, but also provides results on a semi-synthetic dataset from a biological simulator\n\n### Weaknesses\n- the SDI assumption is quite difficult to understand: while the \"Physical Meaning of SDI\" paragraph is helpful to gain some intuition and interpret the condition, it is unclear how the authors came up with or arrived at the precise mathematical formulation in Assumption 3.1; it does not help that definitions and explanations of key concepts (convex hull, polar set, MVIE) are deferred to the appendix (these should be moved to the main paper);\n- the precise relation between the SDI assumption and the Jacobian volume maximisation (J-VolMax) approach, as well as that between SDI and orthogonality maximization of the Jacobian columns (IMA) remain a little unclear; \n- it seems that the proposed approach requires a very large number of observations relative to the number of latent sources (in section 3, it is stated that $m\\geq 2d$ is required, but in the experiments it is much larger, $m\\gg d$) \n- the empirical evaluation is somewhat limited in that it only considers relatively naive baselines; comparison with other nonlinear ICA methods (e.g., IMA or based on sparsity) and reducing the number of observations $m$ (resp. increasing the number of sources) could be useful additions to get a better understanding of empirical performance\n- quantifiers (for all, there exists) are used too sparingly and inconsistently; including these for all formal statements would improve clarity", "was_revised": true, "final_justification": "My initial concerns and questions regarding\n- the intuition behind the SDI assumption and the motivation behind volume maximisation\n- the relation of the proposed method to existing literature on nonlinear blind source separation\n- limitations of the method, in particular the requirement for a large number of measurements compared to sources\n\nhave been satisfactorily addressed by the authors during the discussion, and the authors have agreed to include these clarifications in the updated manuscript. \n\nI believe that this work is sound and a valuable addition to the literature on identifiable nonlinear representation learning. \n\nAfter reading the other reviews, it seems that it was rightfully raised that the authors did not sufficiently credit a prior work in inspiring their approach, but that they have promised to rectify this. I do not consider computation issues in Jacobian computation (mentioned as a key weakness by another reviewer) sufficient grounds for rejection, seeing as this is transparently stated. \n\nTherefore, I recommend that this paper be accepted.", "rebuttal": "## Motivations behind the SDI assumption\n\nThe motivation of DICA is to exploit the pattern of how sources $\\mathbf{s}$ influence the observations $\\mathbf{x}$ via the geometry structure of the mixing Jacobian matrix, in a manner similar to sparsity-based NMMI and the IMA. In particular, while the Principles of Independent Causal Mechanisms [11, 45],\n\n> *“The causal generative process of a system’s variables is composed of autonomous modules that do not inform or influence each other.”*\n\ninspired the orthogonality of columns of the Jacobian matrix as in the IMA, we took a different view. Instead of assuming influence mechanisms being totally independent of each other, we assumed that\n\n> ***the ways that distinct sources $s_1, …, s_d$ influence the observations are sufficiently different***\n\nin the sense that the columns of Jacobian matrices spanned in different directions. **Therefore, the change to observations** $\\mathbf{x}$ **by varying** $s_i$ **should be sufficiently different from that by varying** $s_j$. One can also think of SDI conditions as a relaxed version inspired by IMA, which describes geometrically the spread of Jacobian columns in space, without strict orthogonality.\n\nTo quantitatively pin down this intuition to form the exact SDI condition, we follow the idea in the matrix factorization literature [9, 17, 18, 48] (particularly [48]), where sufficiently scattered conditions (SSCs) are often used to prove identifiability of linear mixture models. SSC describes how column/row vectors of a latent factor matrix are scattered inside a convex polytope. Instead, we describe the scattering pattern of mixing Jacobian matrices at every point $\\mathbf{s}$ over a continuous manifold. Despite the differences, we found properties of SSC helpful in underpinning the identifiability in the nonlinear mixing case after translation to SDI in our case (see Eqs (21)-(33) in Appendix B, which are based on SSC-based matrix factorization work [48]). To be specific, the nice mathematical properties of the SSC, coupled with a volume-maximizing criterion that find a set of maximally spread-out Jacobian columns as guided by our intuition, gives us the theoretical tool to establish an identifiability proof.\n\nWe will include more explanations on SDI along with related convex geometric notions in our updated main paper.\n\n## Connection between SDI and Jacobian volume maximization\n\nHere we briefly explain the geometric intuition behind Jacobian volume-maximizing learning criterion. It is a generalization of geometric intuition of volume max/min based matrix factorization (see [48] and related works, e.g., [17, 9, 18]).\n\nDespite the technical definition of SDI being described in terms of the gradients $\\nabla_{\\mathbf{s}} x_i$, the implication is more clear in the partial derivative domain, as described in “Physical Meaning of SDI” section. The SDI condition at a point $\\mathbf{x} = \\mathbf{f}(\\mathbf{s})$ implies that there exists some $x_i$ that $s_j$ dominantly regulates $x_i$, so $|\\partial x_i / \\partial s_j| \\gg |\\partial x_i / \\partial s_k|$ for $k \\neq j$. For the two partial derivative vectors $\\partial \\mathbf{x} / \\partial s_j, \\partial \\mathbf{x} / \\partial s_k \\in \\mathbb{R}^{m}$, this means that the two partial derivative vectors are spread out and point in quite different directions.\n\nMaximizing the Jacobian determinant as in the DICA objective would indeed encourage the partial derivative vectors to spread in different directions, aligning with the SDI assumption.\n\n## Relationship between IMA and DICA\n\nAs mentioned in the response to the Reviewer’s first question, both the IMA and the DICA framework seek to tackle the NMMI task via exploiting the geometric structure of mixing Jacobian, which describes how the sources $\\mathbf{s}$ influence the observations $\\mathbf{x}$. In particular, the learning criterions in both IMA and DICA include a Jacobian determinant term, which measures how spread-out the partial derivatives are.\n\nThe difference between IMA and DICA lies in how the two data models assume the geometric structure of mixing Jacobian geometric structure, which dictates the goals of their respective learning criterions:\n* In IMA, it is the orthogonality of partial derivatives; hence, **IMA objective looks for a decoder with orthogonal partial derivatives, via maximizing the (normalized) Jacobian determinant**\n* In DICA framework, the SDI data model instead assumes a specific spread-out pattern of gradients in a $\\ell_1$ norm ball, which entails that the partial derivatives are spread out in space (see the second reply as well as “Physical Meaning of SDI” section); hence, **DICA objective looks for a decoder with partial derivatives maximally spread out via maximizing Jacobian determinant** ***inside the constraint set from SDI***.\n\nThe shared ground that DICA and IMA is indeed very intriguing, and we hope that the identifiability results presented in DICA can help shed light on a global identifiability analysis of IMA, in addition to its local identifiability property [5] as mentioned by the Reviewer.\n\n## Invariance of SDI to reparametrization of sources\n\nThank you for raising this point. \n\nIf the sources $\\mathbf{s}$ is reparametrized by an *invertible* element-wise transformation $\\rho_i$, the new Jacobian function of $\\mathbf{f}(\\boldsymbol{\\rho}(\\mathbf{s}))$ is $J_{\\mathbf{f}}(\\boldsymbol{\\rho}(\\mathbf{s}))J_{\\boldsymbol{\\rho}}(\\mathbf{s})$, where $J_{\\boldsymbol{\\rho}}(\\mathbf{s}) = \\text{Diag}(\\frac{d \\rho_1(s_1)}{d s_1}, …, \\frac{d \\rho_d(s_d)}{d s_d})$ is a diagonal matrix. Hence, the gradients $ \\nabla_{\\mathbf{s}} x_i $ are scaled by $J_{\\boldsymbol{\\rho}}(\\mathbf{s})$. By Proposition A.1., the MVIE and the weighted $\\ell_1$ norm ball $B_1$ containing the gradients are rescaled accordingly by $J_{\\boldsymbol{\\rho}}(\\mathbf{s})$, and hence the SDI still holds (albeit with a different $B_1$). A similar argument was used in the proof of Theorem 3.1, particularly (18)-(21).\n\nAs a result, the SDI is invariant to *invertible* element-wise transformation of sources. For *non-invertible* element-wise reparametrization, it is unclear.\n\n## Relation between SDI and anchor features/hard sparsity\n\nThe sparse influence structure such as anchor features (e.g., [41]) or hard Jacobian sparsity (e.g., [50, 51]) indeed have some overlap with SDI; for example, in the base case m = 2d of SDI.\n\nIn fact, for the SDI cases of $|\\partial x_i / \\partial s_j| \\gg |\\partial x_i / \\partial s_k| \\approx 0$ where the non-dominant source $s_k$ has approximately absent influence, one can interpret these cases as a relaxation of the sparse influence. However, we note that the SDI also includes the cases where $|\\partial x_i / \\partial s_j| \\gg|\\partial x_i / \\partial s_k| \\gg 0$, where the non-dominant sources $s_k$ have an influence greater than 0. This departs from the sparse influence structure, which demands (approximately) zero entries in Jacobian matrices.\n\nTherefore, while some cases of SDI overlap with Jacobian sparsity, in general SDI is not a sparsity-based condition.\n\n## Relation between [R1], [R2] and SDI\n\nWe thank the reviewer for pointing out the two related works [R1] and [R2]. To our understanding, the focus of [R1], [R2] as well as [4] are on object-centric settings, which aims to recover disentangled representations of different objects in the presented scene. Therefore, [R1], [R2] and [4] mainly focus on identifying *blocks of latent variables* that represent objects. The goal of DICA aligns more closely with that of nonlinear ICA, which aims to recover the *individual latent variables* from an observed mixture. \n\nAn exciting direction is to examine how the learning principle in DICA can contribute to object-centric representation learning, and we will discuss the relation between DICA and [R1], [R2] in future works.\n\n## On number of observations required relative to number of sources\n\nIndeed, the SDI condition requires at least $m \\geq 2d$, which is more than other nonlinear ICA methods (which typically requires $m = d$ or $m \\geq d$ only). In practice, DICA favors $m \\gg d$ to attain the SDI; see the “Discussions” in Section 3.1 and references therein for further details.\n\nHowever, high-dimensional data is prevalent and often lies inside a low-dimensional latent manifold. Hence, the SDI condition is still very applicable in real-world settings such as computer vision or single-cell biology. For example, we refer the Reviewer to Section E on a preliminary experiment on MNIST, where we showcase how DICA with a vanilla CNN-based autoencoder can discover some latent components that control the semantic digit of output.\n\n## Experiments with IMA and reduced number of observations\n\nAs the Reviewer suggested, we compare DICA with IMA. We also reduce the ratio of number of observations $m$ over number of sources $d$. For IMA, we use the local IMA contrast in [Def. 3.1, A6] as a regularizer, plus the autoencoder loss. We use a setting similar to Mixture C for testing. The nonlinear R2 scores are in the table below.\n\n|(d,m)|DICA|IMA|Sparse|Base|\n|---|---|---|---|---|\n|(3,40) | $0.90 \\pm 0.10$ | $0.63 \\pm 0.15$ | $0.80 \\pm 0.13$ | $0.63 \\pm 0.10$|\n|(3,30)| $0.89 \\pm 0.07$ | $0.63 \\pm 0.12$| $0.77 \\pm 0.12$ | $0.63 \\pm 0.14$|\n|(3,20)| $0.82 \\pm 0.17$|$0.60 \\pm 0.12$|$0.74 \\pm 0.13$|$0.60 \\pm 0.10$|\n\n* Since the setting of SDI-satisfying Mixture C does not favor the IMA assumption, i.e. orthogonal Jacobian, the IMA regularizer did not perform as well as DICA in this experiment.\n\n* Furthermore, the results show that while DICA requires a relatively large number of observations $m$ as predicted in the theory, the performance decays but is still acceptable when $m$ reduces towards $m = 2d$.\n\n*References*\n\n[A6] Ghosh et al., “Independent Mechanism Analysis and the Manifold Hypothesis”, Causal Representation Learning Workshop at NeuRIPS, 2023."}, {"review_id": "x2gjWp9i75", "reviewer": "Reviewer_pCZF", "summary": "The article addresses the problem of identifying latent features from their nonlinear mixtures. The approach proposed is based on exploiting a data model where the the rows of the Jacobian matrix of the nonlinear mapping are assumed to lie in a scaled $\\ell_1$ norm ball. The article proposes the use of two neural networks as parametric nonlinearities, whose parameters are obtained through a volume maximization on the gradients of the nonlinear mixing to exploit their pressumed scattering within the scaled $\\ell_1$ norm ball. The article offers numerical experiments to illustrate the performance of the proposed approach which look convincing.", "questions": "- what is the motivation for the assumption about the Jacobian (inclusion in the l-1 norm ball) besides identifiability? Do you see that it is a practically valid assumption? can we also consider other sets alternative to l-1 norm ball (such as other polytopes in [48])?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n - The article addresses the hard problem of obtaining latent factors from their nonlinear mixtures and offers a solution with proven identifiability under some assumptions about the data model\n\n## Weaknesses\n\nA. One significant concern is the lack of proper attribution to prior work from which key definitions and methodology appear to be adopted. Although the referenced article is included in the general bibliography, the paper fails to cite it at the precise points where the reused ideas are introduced. As a result, the presentation gives a misleading impression of originality. This omission is not merely stylistic—it impairs the reader's ability to understand the origin and novelty of the contributions and violates standard scholarly norms of credit and transparency. Furthermore, this makes it difficult to assess whether the paper's contributions are genuinely novel or derivative.\n\n\nThe main reference under consideration is [48] G. Tatli et.al. “Polytopic Matrix Factorization: Determinant Maximization Based Criterion and Identifiability”. In: IEEE Transactions on Signal Processing 69 (2021), pp. 5431–5447.\n\nAlthough the authors very briefly list this reference at the end of Section 4, the current work appears to closely extend the polytopic matrix factorization (PMF) introduced in [48] for linear mixtures to nonlinear mixtures. In this endaveour, the author(s) appeared to use definitions, criterion and the proof steps in [48] without any precise referals.\n\nTo clarify this point, I checked [48] and summarize main relevant points here: [48] proposes  PMF for the extraction of latent factors from their mixtures, which doesn't assume independence of latent factors but assumes a generative model (in Section II.A of [48]) for the input matrix $\\mathbf{Y}$:\n\n$$ \\mathbf{Y}=\\mathbf{H}_g\\mathbf{S}_g $$\n\nwhere $\\mathbf{H}_g$ is unknown full-column-rank factor matrix and $\\mathbf{S}_g$ is the unknown matrix whose columns are in a polytope $P$. [48] proposes the use of the following volume maximization  problem (in Section II.B)  to identify these factors  from the input matrix $\\mathbf{Y}$:\n\n*PMF Problem*\n\\begin{array}{ll}\n\\displaystyle \\max_{\\mathbf{H} \\in \\mathbb{R}^{M \\times r},\\, \\mathbf{S} \\in \\mathbb{R}^{r \\times N}} & \\det(\\mathbf{S}\\mathbf{S}^T)  \\tag{PMF}\\\\\n\\text{subject to} & \\mathbf{Y} = \\mathbf{H} \\mathbf{S}, \\\\\n& \\mathbf{S}_{:,j} \\in \\mathcal{P}, \\quad j = 1, \\dots, N.\n\\end{array}\n\n\nFor the identifiability condition it introduces the *sufficient scattering condition* (in Section II.c) for the columns of $\\mathbf{S}$ as\n\n*{Definition II.2: Sufficiently Scattered Factor:}* \n$\\mathbf{S} \\in \\mathbb{R}^{r \\times N}$ is called a sufficiently scattered factor corresponding to $\\mathcal{P}$ if\n\n  - (PMF.SS.i) $\\mathcal{P} \\supseteq \\mathrm{conv}(\\mathbf{S}) \\supset \\mathcal{E}_{\\mathcal{P}},$ and\n  - (PMF.SS.ii) $\\mathrm{conv}(\\mathbf{S})^{*, \\mathbf{g}_{\\mathcal{P}}} \\cap \\mathrm{bd}(\\mathcal{E}_{\\mathcal{P}}^{*, \\mathbf{g}_{\\mathcal{P}}}) = \\mathrm{ext}(\\mathcal{P}^{*, \\mathbf{g}_{\\mathcal{P}}}),$\n\nwhere $\\mathcal{E}_{\\mathcal{P}}$ is the MVIE of $\\mathcal{P}$, centered at $\\mathbf{g}_{\\mathcal{P}}$.\n\nSection III.B of [48] focuses on the special case $\\mathcal{P}=\\mathcal{B}_1$ ($\\ell_1$ norm ball), i.e. the sparse PMF case. Then Theorem 3 of [48] proves that the optimal solution of the (PMF) optimization problem above under sufficiently scattered condition (Definition II.2) recovers the factors of the PMF generative model upto sign and permutation ambiguity.\n\nFrom this perspective, the contribution in the current article is an extension of the Sparse-PMF of [48]:\n\n- The Sufficiently Diverse Influnce definition (Assumption 3.1) is exactly the sufficiently scattered condition (Definition II.1 above) in [48] applied to the rows of the Jacobian for the special case of polytope $\\mathcal{P}=C \\mathcal{B}_1$. Figure 1 is the two dimensional version of Fig. 5 in [48].\n\n- Jacobian Volume Maximization problem in Page 5 is the extension of the Sparse PMF problem above, imposing sparsity to the Jacobian matrix, replacing the linear reconstruction constraint in Sparse-PMF with the nonlinear construction constraint.\n\n- The proof of Theorem 3.2 (Identifiability of J-VolMax) in Appendix B, shares exact same steps as the proof of Theorem 3 for Sparse PMF in [48]:\n    * the treatment between equations (22)-(27) in the proof of Theorem 3.2 are the same as the treatment around (15)-(17) in the proof of Theorem 3 in [48],\n    *  Lemma B.1 in Appendix (from a nonexisting reference [55]) is the same as the second stage of the proof of Theorem 3 of [48],\n    * the treatment between equations (28)-(33) in the proof of Theorem 3.2 are the same as the treatment in the third stage of the proof of Theorem 3 (around equation (24)) in [48].\n\nIn summary, the current article appears to be the extension of the sparse-PMF for linear mixing to nonlinear mixings.\nDespite this tight  extension relationship, and  the reuse/generalization of results in  [48],  \n\n- there is no mention of the reference [48] in Section 2 (Background). Section 2 should also discuss the reference\nK. Huang, N. D. Sidiropoulos, and A. Swami, “Non-negative matrix factorization revisited: Uniqueness and algorithm for symmetric decomposition,” IEEE Transactions on Signal Processing, vol. 62, no. 1, pp. 211–224, October 2013\nwhich appears to be the first article where the sufficiently scattered condition for latent extraction, the root of the sufficient diverse influence condition, was introduced,\n- there is no discussion about how the current work extends Sparse-PMF for linear mixings to  nonlinear mixings, \n- there are no precise references at the appropriate points to the original determinant maximization based sparse PMF problem and the corresponding sufficiently scattered identifiability condition definitions, Theorem 3, and the proofs of Theorem 3 in [48]. \n\n\n\n\nB.  The assumptions about the data model (about the Jacobian), their applicability to practical problems could be better motivated and explained.\n\n\nB. Section 5.2 \"Inferring Transcription Factor\" could be better explained in terms of the problem statement and the performance measure.", "was_revised": true, "final_justification": "As I noted in my original review the article addresses a noteworthy theoretical problem. However, the presentation style was confusing as it ignored almost all relevant background in deriving the proposed approach. This led to confusion about how the authors came up with their volume optimization based approach and the SDI condition. However, their response clarifies that  they will perform a significant improvement in that direction. Furthermore, they provided satisfactory answers for general review comments. Due to the significance of this work I increased my score to 4.", "rebuttal": "## Relation between DICA and matrix factorization literature\n\n***[Relation to NMF/PMF - SDI]*** We would like to thank the reviewer for bringing up this issue. We sincerely acknowledge the importance of the work [48] (as well as [9, 17, 18]) and their contributions to matrix factorization’s identifiability analysis via volume-based criteria under scattering conditions. As we mentioned in “Related works”, the SDI is indeed a scattering-based condition as those in [9, 17, 18, 48], though formulated in the Jacobian domain. However, we understand the reviewer’s concern that the mentioning was too brief to fully clarify the connections. To address this, we plan to expand the related works and add remarks as follows:\n\n> “Volume minimization/maximization based approaches have been popular in structured matrix factorization (SMF) [9, 17, 18, 48], where scattering based conditions are used to establish identifiability for these criteria. The work [A4] first used the so-called sufficiently scattered condition (SSC) (the condition itself appeared in [A5] for NMF but no determinant was involved) to show that volume minimization of a latent factor identifies an SMF model under the SSC. The SSC means that the columns/rows of a factor matrix in an SMF model are widespread in the nonnegative orthant. This line of work later was generalized to SSC in different norm balls [48] (also see [17]). The SDI condition can be viewed as a functional extension of the column/row-based SSC condition [48, 17] to the Jacobian domain over a continuous manifold (in particular, the L1 norm ball based SSC in [48]). Compared to SSC, SDI admits rather different physical meaning as it is defined using the first-order derivatives. Nonetheless, the J-VolMax criterion can still be regarded as a nontrivial extension of volume-based SMF [A4, 9, 17, 48] (particularly [48]) into the nonlinear domain. In addition, although the contexts of nonlinear and linear unmixing (e.g., SMF) are drastically different, some proof techniques of volume based SMF [48] are found useful in our case (see Appendix B).”\n\nAs the reviewer suggested, we will also revise the “Background” section accordingly to reflect the above.\n\n***[Relation to NMF/PMF - Proof Contributions]*** We thank the reviewer again for raising this important point. While there was no intention to omit proper attribution to prior work, we agree in hindsight that key connections—particularly in parts of the proof—were not made sufficiently explicit. **We take full responsibility for this lack of clarity and are committed to improving it**.\n\n**To clarify, the non-existing [55] in the Appendix B was exactly [48], from which we cited parts of [Theorem 3, 48] as Lemma B.1 to use as a key component in the proof of Theorem 3.2**. As the main paper and the appendices were submitted separately (with one week apart), the bibliography was not cleanly managed and such missing references happened inadvertently. We apologize for this oversight.\nRegarding the proof contributions, note that the proof of Theorem 3.2 consists of the following components:\n\n1. Lemma B.2 first relies on recasting the nonlinear identifiability problem J-VolMax into the first-order derivative domain. This leverages the smoothness of the nonlinear function and the availability of infinite data (so that the operations are in a continuous manifold). This part includes (12)-(21).\n\n2. After establishing (21) in the Jacobian domain, the steps in (22)-(33) utilizes the SDI and a series of facts from convex analysis. In particular, these facts were presented in multiple prior works, particularly, [48] and [17].   \n\n* The steps in (22)-(27) utilizes insights from convex geometry to characterize the Jacobian. Similar facts (in form of matrix columns/rows) were presented in prior works in polytopic matrix factorization [Theorem 3, 48] (and later dictionary learning [Lemma A.2, 17]);\n\n* In step (28), we apply a key insight from parts of [Theorem 3, 48], cited as Lemma B.1 from the non-existing [55] in our manuscript, to establish a condition for when an involved Jacobian matrix is orthogonal; \n\n* Finally, (29)-(33) are direct consequences of (28), i.e., Lemma B.1 from [48], and the SDI condition to establish that an involved Jacobian is not only orthogonal, but indeed a permutation matrix. This part again characterizes the Jacobian, but similar facts exist in [48] characterizing matrix factors.\n\n3. From (34) to (53), the proof shows that the Jacobian volume must be maximized as in the learning criterion, such that (33) is not contradicted.\n\n4. Finally, in Theorem 3.2, the proof is to utilize Lemma B.2 and continuity to unify permutation ambiguity over the smooth manifold.\n\nTherefore, we hope the reviewer agrees that, while our proof draws on known convex analysis tools in one segment, it goes substantially beyond previous work in both scope and technical content. In particular, the shift from factor matrices to Jacobians, and the treatment of identifiability over continuous manifolds, present challenges not encountered in linear models such as PMF or NMF. **That said, we fully agree with the reviewer that the use of known results—even when adapted or generalized—should be acknowledged explicitly**. It is unfortunate that the key cited lemma from [48] appeared as non-existing [55], because Lemma B.1 is an important step in the proof that we hoped to credit properly. We will revise Appendix B to include the correct citation, i.e., [48], and make the following remark after (22)–(33):\n\n> \" The convex geometric arguments in Eqs. (22)–(33) are from tools developed in prior matrix factorization literature, particularly [48], where they were used to establish identifiability under scattering-based conditions.”\n\n*References*\n\n[A4] Fu et al., “Blind Separation of Quasi-Stationary Sources: Exploiting Convex Geometry in Covariance Domain”, IEEE TSP, 2015.\n\n[A5] Huang et al., “Non-negative matrix factorization revisited: Uniqueness and algorithm for symmetric decomposition”, IEEE TSP, 2013.\n\n## SDI motivation and real-world applicability\n\nWe thank the Reviewer for this question. Please find our answer in the following, which contain parts of a reply to Reviewer Sbtv.\n\nThe motivation of DICA is to exploit the pattern of how sources $\\mathbf{s}$ influence the observations $\\mathbf{x}$ via the geometry structure of the mixing Jacobian matrix, in a manner similar to sparsity-based NMMI and the IMA. In particular, the latent variable model of DICA assumes a prior belief that\n\n> ***the way that distinct sources $s_1, …, s_d$ influence the observations are sufficiently different***,\n\nin the sense that the columns of Jacobian matrices spanned in different directions. **Therefore, the change to observations** $\\mathbf{x}$ **by varying** $s_i$ **should be sufficiently different from that by varying** $s_j$. This can be described by how columns/rows of Jacobian are spread out.\n\nTo quantitatively pin down this intuition to form the exact SDI condition, we follow the idea in the matrix factorization literature [9, 17, 18, 48] (particularly [48]), where sufficiently scattered conditions (SSCs) are often used to prove identifiability of linear mixture. SSC describes how column/row vectors of a factor matrix are scattered inside a convex polytope. Instead, we describe the scattering pattern of mixing Jacobian matrices at every point $\\mathbf{s}$ over a continuous manifold. Despite the differences, we found properties of SSC helpful in underpinning the identifiability in the nonlinear mixing case after translation to SDI in our case (see Eqs (21)-(33) in Appendix B, which are based on SSC-based matrix factorization work [48]). To be specific, the nice mathematical properties of the SSC, coupled with a volume-maximizing criterion that find a set of maximally spread-out Jacobian columns as guided by our intuition, gives us the theoretical tool to establish an identifiability proof.\n\nWe further found the SDI condition being applicable in real-world tasks such as single-cell transcriptomics analysis (Section 5.2) and semantic disentanglement in images (Appendix E). We hope to experiment other real applications of DICA in future.\n\n## Alternative constraint sets to the scaled L1 norm ball in SDI\n\nWe thank the Reviewer for their suggestion. While the weighted L1 norm constraint in SDI yields an arguably interesting nonlinear model, as elaborated in \"Physical Meaning of SDI\" section and the experiments, we believe there are other useful alternative constraint sets for SDI. This is an interesting direction that deserves attention in future works. \n\nHowever, we note that not all polytopes considered in NMF/PMF literature would yield a meaningful/useful data model in a nonlinear mixture context. This is because the physical meaning of polytope constraints in SDI is very different from the physical meaning of polytope constraints in PMF/NMF, due to the two being in totally different domains (Jacobian of nonlinear function vs. mixing matrix).\n\nFor example, in the case of NMF [9], the polytope is a probability simplex. In Jacobian domain, this means the L1 norms of gradients are bounded by 1, and the gradients are non-negative. The non-negativity of gradients imposed by the NMF polytope is quite restrictive and *the physical meaning of a nonlinear mixing function with non-negative gradients at every point is quite unclear*. In contrast, in the linear data model as in NMF/PMF, constraining the rows/columns of a mixing matrix inside probability simplex yields a sensible data model, which found many applications.\n\nHence, one should consider carefully whether a proposed set constraining the Jacobian of nonlinear mixing function would yield meaningful physical meanings for the data generating process. Since the physical meaning of the polytope in SDI of DICA and in SSC of NMF/PMF are very different, not all constraint sets in NMF/PMF would yield a meaningful Jacobian constraint for a nonlinear mixture."}]

Author rebuttal:

Meta-review: This paper proposes a novel approach to identifiability in  nonlinear source separation. Initially, the reviewers had a number of concerns. These included: intuition and motivation of the SDI assumption; invertibility of the learnt function; cubic computational cost of the Jacobian determinant; and relationship with the existing nonlinear identifiability literature.

Overall, all reviewers were satisfied with the author responses during the rebuttal period and raised their scores. For a final version of the manuscript, it will be important for the authors to incorporate the promised changes into the main paper.
Decision: Accept (poster)