Function Classes for Identifiable Nonlinear Independent Component Analysis
Authors: Simon Buchholz
Michel Besserve
Bernhard Schölkopf
Venue: NeurIPS 2022 Accept (2022)
Keywords: nonlinear ICA
independent component analysis
identifiability
blind source separation
unsupervised representation learning
conformal maps

Abstract: Unsupervised learning of latent variable models (LVMs) is widely used to represent data in machine learning. When such model reflects the ground truth factors and the mechanisms mapping them to observations, there is reason to expect that such models allow generalisation in downstream tasks. It is however well known that such identifiability guaranties are typically not achievable without putting constraints on the model class. This is notably the case for nonlinear Independent Component Analysis, in which the LVM maps statistically independent variables to observations via a deterministic nonlinear function. Several families of spurious solutions fitting perfectly the data, but that do not correspond to the ground truth factors can be constructed in generic settings. However, recent work suggests that constraining the function class of such models may promote identifiability. Specifically, function classes with constraints on their partial derivatives, gathered in the Jacobian matrix, have been proposed, such as orthogonal coordinate transformations (OCT), which impose orthogonality of the Jacobian columns. In the present work, we prove that a subclass of these transformations, conformal maps, is identifiable and provide novel theoretical results suggesting that OCTs have properties that prevent families of spurious solutions to spoil identifiability in a generic setting.

Function Classes for Identifiable Nonlinear Independent Component Analysis
Simon Buchholz, Michel Besserve Bernhard Schölkopf
Max Planck Institute for Intelligent Systems
Tübingen, Germany
Maauchoolz,,bbesserre ba bs) mpg de
Abstract
Unsupervised learning of latent variable models (LVMs) widely used to represent data machine learning. When such models reflect the ground truth factors and the mechanisms mapping them to observations, there is reason to expect that they allow generalization in downstream tasks. It is however well known that such identifiability guaranties are typically not achievable without putting constraints on the model class. This is notably the case for nonlinear Independent Component Analysis, in which the LVM maps statistically independent variables to observations via deterministic nonlinear function. Several families of spurious solutions fitting perfectly the data, but that do not correspond to the ground truth factors can be constructed in generic settings. However, recent work suggests that constraining the function class of such models may promote identifiability. Specifically, function classes with constraints on their partial derivatives, gathered in the Jacobian matrix, have been proposed, such as orthogonal coordinate transformations (OCT), which impose orthogonality the Jacobian columns. the present work, we prove that subclass of these transformations, conformal maps, is identifiable and provide novel theoretical results suggesting that OCTs have properties that prevent families of spurious solutions to spoil identifiability in generie setting.
Introduction
Unsupervised representation learning methods fit Latent Variables Models (LVM) to complex real world data. While those latent representations allow to create realistic novel samples or represent the data in compact way 32 16, they priori not related the ground truth generative factors of the data. It highly desirable to recover the true underlying source distribution because those are expected to help with various downstream tasks, e.g., out of distribution generalization 45 2
One principled framework for representation learning is Independent Component Analysis (ICA) where one tries to recover unobserved sources IR from observations = f(s) and one assumes that the components s; are independent. An important result is that for linear functions f it is possible recover from observations up to certain symmetries, i.e., the model is identifiable M. In contrast, for general non-linear models f is highly non- ian-identifiable (25) This has important consequences for representation learning, in particular the learning of disentangled representations is also unidentifiable without some access to the underlying sources [37]. Notably, this makes theoretical analysis of large body of methods (see, e.g., (0 30 42)) that enforces disentanglement difficult.
Several additional assumptions were suggested to make the ICA problem identifiable. Broadly, there two directions. First, some works imposed additional or different restrictions the distribution of the sources. One line of research adds temporal structure by considering time series data 19 23 24
More recently, Hyvärinen et al. 26 proposed to introduce observed auxiliary variable e.g.. class label, such that the source distribution has independent components conditional on the auxiliary
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

variable. They show that under suitable assumptions on the distribution of and arbitrary nonlinear mixing functions can be identified. Several recent works extended this approach (2
48 53
Another possibility is to restrict the class of admissible functions by considering more flexible classes than just linear functions, but not allowing arbitrary non-linear functions. 'he general aim of this approach is to find sufficiently "small" function classes such that ICA is identifiable within this class while making them as large as possible allow flexible representation of complex data and being applicable to real world problems. So far results in this direction are rather limited. It was shown that the post-nonlinear model identifiable maps in dimension 2 is almost identifialle [25] More recently. identifiability of volume preserving transformations was investigated in the auxiliary variable case in [53 (combining the two possible restrictions) and identifiability based on sparsity of the mixing function was studied in [54].
449 Moreover, it has been shown that ICA with conformal
In this work, we extend the previous works by proving new identifiability results for unconditional ICA. We consider conformal maps (i.e., maps that locally preserve angles) and Orthogonal Coordinate Transforms (OCT) (i.e., maps where Df Df is a diagonal matrix where D denotes the derivative of f). OCTs, that we will also call orthogonal maps for simplicity, were recently introduced in the context of representation learning in (77 motivated by the independence of mechanisms assumption from the causality literature. The main focus of this work is to prove new identifiability and partial identifiability results for this class of functions. An overview of our results can be found Table We summarize the main contributions and the structure of this paper as follows.
We prove that ICA with conformal maps is identifiable in d 2 3 and extend the earlier in Section 3)
results in dimension 2 (Theorems 2
We define a new notion of local identifiability (Definition 5) and prove that ICA with
orthogonal maps is locally identifiable (Theorem 4 in Section
On the contrary we show that ICA with volume preserving maps is not identifiable not even in the local sense (Theorem c in Section 5
We introduce new tools to the ICA field: our results are based on connections to rigidity theory (see Section 5). restricting the global structure of functions based on local constraints. Moreover, exploit the global structure of partial differential equations related to the identifiability problem while most earlier results argue locally using linear algebraic tools.
2 Setting
Independent component analysis investigates the problem of identifying underlying sources when observing a mixture of them. We will consider the following general setting: there exists some random hidden vector of sources E Rd and the observed data is generated by
Ps(s) II p.(s;) = P
x = f(s),
(1)
where f IRd - Rd is smooth and invertible. The condition on means that its coordinates (often referred to as factors of variation) are independent. Formally this means that the distribution IP of satisfies IP E M1 (TR) where M1 (TR. denotes the probability measures on TR. The goal of ICA is to find an unmixing g : Rdd + Rd such that g(x) has independent components. Ideally, this should recover the true underlying factors of variation and achieve Blind Source Separation (BSS), i.e., g f-1 up to certain symmetries. Identification of the true generative factors of variations of the observations is of interest since these provide causal and interventional understanding of the data.
An important observation was that, in the generality stated above, identification of is not possible. In 2 two general constructions of spurious solutions were given: the well known Darmois construction and a construction based on measure preserving transformations. The latter one is closer to our work here and we will discuss those in more detail in Section 4 and Appendix C In a nutshell is based on the observation that for measures P with smooth density one can construct smooth Measure Preserving Transformations (MPT), m IRd + Igd (that mix the different coordinates), i.e., maps that leave IP invariant, such that ms))
D s if 8 P This implies that all functions (f o m)- recover independent sources since (f o m) (x)
D
s making BSS impossible.
We use the notation X 2 Y indicate that the two random variables X and Y' follow the same distribution

---

(a)
(b)
(c)
(d)
Figure Illustration of the considered function classes. (a) shows standard coordinate frame, (b) a conformal map applied to this frame which preserves angles, (c) an orthogonal map (polar coordinates) that preserve the orthogonality of lines parallel to the coordinate axes but all angles (see red line), (d) a volume preserving map.
Thus is natural question whether additional assumptions on the mixing function distribution of allow us to identify f. Let us define a framework for identifiability. We assume data is generated according ! where f belongs to some function class F(A, B) of invertible functions A B. and P satisfies P P for some set of @dd Finally, let be a group of transformations g Rdd RRd that encodes the allowed symmetries up to which the sources can be identified follows. The function class will be simply denoted when domain/codommin information irrelevant.
which we will always assume to be diffeonorphisms probability distributions C M1 (TR)
Definition 1. (Identifiability) We say that independent component analysis in (F,P) is identifiable up to S if for functions f, E F and distributions IP, P' E P the relation
f(s) 2 f'(') where ~ P and s' ~ F
(2)
implies that there is such that fr- f on the support of P.
Note that we require the identity h f" o f only to hold on the support of IP because, for complex classes F, there in general no unique extension of f beyond the support
and without data the extension cannot be identified. We do not always make this explicit in the following. Put differently, identifiability means that given observations of x = [(s) and knowledge of (F, PF. we can find such that y related by a symmetry transformation in S. We discuss in Appendix [how to identify the set S and how spurious solutions to the identification problem can be constructed. In the following, it will be convenient to use the notation f.P' which denotes the push-forward of the measure along the function f. We refer to Appendix for formal definition, but note here that the distribution of f(s) equals f.P whenever $ ~ P. Therefore, can be equivalently written as f.P = f.P..
S. in particular the reconstructed sources s' = g(x) and the true sources are
We illustrate Definition I through the well known example of linear maps
Fin (f R I is linear and invertible),
(3)
i..., x. = As for some invertible matrix T m We further define
Pin = (P MM(RR most one component of P Gaussian);
(4)
Sin = (PA P is permutation matrix and A is a diagonal matrix);
(5)
is easy to check that S is group. Then the following identifiability result for Fiin is well known. Theorem 1. (Theorem in (8) The pair (Fin PIn) is identifiable up to Sun-
This result optimal as the ordering and scale of the s; is unidentifiable and the restriction one Gaussian component is required to avoid linear MPTs of multivariate Gaussians. We provide a proof of this result in Appendix D as this serves as a preparation for the more involved Theorem below.
An important observation which also made in 22 is that with minor differences also consider the case where f Rdd > M maps to d-dimensional Riemannian manifold M. An
A diffeomorphism is differentiable bijective map with differentiable inverse.

---

Table 1: Overview of new identifiability results. Note that Identifiable implies Locally identifiable and if Locally identifiable does not hold neither of the other two properties can hold.
Function class
Identifiable (Def. D
Locally identifiable (Def. 55)
Gaussian only
spurious solution
Linear
Conformal
(Thm. &
Orthogonal
/
(Thm.
X
(Prop.
Volume preserving
(Thm.
X
General nonlinear
X
X
(Lemma X
important example for this setting is the case where M C R" is submanifold of a higher dimensional Euclidean space. This covers the standard setting of unsupervised representation learning where high dimensional observations (often images) are created from low dimensional factors of variation mirroring the well known manifold hypothesis 150]. Note that this setting essentially covers the case of undercomplete ICA, where we consider f RR + R" with assume that we already know the submanifold M that f maps to. This manifold can, however, be identified from the observations the data distribution. To avoid technical difficulties we assume that the manifold is already known. Note that we restrict our attention the case where the factors of variations are parametrized by Euclidean space. An extension to product manifolds and combination with the approach in [2] is an interesting question left to future work.
d. The only difference is that we
f() under minor regularity assumption on f and the support of
In the next sections we discuss our results on identifiability of ICA for different function classes. An illustration of the considered classes can be found in Figure They all characterized by local condition on their gradient. Previously, in [222] it was shown that the function class of local isometries identifiable. Local isometries have been used frequently in machine learning, and more specifically in representation learning 50 Our main results consider two generalizations of these function classes, conformal maps and orthogonal coordinate transformations. Conformal maps preserve angles locally and have been used in computer vision A6 51 [8]
For d = 2, conformal maps essentially consist of all biholomorphic mappings of simply connected open domains of the complex plane, and thus constitute a "large". non- mon-parametri family, as a consequence of the Riemann Mapping Theorem (38.. For 2 3, Liouville's Theorem implies that this class contains relatively "few" functions in fixed dimension (i.e., mapping Rdd -> RR,), in the sense that it is parametric family, with parameter dimension quadratic in d (see Theorem Z). However, this is rich class when the target space is higher dimensional than the domain (i.e., Rd - d < n). OCT are even more general class that were motivated based on the principle of independent causal mechanisms in [[7] Notably, it contains all conformal maps precomposed with nonlinear entrywise reparametrizations of the source components (see Corollary D. It is however much larger, as one can for example concatenate arbitrary functions from the large family of 2d conformal mappings to obtain higher dimensional OCTs. Moreover, many works showed that training VAES promotes orthogonality of
the columns of the input Jacobian [45] 55
47 and this has been empirically shown to be good inductive bias for disentanglement. Indeed, these algorithms are widely used in representation
33
40
learning and often recover semantically meaningful representations
4
3 Results for conformal maps
Our first main result is an extension of Theorem T to conformal maps. conformal map a map that locally preserves angles, i.c. locally it looks like a scaled rotation. It can be shown that this is cquivalent to the following definition.
Definition 2. (Conformal map) We define for domains Ω C Rd the set of conformal maps by Feonf = {f (2, Df(x) = (z))0(z)} where A: Ω2 + R \ {0} is a scalar function and 0: > ((d) a map to orthogonal matrices (i.e., O(x) = (zz)
All our results also hold for the more general class of conformal maps f : Ω + M where M is a Riemannian manifold. The complete definition be found in Appendix For convenience define signed permutation matrices by
Perm+ (d) = (P E R ddxd
Q E Raxd such that Qij = |Payl is permutation matrix),
(6)

---

Lec. the set of matrices whose entry-wise absolute value is permutation. Later we will also use the notation (iag(d) and Perm(d) for d diagonal and permutation matrices, respectively. We define
Sconf = (x RPx + a where E Perm (d)m (d), E RR RR,KK K. E (0)}
(7)
and
Peonf = p n Plins
where
Pi PI (R()) there is 0 0 C IR open, s.t. has positive C² density on O}.
While this condition might appear a bit technical it actually only rules out pathological cases like the cantor measure or densities which are nowhere differentiable and probably it could be relaxed further. In particular P1 contains all probability measures with piecewise smooth densities. Then the following identifiability for conformal maps in dimension > holds.
Theorem 2. For d > 2, ICA with respect to the pair (Foont, Peont) is identifiable up to Scouf-
This means that can identify conformal maps up to three symmetries, namely constant shifts of the distributions, rescaling of all coordinates by the same constant factor, and permutations of the coordinates. The proof is in Appendix E The main ingredient in the proof is that conformal maps in dimension > are very rigid and can be characterized explicitly, as we will discuss Section Ø
We remark that it might be more natural to not fix the scale of the sources and allow arbitrary coordinate- wise rescaling. The result can be easily extended to accommodate this. We define
Steparam = {g: Rd + R2 = P h where Perm+ (d) and Rd Rd with h(x) = (hi
(9)
for some h2 E (R, IR) with h4 > 0}.
is easy to see that Sreparam is a group. We define the class of parameterized conformal maps by Fr-conf = (f o h If Foont, E Steparam n C3 (S2, M) and then get the following Corollary. Corollary 1. For d > 2, ICA with respect to the pair (Fr-cont; Pront) is identifiable up to Sreparam if we assume addition that the observational distribution cannot be expressed as J.P for some E Foout and IP E 2MMiBB (RR) which has least two Gaussian components.
The additional restriction on the observational distribution is clearly necessary to exclude the non- identifiability of Gaussian distributions.
For dimension d 2 it was shown [25 that conformal maps can be identified up rotation when fixing one point of the conformal map (setting f0) = 0). The authors also claim, without proof, that the remaining ambiguity can be removed for typical probability distributions. We extend their result by removing the condition that one point is fixed and prove full identifiability with a minor assumption the involved densities. We define the following set of probability measures R²
P P
PP Pi & P2 E M(R) s.t. supp(P.) is bounded interval I, and
(10)
P; has density bounded above and below on 7;)
Then we get the following result.
Theorem 3. For d 2, ICA with respect to the puir (Founf Prontz) is identifiable up to Scoof-
This means that we can identify conformal maps compact domains in dimension 2 up to shifts, permutations coordinates, and scale. Note that we can also identify conformal maps IP has full support 2 using the same proof for d > 2 (see Lemma 2 in the supplement) and extension as in Corollary is possible. The proof of this result is in Appendix
B
Results for orthogonal maps
Recently, in [17], the more general class of OCTs was considered in the context of ICA. They referred to orthogonal coordinates as IMA maps, referencing to independent mechanisms. This nomenclature was motivated by the causality literature and we refer to their paper for an extensive motivation and further results. As we focus on theoretical results for this function class we stick to the more common term of OCTs. Orthogonal coordinate transformations are defined as the set of functions whose derivative have orthogonal columns, i.c., the vectors 0. and a f are orthogonal for j.
Definition (OCT maps) We define for domains Ω C TRd the set of OCT maps (orthogonal coordinates) by FooT {f (2, Df(x) Df(x))Df(s)) Df(x)
Diag(d)).
5

---

OCTs constitute rich class of functions. The study of OCTs has a long history and already in the 19th century the structure of all OCTs defined in a neighbourhood of a point were characterized 10 4 Later, also the set of global orthogonal coordinate systems on Rd was characterized [28] As those results are not easily accessible we will provide here a simple argument showing that OCTs constitute rich class of functions. We first note that FOCT contains the above F--con as functions in the later class have a D1(x) that takes the form of a Jacobian of a conformal map whose columns are rescaled by derivatives of the entry-wise reparametrizations, such that they remain orthogonal. However, FOCT is much bigger than -o-c For example, take a n-tuple ...,, of arbitrary injective 2D conformal maps fb S2 + R E Foont where S2 C IR2 and build the "concatenated" map Soone 92 X ... x f2n - R22n given by
fcone (s) (s1, 82))...... (12n-1,, 820), (12n 82n))
(1I)
The Jacobian of Scone is block diagonal, such that columns associated different diagonal blocks are obviously orthogonal, and columns pertaining to the same k-th diagonal block are orthogonal by conformality of fa. With such a construction, that we can also further post-compose with transformations in Foont on 2",", we can thus build a large non- -on-ammmcttic subclass of FoeT on ?" This construction can be easily adapted to the case of odd dimensions.
Setting for identifiability with OCTs. OCTs can also be generalized to maps whose target is a d-dimensional manifold (see definition in Appendix B). and the following results will also apply to such case. First, we note that we can only hope to identify a mechanism f E FocT up to coordinate-wise transformations and permutations, i.e., maps in Steparam- Indeed, if f E FOCT and 9 Steparam then f o g FOCT. Thus, in particular f-l (f given observations from f.P we can identify f and P only up to E Steperam- More precisely, for any (sufficiently smooth) F' there is I" such that f.P = /'P' where we pick such that P' - P
This implies that
As the distribution of the is not identifiable, we map it to fixed reference distribution that we choose to be the uniform distribution on 1)4. We introduce the shorthand C. = (0, 1) for the standard open unit cube (exclusion of the boundary will be important for our result) and denote by the uniform (Lebesgue) measure on Ca. For fixed base measure v the symmetry group is reduced to permutations and reflections, i.e., maps in P E Perm (d).
We conjecture that for 'typical' pairs (f. IP) E FOCT X POCT, ICA is identifiable with respect to Steparam (with suitable definition of Poer, e.g., Poet = Peont). However, we leave precise statement for future work. Below we will show weaker notion of identifiability for OCTs, but, before that, we first exhibit exceptional classes of spurious solutions for ICA with OCTs.
Spurious solutions for ICA with OCTs. We first note that just as for linear maps and conformal maps (see Thm. D Gaussian distributions can hamper identification. This is because arbitrary measures with a factorized density can be pushed forward into multivariate Gaussians using a suitable wise transformation.
Fact 1. Let P be probability measure I with bounded density and cumulative distribution function Fp. Denote the cumulative distribution function of the standard normal by Fx. Let hp = FN Fr. Then (hp).P PP has a standard normal distribution.
This implies that if = P1 ... @ Pa and x = f(s) (he, (s1).... he (sa)) then f.P follows a standard normal distribution. In particular, for every A E O(d) the distribution of Ax is standard normal and its components are independent. Note that, in contrast to conformal and linear maps, is not sufficient to exclude Gaussian source distributions: due to the flexibility of the function class, we have to exclude that the pair (f. IP) has a Gaussian observational distribution f.P. Next we show that a more general construction using OCTs is possible.
Proposition 1. Let PP be a rotation invariant distribution on R.d with smooth density. Then there is a smooth and invertible (on its image) function f C.a - Rdd with f Focr such that f.v P.
The proof of this result can be found in Appendix F The main idea in the proof is that d-dimensional polar coordinates do the trick up to coordinate- wise rescaling. This proposition implies that the entire family {fR- = R o fl R E O()) satisfies (fR).v= = f.V and, by definition of FocT we have
'This is possible if both distributions have compact connected support where they have smooth positive density. We ignore difficulties associated with unbounded support non-regular measures here

---

Po
Cisarvation space
Figure 2: Illustration of radius dependent tations as defined in Lemma] The left figure shows the initial sources. In the right figure radius dependent volume preserving transfor- mation was applied (see Appendix
Figure Smooth invariant deformations. The blue grid indicates the transformation f. while the green grid shows the deformed map e,, Outside the red box d; ft holds (see Definition
JR E FooT because I E FoOt implies R o f E FOCT. In particular all inverses on Jn recover independent sources because J.v) v and BSS is not possible in a meaningful way this (special) case. The construction in Proposition D gives spurious solutions for substantially more observational distributions than just the Gaussian (this is indicated Table [T. Nevertheless we do not view this as a general obstacle to identifiability results for OCTs for two reasons. Firstly the spurious solutions only apply to carefully chosen pairs of function and base measure such that we obtain a still very non-generic (radial) observational distribution. Moreover, the function constructed in Proposition 1 cannot be extended to C. [0, 1] such that it remains invertible (in the language of differential geometry: I is only an immersion not an embedding of submanifold). In this sense the main message of Proposition I is that an identifiability result for FOOT needs to contain assumptions ruling out those spurious solutions (just like Gaussians are excluded for linear ICA).
Local Stability of OCTs. We now give partial results towards identifiability of OCTs. While we do not prove general identifiability for this class, we demonstrate their local rigidity: OCTs cannot be continuously deformed to obtain spurious solutions. This is in stark contrast the general nonlinear case, which we will discuss for comparison below. Actually, we will show the following slightly stronger statement. Suppose we know some initial data generating mechanism fo because we have, c.g., access to samples (s, fs)). Now we assume that the mixing fo depends smoothly on some parameter which could be, e.g., time or an environment. Then we can identify fa ft Focr for all f and given access to samples (s). This would not be true if fi 's would be unconstrained nonlinear mixing functions. We now make these statements precise. We first define smooth deformations of mixing function.
Definition 4. (Smooth invariant deformations) Let F be some function class. Consider family ((-TT,T) T) x Rd, for some T > and smooth, d-
of differentiable transformations dimensional manifold M such that 4 o(t, :) is diffeomorphism onto its image. We call 4, smooth invariant deformation (.) E F for L.
An illustration of this definition be found in Figure Based invariant deformations now define a local identifiability property of ICA given function class.
Definition 5. (Local identifiability of ICA). Consider a function class F. Let ft be a smooth invariant deformation in JF that is analytic in t. Let d be another smooth invariant deformation in F analytic in t such that Lo = fo and (b:).1/ = (V for all t. Assume that there is E > such that (s) fe(s) if dist(s, aCa) < Then these assumptions imply &; = ft for all f. We call F locally identifiable it is locally identifiable at (fi, v)) for all analytic local deformations ft-
say that ICA in F is locally identifiable at (fr,v) if
We remark that our notion of local identifiabiliiy is indeed similar to the concept of local identifiability in the context of parameter identification of neural networks [3]. Local identifiability of a function class means that we can identify smooth deformations ft E JF from some initial mixing fo E F if fo is known on the whole latent domain, and the behaviour of fr close to the domain's boundary is known for all I. This can be interpreted plain identifiability in the concept drift setting 522 (15). Formulated differently, it means that an adversary cannot smoothly deform the function so a subset whose boundary is away from the boundary of the domain, such that the outcome is ambiguous given
7

---

the resulting observational distributions. An experimental illustration of this setting and Theorem below can be found in Appendix [[] The notion of locality in Definition 5 should be understood as "non-global" and notably does not imply restrictions to a small neighbourhood, as local properties often do. The non-globality manifests itself in two ways: we consider smooth transformations of the ground truth, i.e., small changes of the data generating function fo and in addition we assume that the changes are not everywhere in s, i.c., sources close to the boundary are kept invariant. An extension to other source distributions beyond fixed (and possibly changing with t) is possible but not necessary in the context of OCTs as explained above. We can show that local identifiability holds true in FOCT.
Theorem 4. The function class FOCT is locally identifiable.
The proof, in Appendix M introduces new tools to the field of ICA. The main idea is to consider the vector field X that generates the deformation D, and then rewrite the assumption as systems of partial differential equations for X. The proof is then completed by showing that the only solution of this system vanishes. Let us state one simple consequence of this theorem.
Corollary 2. Let D be smooth analytic invariant deformation in Foct such that (.t) fav for all t and there is E > 0 such that (s) = fs) if dist(s, aC,) < E. Then D = f for all t.
Let us reiterate what this corollary shows: we cannot smoothly and locally transform the function f such that (1) the observational distribution remains invariant, i.e., equal to f.V, and (2) the deformed functions remain OCTs.
At a high level this result suggests that OCTs can be identified if we know f close to the boundary of the support of 8, e.g., by having, addition to unlabelled data (s), labelled data (s, [s)) for those where coordinate s is extremal. Note that we actually do not show this result as there might be further solutions which are not connected by smooth transformations. We expect that those results can be generalized substantially. In particular, we conjecture that for "most" functions f the boundary condition can be removed thus giving a stronger local identifiability result up to the boundary of the support of As partial result in this direction we prove the following theorem.
Theorem 5. Let f Ca + Rd be given by f(z) = RDx, with R E O(d) and D Diag(u1: ..... Ma) where Pr are i..dd samples from a distribution supported on the positive reals R2 which has a density. Suppose that 4, is a smooth invariant deformation in FOCT such that vo = f, (f:) v = fav, and o is analytic in t. Then for almost all μ4 (i.e., with probability one) this implies b = f on Ca. i.e,
Dr is constant in time.
In Appendix F we show that this theorem follows from a slightly stronger result stated as Theorem TO which has a similar proof Theorem 4 We do expect that the conclusion of the Theorem actually holds for all μr not just almost all, but we are unable to show this.
Comparison with ICA for general nonlinear functions. Let us emphasize that those results are non-trivial as they establish a large difference between ICA with generic nonlinear maps and ICA with OCTs. To clarify this. we state that no result similar to Theorem 4 holds without the assumption that 4 FOCT. Put differently, the function class F Fuoulimear not locally identifiable.
Fact 2. Suppose f Ca IRd is diffeomorphism on its image. Then there are uncountably many smooth deformations &, of (ff, v) such that (i) (L.) = f.v and (ii) there is an > 0 such that (s) = (s) whenever dist(s, aCa)= E.
For completeness, we provide general construction that is close to our proof of Theorem in Appendix Cin in the supplement. very clear construction for this result was given in (25]
Lemma (Smoothly varying radius dependent rotations (see (25)). Let R : IR x R2 + O(d) be smooth function mapping orthogonal matrices and let Ca- Assume that R(t.r) = id for r 2 dist(a, @Ca). Then the map s + lRR (1, (t, 8) = R([s a), t)(s a) + a preserves the uniform measure v on for all so that f o ha, (t, s) 2 fs) for all if is distributed according to V.
An illustration of this construction is shown in Figure 2 (see App. C for details). Clearly, by
concatenation this allows to create vast family of spurious solutions. Note that those solutions are excluded when restricting to OCTs which is a corollary of Theorem
Corollary 3. Suppose J E FOOT. Let b be the smooth invariant deformation defined by t. fo (l. where in is as in Lemma If t. FOOT for all I this implies that hn, (1, 8) = 8 and 4 f for all t.
8

---

We now summarize our view on the results of this section informally (we do not claim that the statements below regarding (infinite dimensional) manifolds can be made rigorous). For given data generating mechanism (Jo, v) we expect that typically the set of all solutions MocT { E FOCT f.v = (fo).u} C FOCT is a zero dimensional submanifold, i.e., consists of isolated spurious solutions and we prove this when fixing the boundary (see Theorem B while the corresponding submanifold of general nonlinear spurious solutions Mnonlinear = {f : Cd 7 IK f.u= = (fo) is
infinite dimensional even when requiring f(s) = fo(s) close to the boundary of Ca.
5 Results for volume preserving maps
Let us finally consider volume preserving transformations. For Q Rd, those are defined as the set of functions Fvol = {f : S + gd det (z) = for all x E S2}. Invertible volume preserving deformations have the property that they preserve the standard (Lebesgue)-measure) À the sense that f.Ao A Recently it was proposed that volume preserving functions are a suitable function class for ICA. Here we show that those functions are not sufficiently rigid allow identifiability of ICA in the unconditional case. Note that Lemma T and Fact 2
already show how to construct spurious solutions for the case that the base distribution the uniform measure v. However, for an arbitrary distribution P this is slightly more difficult because we need to find maps y that preserve IP, ie., g. PP = P., and are volume preserving, i.e., preserve the standard measure.
Theorem 6. Let be twice differentiable probability density with bounded gradient. Suppose that I = f(s) where the distribution P of s has density p and a diffeomorphism with det Df 1 for Rd. Then there a family of functions Se R R Rdd with fo = f and Se # for t + 0 such that det Df((z) = and (ft),.P- = f.IP.
The proof and an illustration are in Appendix G. is based on the flows generated by suitable explicit vector fields. As those flows can concatenated we obtain a large family of spurious solutions. We think that the approach used here is a powerful technique to construct counter-examples to identifiability in ICA. As local identifiability is weaker than identifiability we conclude that ICA in Fvol is not identifiable. Note that this is even true when we know the distribution of s. A rigorous version of this statement is that if (Fvol, Pvot) is identifiable with respect to Syol then Svol contains functions mixing coordinates 8; 8j for + j (i.e., there is h E Syol such that 0,0,h h + 0).
6 Relation to rigidity theory
Our results rely on rigidity properties of certain function classes. Rigidity refers to the property that a local constraint on the derivative of a function implies global restrictions on the shape of the function. These type of results are of interest in the field of continuum mechanics ( where, e.g.. the condition Df E (Od)) is used to describe deformation of rigid solids and the condition Det Df = 1 to describe incompressible fluids.
We now state a well known rigidity result for conformal maps that is the main input in the proof of Theorem 2 This result shows that there very few conformal maps in dimension d >>> 2.
Theorem (Liouville). Let d > 2, Ω C Rd open and connected, f f. > IRd conformal. Then f(x) + AAAr a) als
(12)
where b, a E R a R, A Odd) and E E (0,2).
Originally this result was shown by Liouville 3d], a modern treatment is (27] In particular, this shows that conformal maps are (up translations) rotations or rotations followed by an inversion.
To illustrate the strength of Theorem / we compare it to the setting of volume preserving maps which satisfy similar rigidity property. Intuitively the different rigidity properties are already apparent from the connection to solids, which can merely be rotated and shifted, and fluids which can also be stirred leading chaotic deformations. Rigorously the different behaviours can be clarified by the observation that conformal maps have a finite number of parameters and thus a finite number of constraints (e.g., of the form f(x = y) allows to identify them. In contrast volume preserving maps cannot be identified from finitely many constraints as the following proposition shows.
Proposition 2. For d 2 and {=1..... En. U1 ... yn Rd, all pairwise different, there is volume preserving diffeomorphism f RR + Rd, f E Fvol such that fz:) Ye.

---

Let us emphasize that the different rigidity properties are not at all surprising when arguing based on degrees of freedom or numbers of constraints. While volume preserving maps enforce only a single scalar constraint on the Jacobian DJ the condition for conformal maps gives n(n + 1)/2 constraints on the Jacobian.
Let us finally comment on OCTs where the picture is not as well understood. As discussed in
4 it is known T0 ! that OCTs constitute rich, non-parametric class of functions and therefore OCTs are much more flexible than conformal maps. We illustrated this with example OCT constructions Section leveraging 2D conformal maps. Nevertheless, it not known if and what 1
Section
rigidity properties can be derived for OCTs. However, our results suggest that the additional measure preservation condition in the context of ICA gives enough rigidity to (almost) give identifiability of ICA. In this sense OCTss might be good function class for ICA as it is rich enough to allow complex representations of data while at the same time being sufficiently rigid to still provide a notion of identifiability whose strength remains to be determined.
7 Discussion
ICA is long known to be identifiable for linear maps, baring pathological cases, and highly non- identifiable for general nonlinear ones. Surprisingly, similar results for function classes of intermediate complexity remain scarce. In this work address this question with several identifiability results for different function classes. Our first main result is that ICA is identifiable in the class of conformal maps (up classical ambiguities). This considerably extends previous claims, limited to a specific 2D setting (25). and ruling out several families of spurious solutions [17. On the negative side we show that the ICA problem for volume preserving maps admits large class of spurious solutions. Finally, we show that OCTs satisfy certain weaker notions of local identifiability.
In our proofs, we draw connections to methods and techniques that, to the best of our knowledge, have not been used in the context of ICA before. We relate the identifiability problem in ICA to the rigidity of the considered function class F and use tools from the theory of partial differential equations. These techniques have been applied very successfully to the analysis of elastic solids 76 and we believe that there are many applications of these methods in the field of ICA.
While the main focus of current research after the seminal work of Hyvärinen et al. (2 is on the auxiliary variable case, there are three reasons to consider unconditional ICA. Firstly. it is fundamental research question that is, as illustrated by our results, deeply rooted in functional analysis. Secondly there is high application potential for completely unsupervised learning without any auxiliary variables, as the corresponding datasets do not require labelling or specific experimental settings. Thirdly, it is very likely that the techniques can be generalized to the auxiliary variable case.
Another important open problem assessing the type of constraints on ground truth mechanisms, encoded by function classes, that are relevant for real world data. It is plausible that those mech- anisms are typically much more regular than generic nonlinear functions. Recently, Gresele et al. 17 suggested, based on arguments from the causality literature that Foet is natural class for representation learning (and our results show it also has favourable theoretical properties), but this will require experimental confirmation on real world data.
Finally, central question from machine learning perspective is the ability design learning algorithms that can train LVMs with identifiable function class constraints. Interestingly, Gresele et al. 17 showed that OCT maps can be learnt using closed from regularized likelihood loss, thereby providing, supported by our result, a full-fledged identifiable nonlinear ICA framework.
Acknowledgments and Disclosure of Funding
We thank Luigi Gresele and Julius von Kügelgen for helpful discussions and feedback on an earlier version of this paper. We thank the anonymous reviewers whose comments helped us to make the paper more accessible. This work was in part supported by the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011818039B) and the Machine Learning Cluster of Excellence, EXC number 2064/1
Project number 390727645.
10

---

References
[1] L. Ahlfors. Complex Analysis: An Introduction to the Theory of Analytic Functions of One Complex Variable, Third Edition. AMS Chelsea Publishing Series. American Mathematical Society, 1979. ISBN 9781 4704676777.
[2] Y. Bengio, A. C. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-18) 1828, 2013. doi: 10.1109/ TPAMI.2013.50. URthtts::///dd org/ /trgg/1..r
2013.50
[3] Bona-Pellissier, F. Malgouyres, and F. Bachoc. Local identifiability of deep reL.U neural networks: the theory. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htllattp://pperreeeeeee////oooro
+3cHWtbbLY
[4] E. Cartan. La géométrie des espaces de Riemann. Gauthier-Ville 1925. //eudml.. org/ wwt////g//////o/ooo//o///./o..//
URL http:
[5] T. Q. Chen. X. Li, R. B. Grosse, and D. Duvenaud. Isolating sources of disentanglement in variational autoencoders.
S. Bengio, H. M. Wallach, Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2615-2625, 2018. URL https: haw.aaad222atttttt.hhhaa sangg.neurip.. Abstract.htm
[6] P. G. Ciarlet. Mathematical Elasticity: Volume II: Theory of Plates. ISSN. Elsevier Science, 1997. ISBN 9780080535913.
[7] G. Ciarlet. Mathematical Elasticity, Volume I: Phree-Dimensional Elasticity. Classics in Applied Mathematics Series. Society for Industrial and Applied Mathematics, 2021. ISBN 9781611976779.
[8] P. Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287-314, 1994. ISSN 1684. doi: http:///....oo/o//..oooooo616000 URL https
//www. www.ssciencedirecc. com/ httpa//www................oa///a Statistics.
Higher Order
[9] B. Dacorogna and J. Moser. On a partial differential equation involving the jacobian determinant. Annales de 1 P. Analyse linéaire, 7(1):1-26, 1990. URLhttpp //w::///wwwwww/./ numdam. org/
item/ Atem/AIHPC. AIHPCAHHCCCCCCCC 1990 7 7_1_1____11
[10] G. Darboux. Leçons sur la théorie générale des surfaces. 1896.
[11] D. L.. Donoho and C. Grimes. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences, 100( 100(10)5551-5554 2003. doi: 10. 10.1073/pnas.. http3//aa..131596111 103159610.. URL https::/wwit pnas wrg/doi/abs.....oooooooooos.o 1073/ pnas. 1031596100
[12] D. R. Dunninger and E. C. Zachmanoglou. The condition for uniqueness of solutions of the dirichlet problem for the wave equation in coordinate rectangles. Journal of Mathe-
matical Analysis and Applications, 20(1):17-21, 1967. ISSN 0022-247X. doi: https://doi. 11201001116000222277746777//0189 brricle/p11/0022277777771011
https:: httpas///www.scccceeerreett wwweeeectttccmceeeeeeeeeeeeeeese www.sciiencaaaaoooocccccccccooce
[13] C. Durkan, Bekasov, I. Murray, and Papamakarios. nflows: normalizing flows in PyTorch, Nov. 2020. URL htt.ttttt/:/oooo..
http://..022222e2ooooo.4 4296287
429
[14] I.. C. Evans. Partial differential equations. American Mathematical Society, Providence, R.I.,
2010. ISBN 9780821849743 0821849743.
[15] J. a. Gama, I. Zliobaitundefined A. Bifet, M. Pechenizkiy, and A. Bouchachia. A survey on concept drift adaptation. ACM Comput. Surv. 46(4), mar 2014. ISSN 0360-0300. doi: 10.1 10.1145/2523813. URL https:://oi.. org/ /0. 1145/2523813

---

[16] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Networks. arXiv: 2666 [cs, stat), June 2014. URL http://arxiv.orre
1406. 2661 arvi: 1406.2661.
[17] L. Gresele, J. von Kügelgen, V. Stimper, B. Schölkopf. and M. Besserve. Independent mecha- nism analysis, a new concept? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual.
pages 28233-28248, 2021. htRthttps:/ p wtw.eeeediinnnn..rrpss..s.. edd
wtt.a///wwoooooooaaaaot....oco.. Abstract.hctt html
[18] X. Gu. Y. Wang. T. F. Chan, P. M. Thompson, and S. Yau. Genus surface conformal mapping and its application to brain surface mapping. IEEE Trans. Medical Imaging, 23) 949-958, 2004. doi: (L99/TM..004483 2004. 831226
URL https://doi http:////ooo..///////// wwt.///../////.0/1..11//
[19] S. Harmeling, A. Ziche, M. Kawanabe, and K. Müller. Kernel-based nonlinear blind source separation. Neural Comput., 15(5): 1124, 2003. doi: 10.1162/8996600762266777 URL htl.https://dd org. 10. 1162/ 1111/08997660007622222222
[20] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
http://ppenneeieee.eeeefrrrrrr
[21] Higgins, D. Amos, D. Pfau, S. Racaniere, I. Matthey, D. Rezende, and A. Lerchner. Towards definition of disentangled representations, 2018. URL httpp:///axxiv.or//bbbb..
02230
[22] D. Horan, E. Richardson, and Y. Weiss. When is unsupervised disentanglement possible? In M. Ranzato, A. Beygelzimer, Dauphin, P. Liang, and W. Vaughan, editors, Ad- vances in Neural Information Processing Systems, volume 34, pages 5150-5161. Curran Associates, Inc., 2021. URL hatps://prooeedingss www.ineeriiscoo httpp////cccppppp///////// 29686c6449c90e2490 wtw.:44f11190000000em LF09a0a400225a- Paper. .dd
[23] A. Hyvärinen and H. Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Gar- nett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3765-3773, 2016. URL http:tt//pprrcceehiinng htw.ooeeeeddinn...nnie...e.... wwwprrisscccppapreeeeee
Abbttti11--AAtttttt.... Abstract.
[24] A. Hyvärinen and H. Morioka. Nonlinear ICA of temporally dependent stationary sources. In A. Singh and X. J. Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Slutistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL. USA, volume 54 Proceedings of Machine Learning Research, pages 460-469 PMLR, 2017. URL http::/ppocceeiingg wttp:/s...rrrressseeeeeeeee
mlr, prw.s ptras//554/hyariiennt1 html
[25] A. Hyvärinen and Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3))429 1999. doi: 10.1016/S0893-600 (0089)98))) 00140-3. URL httpss://ooi org/100///. 11161/00.111188888888 6080 (014000-333
[26] A. Hyvärinen, H. Sasaki, and R. E. Turner. Nonlinear ICA using auxiliary variables and gener- alized contrastive learning. In K. Chaudhuri and M. Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statisties, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 859 868. PMLR, 2019. URL. httl:ttt:://poceediiiii sereeeiiins............mm.mddmee mlr. www.epreessvvve httpa///////yyyrreeneeeeee html
[27] T. Iwaniec and G. J. Martin. Geometric Function Theory and Non-lineur Analysis. Oxford mathematical monographs. Oxford University Press, 2001. ISBN 1780199509295. URL
https://books. google. de http://ee.ee.ooo.oooo/oo///EPPE.
12

---

[28] E. G. Kalnins and W. Miller. Separation of variables on n n-dimensional riemannian manifolds. i. the n-sphere sn and euclidean n-space m. Journal of Mathematical Physics, 27(7):121-1136. 1986. doi: 1.270888 URL https://doi. org/10. 1..0..063/1.. 00063/1..27001 527088
[29] I. Khemakhem, D. P. Kingma, R. P. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In S. Chiappa and R. Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. volume 108 of Proceedings of Machine Learning Research, pages 2207-2217. PMLR, 2020. URL WRLLLtttp//rrccednnngg mlr press/v108/
khemakhem22a. html
[30] H. Kim and A. Mnih. Disentangling by factorising. J. G. Dy and A. Krause, editors, Proceed- ings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, URL http://proceeddingg mlr press/ vt/////000///m /kim1bb
pages 2663-2663. PMLR, 2018.
html
[31] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
7 //arxiv. larxiv.org//a www.//gg//////e/eee/..oe..eeeee/ /4t./.11....0 6980
[32] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. arXiv:1312.6114 [cs, stat], May 2014. URL htLphttp:/// //axxi arxiv wrt/ 111///1122.. 6114 arXiv: 1312.6114.
[33] A. Kumar and B. Poole. On implicit regularization in B-vaes. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5480-5490. PMLR, 2020. URL http:tt////cceeee///
mlr prres/vvi19 http//1111//xrreeccc// k0mar20d html
[34] A. Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL. http:tttt:///pereeeetee
nett/EoruumiiieHH11CCCC..
[35] J. M. Lee. Introduction to Smooth Manifolds. Graduate Texts in Mathematics. Springer, 2003. ISBN 9780387954486.
[36] J. Liouville. Extension au des trois dimensions de la question du tracé géographique. Note VI, pages 609-617, 1850.
[37] F. Locatello, S. Bauer, M. Lucic, G. Rätsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Con- ference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4114 4124. PMLR, 2019. URL http:://proceeddngg/ mlr press/
pitess//v7///e ht////99/1//1/ http//ooatelll1a...m html
[38] W. F. Osgood. On the existence of the green's 's function for the most general simply connected plane region. Transactions of the American Mathematical Society, 1(3):310-314 1900.
[39] G. Papamakarios, F. T. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res., 22:57:1- 57:64, 2021. URL http:// jmlr. www.oo////oooooo........ wwppps/pppppppa///pppppppp 2tt//////1911//1 1028. html
[40] Reizinger, L. Gresele, J. Brady, J. V. Kügelgen, D. Zietlow, B. Schölkopf, G. Martius, W. Brendel, and M. Besserve, Embrace the gap: VAEs perform independent mechanism analysis. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttps://openreviiieeece www.:nneetooonnninnnnntt wttp://wwwweeoooooooooo/oooaAoaa
[41] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference
13

---

Proceedings, pages 1530-1538. JMLR.org. 2015. URL http://proceedingg mlr press/ v37/rezende15. html
[42] K. Ridgeway and M. C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Sesa-Bianchi and R. Gar- nett, editors, Advances in Neural Information Processing Systems 31: Anmual Conference Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 194, 2018. URL http:: htttt:::///poccenngggs. www.nerrippe marrrpp.... htt.cc//ppppa hwpp:/apppr///e 2018
haal/222460022866666666666232282 Att.saaa html
[43] M. Rolínek, D. Zietlow, and G. Martius. Variational autoencoders pursue PCA direc- tions (by accident). In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2019, pages 12406-12415. Com- puter Vision Foundation IEEE, 2019. doi: 10.1109/CVPP.2011
URL http: Variational.
/ooennaccess thecvf com/ content Bttpet__v_____1111101111111111 Autoencoders, Starredddrrrruuuuuueeeee PCA Sil.lDDirettioo.oo ayc.iiiioonsbb_.cccdceett...... 2019 _aap____prp______ html
[44] T. Roweis and L. K. Saul. Nonlinear dimensionallity reductioon by locally linear embedding.
Science, 29065500):2323-2222 2000. doi: 10.11266sccemmc22205500223333
URL https
/www.science.cr /aw..acieence.or//oo///bbbb wwwpo///////o//ooooo 1126 ww122//ciennee222 290. 2323
[45] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchhrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proc. IEEE. 109(5):11--334, 2021. doi: 10.1109/ PPROC.2021.3058955 URL htttp:://doi.orgeeeeeeeo
2021 3058954
[46] E. Sharon and D. Mumford. 2d-shape analysis using conformal mapping. Int. J. Comput. Vis.,
70(1):55-75, 2006. doi: (1007/s11263-006661222 URL http:::///oi.or/1100 s 611263-006-6121-
[47] J. Sliwa, S. Ghosh, V. Stimper, L. Gresele, and B. Schölkopf. Probing the robustness of independent mechanism analysis for representation learning, 2022. URL https://arxiv
arg/abs/Be /20.:/227..220 066177
[48] Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ICA with general incompressible-font networks (GIN). In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https://openreviie wwt pet./foruumdddryeeeessmmm
[49] A. Taleb and C. Jutten. Source separation in post-nonlinear mixtures. IEEE Truns. Signal Process., 47(10):2877-2220, 1999. doi: 10. 10.1109/78.7906 URL https://doi.org: 1109/78. 790661
[50] J. B. Tenenbaum, de Silva, and C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500) Dec. 2000.
[51] Wang, Y. Wang. M. Jin, X. D. Gu, and Samaras. Conformal geometry and its applications on 3d shape matching, recognition, and stitching. IEEE Trans. Pattern Anal. Mach. Intell., 29 (7):1209 2007. doi: 10. 109/TT/MM..007.1000000 URL httph::t///att aecooi.oorg// 1109/
TPAMI. 2007.
[52] G. Widmer and M. Kubat. Learning in the presence of concept drift and hidden contexts. Machine Learning, (3(1):69-111, 1996. doi: 10.1007/BF001 16900. URL httpttt///oo..oooo org/ 10..107/BF001160
[53] X. Yang, Y. Wang, J. Sun, X. Zhang, S. Zhang, Z. Li, and J. Yan. Nonlinear ICA using volume- preserving transformations. In International Conference on Learning Representationns 2022. URL https::/ htt.:eeeeee.ooorrraaaaaaaaaaaaa.
[54] Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: sparsity and beyond. CoRR, abs/2206.0775l. 2022. doi: 10.4485504aXXx.220007755 URL //doi. org/
18550/arXtae 2206.07751
14

---

[55] D. Zietlow, M. Rolinek, and G. Martius. Demystifying inductive biases for (beta-)var based architectures. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12945-12954, PMLR, 2021. URL http://procceeding mlr. wtt.:mmww...m.........m.mmmmme wwt.prss//v199//eee1oo22.. html
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See discussion (Section and assumptions of our results
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Due to the theoretical focus of this work no direct negative impacts are expected.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all thcoretical results? [Yes] All results stated including all required assumptions
(b) Did you include complete proofs of all theoretical results? [Yes] The proofs can be found in the supplementary material (Appendix D Appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Appendix H
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See end of Appendix H
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [Yes] See Figure and S
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See end of Appendix H
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "lPPt1qod21J", "reviewer": "Reviewer_4HKX", "summary": "Two common ways of avoiding the non-identifiability in nonlinear ICA are 1) conditioning on auxiliary variables, and 2) staying in the unconditional setting but instead restricting the function classes.  This paper focuses on the latter (i.e. unconditional ICA), and studies the identifiability for conformal maps, orthogonal coordinate transforms (OCTs), and measure-preserving maps.\n\nSpecifically, the results are (summarized in Table 1):\n- For conformal maps, the paper extends Hyvarinen & Pajunen 1999's results from d=2 to d>2.\n- For OCTs, the paper proves a weaker notion of \"local identifiability\", and conjectures that ICA would be identifiable for \"typical\" pairs of elements, though the formal statement is left for future work. In addition, local identifiability does not hold for nonlinear maps in general, highlighting the importance of constraining to OCTs.\n- Measure-preserving maps are not locally identifiable and hence also not identifiable.\n\nProof technique wise, the paper draws connection to rigidity theory and tools from PDEs.\n", "questions": "- Thm 3: Noting that $\\mathcal{S}_{\\text{conf}}$ contains a shift, could you explain how is this stronger than the identifiability result in Hyvarinen & Pajunen 1999?\n- Clarification: Thm 8: does $|\\cdot|$ denote the l2 norm?\n\n=== Update after discussion ===\nThe authors have answered this in the reply.", "limitations": "The paper discusses the limitations well.\n\nThere's no direct societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper provides an alternative lens (i.e. restricting the function class) to circumvent the non-identifiability problem in ICA.\n- The paper draws connection to techniques in PDEs. Concretely:\n  - The local identifiability result in Theorem 4 follows from first casting the independence conditions of ICA as second order PDEs and then applying the uniqueness of the Cauchy problem of second-order hyperbolic PDEs.\n  - The constraints in Theorem 5 are because the Dirichlet problem may not be well posed in general and doesn't always have a unique solution, unless under some specific constraints.\n- The paper is clear about technical limitations, provides partial results whenever possible, and provides interesting open directions.\n\nWeaknesses: I have no major concerns about the paper. I should note that I'm not the best judge for the impact of the results and the technique novelty.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U8GJ-MWyxpj", "reviewer": "Reviewer_PqGg", "summary": "This paper studies the identifiability of nonlinear ICA by focusing on different types of function classes (i.e., conformal, orthogonal, and volume preserving maps). In particular, the paper extends previous results of dimension 2 to show that ICA with conformal maps is identifiable for dimension larger than 2. The authors then define the notion of local identifiability and show that ICA with orthogonal maps is locally identifiable. Finally, the authors show that ICA with volume preserving maps is not identifiable. Throughout the theoretical analysis, the authors draw connection to the rigidity theory.", "questions": "As noted in the \"weaknesses\" section above, it is helpful to provide more explanations and empirical studies regarding local identifiability and the different function classes.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The notion of local identifiability is novel to the best of my knowledge\n- Theoretical analysis for different function classes and the connection with rigidity theory are provided and appear to be sound\n\nWeaknesses: Overall, while this paper provides theoretical analysis for nonlinear ICA, in my opinion it will be helpful if there are some empirical studies to illustrate the theoretical claims. Details comments are as follows.\n- Since the concept of local identifiability is new, it would be more appealing to provide more explanations regarding the intuition of local identifiability (e.g., how it specifically contributes to one identifying the desired independent components in practice), and empirical studies to demonstrate its practical applications. This may be necessary for others to better understand the significance of local identifiability. On the other hand, for the empirical studies, how does one evaluate the performance of an algorithm w.r.t. local identifiability?\n- Related to my point above, more explanations (e.g., in what situation one will expect the corresponding function assumption to likely hold) and empirical studies for the function classes will also make the contributions more convincing.\n- While the theoretical analysis of identifiability for conformal map might be potentially interesting, it might be rather restrictive and less likely to hold, and thus might limit its practical applications.\n\n-----\nAfter reading the responses and discussion, including the further comments from the authors and also the other reviewers, my concern about the applicability of the assumption persists.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MTXicisdW7", "reviewer": "Reviewer_WFjz", "summary": "This manuscript extends the previous identifiability result on conformal maps where the dimension is fixed to two. Specifically, The main result is that the Mobius transformation, which is a conformal map where the dimension is larger than two, is identifiable up to trivial indeterminacies. Besides, partial/local results on the orthogonal coordinate transformation (OCT) and unidentifiability of volume-preserving transformation have been presented. No experiments have been conducted.\n\nThe theoretical contributions could be summarized as follows:\n\n1. (Main result) Identifiability of high-dimensional conformal maps.\n2. Unidentifiability of OCTs and volume-preserving transformation.\n3. New notion of local identifiability and its connection with OCTs.", "questions": "Please refer to the above section for the corresponding questions. I will change the score if the related confusion can be addressed during the rebuttal.", "limitations": "The authors discussed the limitations of the work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. The paper is well-written.\n2. Theoretical results are clearly and rigorously presented. The proof seems rigorous and its technique provides novel insight.\n3. The non-identifiability of OCTs and volume-preserving transformations are helpful and consistent with intuition.\n4. Limitations have been clearly and insightfully discussed.\n5. The notion of local identifiability is original and interesting.\n\nCons:\n\n1. The core assumption of the main result, i.e., high-dimensional conformal map, seems to be impossible in practice. In [1], OCTs (orthogonal columns of the Jacobian of the function) have been shown to be helpful to the empirical identification of nonlinear ICA. Although no identifiability result is given in [1], the corresponding intuition of independent mechanism could be found in the real world. However, compared to OCTs, high-dimensional conformal map restrict the Jacobian of the mixing function to be a scaled version of an orthogonal matrix, where the L2 norms of the columns and rows are exactly identical to each other, which seems to be impossible in practice.\n\n2. Lack of discussion of the assumption, especially for its motivation and rationality. In L153, the motivation of OCTs has been referred to [1]. However, as described above, there exist key differences between high-dimensional conformal maps and OCTs, where the former seems to be impossible in practice. Therefore, it may be better to discuss the assumption of the main results, perhaps from a perspective of the chance/scenario of it being true. I couldn't find a related discussion in the current manuscript.\n\n3. Lack of experiments. According to the reasons mentioned above, it is hard for me to check the significance of the main result. It may be of independent theoretical interest to study the identifiability of a specific functional form, i.e., the high-dimensional conformal map (or Mobius transformation). In that case, simulations focusing on the validity/contribution of the proposed assumption should be very helpful to support the claims. In the current manuscripts, no experimental results are presented.\n\n4. Although OCTs (or IMA [1]) and volume-preserving transformation [2] have been proved unidentifiable in the manuscripts, there exists previous work that provides the full identifiability result with assumptions only on the mixing function. For example, [3] prove that, under assumptions of independent influences and factorial changes of volume, unconditional ICA could be identified up to trivial indeterminacies as the same as those shown in this manuscript. The assumption of independent influences seems to be closely related to OCTs and the assumption of factorial changes in volume contains volume-preserving transformation as a special case.  Therefore, it seems that the combination of the principles of the two assumptions discussed in this paper is sufficient for the identifiability of unconditional ICA. Thus, I believe the identifiability result of unconditional ICA in [3] is related to the task of this manuscript, which is currently missed in the discussion of related works.\n\n\n-------\n\n[1] Luigi Gresele, Julius von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, a new concept?\n\n[2] Xiaojiang Yang, Yi Wang, Jiacheng Sun, Xing Zhang, Shifeng Zhang, Zhenguo Li, and\nJunchi Yan. Nonlinear ICA using volume-preserving transformations.\n\n[3] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA with unconditional priors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fc-tOaNEELw", "reviewer": "Reviewer_uZNP", "summary": "This paper gives some deep insights for the identifiability of the limited function classes between linear non-Gaussian ICA and unrestricted nonlinear ICA, including conformal maps, orthogonal maps and volume preserving maps.", "questions": "See above", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is well written and easy to follow. I am no expert in this area, but as far as I can tell, the theoretical contributions appear sound, the quality of writing is generally strong. I believe it work will be a welcome contribution to the identifiability of nonlinear ICA. I am just confused by the real application scenarios of these function classes mentioned in this paper. It would be great if the authors can give a discussion about the real application scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept