Some Remarks on Identifiability of Independent Component Analysis in  Restricted Function Classes
Authors: Simon Buchholz
Venue: Accepted by TMLR ()

Abstract: In this short note, we comment on recent results on identifiability of independent component analysis.
We point out an error in earlier works and clarify that this error cannot be fixed as the chosen approach is not sufficiently 
powerful to prove identifiability results. In addition, we explain the necessary ingredients to prove stronger identifiability results.
Finally, we discuss and extend the flow-based technique to construct spurious solutions for independent component analysis problems
and provide a counterexample to an earlier identifiability result.

Published in Transactions on Machine Learning Research (09/2023)
Some Remarks on Identifiability of Independent Component Analysis in Restricted Function Classes
Simon Buchholz Mar Planck Institute for Intelligent Systems, Tübingen
sbuchholz@tue.mpp mwg.
Reviewed on OpenReview: https: // openreview. net/ for/formm? id- REtKapdky.
Abstract
In this short note, we comment on recent results on identifiability of independent component analysis. We point out error in earlier works and clarify that this error cannot be fixed as the chosen approach is not sufficiently powerful to prove identifiability results. In addition, we explain the necessary ingredients to prove stronger identifiability results. Finally, we discuss and extend the flow-based technique to construct spurious solutions for independent component analysis problems and provide a counterexample to an earlier identifiability result.
1 Introduction
Independent Component Analysis (ICA) is principled framework for representation learning, The goal is to recover independent factors of variation from observed mixture of the sources. It is well known that ICA is not identifiable without additional assumptions on the mixing function (yvärinen & Pajunen, 1999h- On the other hand, is known that for a linear mixing function the independent components are identifiable Comon (1994). So, a natural question whether there are larger function classes such that ICA is nevertheless identifiable within this function class. Recent works investigated the question of identifiability for volume preserving functions (Yang al.] 2022) (in the auxiliary variable setting) and volume preserving orthogonal coordinate transformations (Zheng al. 2022a b). Unfortunately, proofs of identifiability results are error-prone. Given the recent surge of interest in identifiability results for latent variable models, we think that it becomes increasingly important to gain a clear understanding of the proof strategies available and their limitations.
In this regard, the purpose of this note is twofold. First, we point out a mistake in Lemma in Yang el al., 2022) which also used crucially in the proof of (Zheng et al., 2022ab invalidating the theoretical main results of those two works. Secondly, we review a bit more generally techniques to prove identifiability non-identiffabiiity of ICA in restricted function classes.
In particular, we show that their proof strategy is probably too restrictive to show identifiability in inter- esting function classes. We also review some rigidity results that allow us to characterize function classes characterized by a local condition on its gradient, and explain how such an approach could be used to prove identifiability for ICA. Finally, we discuss and extend a recent construction of counterexamples for identifiability of ICA based on the flow generated by suitable vector fields (Buchholz al.) 2022). This allows us to construct a counterexample to the main result, Theorem 1, in Yang et al. (2022). We do not resolve whether a variant of the main result (Proposition 2.1) of Zheng et al. (2022a) which also appears as Proposition 2 in Zheng et al. (2022b) holds true, i.c., whether TCA with volume preserving orthogonal coordinate transformation is identifiable. However, this note clarifies that a complete proof requires more involved techniques.
This work is structured as follows. In Scction 2 we introduce the setting of ICA in restricted function classes and define identifiability in the context of ICA. Then we discuss the limitations of the proofs in the earlier works (Zheng et al. 2022a Yang et. al.] 2022 and sketch potentially more general approaches in Section

---

Published in Transactions on Machine Learning Research (09/2023)
In Scction 4 discuss how the flows of suitable vector fields can be used to establish hon-identiffabiltyy for ICA problems, which generalizes earlier results.
Notation
We denote the set of skow-symmetric matrices and diagonal matrices by
Skew(d) = {A E Rdxd A + + = 0}, Diag(d) = { E gdxd A is a diagonal matrix).
(1)
By (d) we denote the set of signed permutation matrices, i.e.,
Perm+ (A E Rdxd B with by = |a.yl is a permulation matrix].
(2)
We denote the orthogonal and special orthogonal group as usual by
((d) = {Q E dxdd Q Q = Ida), SO(d) = {Q e Rddd QTQQ= = Ida, det Q = 11.
(3)
We use the language of differential geometry freely throughout the lext and refer to any standard textbook for definitions of, e.g. the tangent space of a manifold.
2 Setting
To fix notation and set the stage, we briefly review identifiability of ICA. This material is standard and close to Buchholz et al. (2022). ICA deals with data generated according to
x = f()), p(s) II p((s:)
(4)
where f: Idd - IRd an invertible mixing and p is a probability density with independent components st (also called factors of variation). Without further indicating this, we will always assume that all consid- ered functions f are bijective their image and sufficiently smooth with smooth inverse. The goal is to reconstruct the sources given only the observations up to some tolerable ambiguities, i..., one tries to learn f-1. It is well known that identification in any meaningful sense is not possible for general nonlinear functions f. There are two roads that can lead to identifiaility. One is the restriction of the function class, i.e,, it is possible to obtain (partial) identifiability results by restricting the mixing to be in some function class F. The alternative to consider multi-view settings introduced in Hyvärinen et al. (011 where we have an auxiliary variable u such that conditional on u.
Fu f(sw) Su p(slu)
II (s.lul
(5)
Then can be shown that given a sufficiently diverse set of conditional distributions p(slu) the mixing f can be identified (Hyvärinen et al. 2019: Khemakhem al. 2020
In all ICA problems we can only hope to identify the mixing up to certain ambiguities, e.g,, for linear mixings we cannot identify the scale and the order of the latent variable. We now define the maximal possible ambiguity set that still allows us to identify the factors of variation. For this we consider the set of 1-dimensional diffeomorphisms that reparametrize one coordinate
roparaa = {f R + Rf is is bijective and f' > 0}.
(6)
Then we define coordinate-wi wise reparamctrizationss by
Freparam- (f : R =hhi(s)).....
by E Fiddreparam)
(7)
Now we define the maximal ambiguity set of coordinate-wise reparametrizations and coordinate permutations
Snax = {f Rd R = o where A E (erm=_(), f Freparam).
Now that we defined the maximal possible ambiguity set that allows identification of the individual factors of variations (although relabelled and rescaled) we can define identifiability of ICA in some function class F. It convenient to use the pushforward of measures. If f f(s) and P then the distribution of x is given by the pushforward measure f.P.
2

---

Published in Transactions on Machine Learning Research (09/2023)
Definition 1. We call ICA identifiable in some function class F for some class of admissible base distributions P if the relation
J.P 9
for f.g F and P,Q P immlles that 9 h/a for some h E Smax where 52 Supp(P). In other words, two mechanisms generating the same observational distribution agree up to relabelling and reparametrizing the coordinates. This can be extended to the auxiliary variable case by requiring the same conclusion under the assumption that f.Pu = gu for all u where P. = p(-lu) E P and Qu = q(-lu) E P.
For our construction Section below we show that even a weaker form of identifiability does not hold. Namely, we assume that the distribution of the sources known.
Definition We call ICA in some function class F identifialle given the source distributions P. = p(-lu) if the relation
f.P,= g.lP
(10)
for all u implies that o hi/p for some Smax where S2 Uw, (upp(
We remark that equation 10 implics h.PP P. and thus restricts h substantially.
Now we introduce the function classes of interest. We consider function classes that are characterized by pointwise restriction of their gradient. For Z C Idd open and connected we define
Foet ( : Rd DS Df Diag(d)),
(11)
Fvol = [f S2 + Rd || Df = 1}.
(12)
Those are function classes considered in Zheng al. (002bb); Yang al. (2022); Buchholz al. (2022); Gresele et al. (2021) and we refer to those works for motivation. The function class Foet has the property that o h E Foct whenever h E Freparam and ʃ E Foct- Therefore, up to regularity issues the identifiability given the source distribution (Definition 2 is equivalent to identifiability in the unconditional
case (Definition] D.. This not true for the auxiliary variable case and other function classes.
3 Proof Techniques for Identifiability
The main goal of this section is to point out an error in the recent works Yang et al. (2022): Zheng et al. (2022a b) and we addition discuss more generally the limitations of their approach. We structure this section in two parts. First, we discuss a folklore technique to exploit the independence assumption. Then we discuss how the proof proceeds in the mentioned works, what is really shown, and finally, explain why this cannot be used to give full identifiability result without further ingredients. For readers interested in the details, we provide an extended discussion in Appendix A
3.1 A relation for Jacobian and Hessian
Let us fix function class F for which we want to show identifiability Suppose have P, Q and f.g F such that
2 g(s),, 8 8~ s' Q
(13)
and p and q have C densities. Then the standard approach to exploit the independence (manifest in the factorization of the densities) is to use that for i #
0,0, In(p(=)) 0,0, In(),(a.)) =0
(14)
The paper Zheng (022b) is extension of the carlier workshop paper Zheng al. (2022al We will only refer to the
work /Zheng al. (2122a our remarks apply equally Proposition in Zheng (2022)) We emphasize that this
only refers results Section of this paper and does not comment the results sparsity.
3

---

Published in Transactions on Machine Learning Research (09/2023)
We introduce the notation In(p) and q In(q) for the log-densitie.. We write h g and then the condition equation 13 can be expressed
s h()).
(15)
Then we can conclude that the following relation holds.
Lemma Assume that h is defined as above and volume preserving. Then the relation
Dh MDD
Σ DeD A DDag(d).
(16)
holds, where Diag((Ofg) h,.... (0 o h) and = Dia ..p) point and v= = [Vg) h.
8ip) are diagonal matrices at each
The proof follows by direct calculation and can be found in Appendix
Remark 1. Note that both works works|Yang et al. 2022); Zheng et al. [2022a] use the assumption that the mixing functions f. are volume preserving. Moreover, adding this assumption makes the considered function class FF smaller, so identifiability becomes easier. In particular, the argument extends to larger function classes without this restriction, e.g., showing that the argument does not allow to prove identifiability for FooTNFvol implics that the same is true for FOCT-
3.2 Proofs in Yang al. (2022); Zheng al. (2022a)
The main problem of the proofs in Yang al. 2022 Zheng et al. (202aa] is that they both rely Lemma in Yang et al. (2022) which is an erroneous version of Lemuna above. Essentially they claim, rephrased in our notation, that
(D(()))))(D)(>>)) E Diag(d),
(17)
i.e, when comparing with equation 16 we see that the e term is missing and is constant (because they assume Gaussian density for q). Indeed, the last relation is the matrix form of Equation (20) in Yang et al.) 2022 which states (slightly adopted to notation)
Thu
Oh; (8) 0.
(u)
(8) Os;
(18)
OSk
To bridge the different notations, we pinpoint the problem with their lemma in Appendix D Given that both proofs rely on the wrong relation equation [7] it clear that they are not valid as stated. Nevertheless, we want to clarify here in a bit more detail that it is highly likely that the proofs cannot be fixed by using similar arguments together with the (correct) relation equation 16 and more involved arguments are required.
Let us first explain how based on the relation equation 17 the proofs in the aforementioned works is finished. In both works can find matrices A2, 92, 92 such that
Dh Dh =A;
(19)
for i = and i = 2. Indood, in the auxiliary variable case in Yang et al. (022) they conclude that the relation equation 17 holds for the two pairs of densities (p"r , and (p"2, g0 ). In Zheng et al. (2022a) it is assumed that h E FOCT, i.c., Dh" Dh Diag(d). By defining f2 = Id and A2 Dh Dh (and 22 = 22 with Q, A from equation 16 we obtain that equation 19 also holds in this case.
In both papers, they assume in addition that the ratios (Ql2) sumptions entail that the diagonal entries of Ω are positive. Then the following lemma is used.
pairwise distinct, and their as-
Lemma 2 (Lemma 2 in Yang et al. (2022 ). Suppose that there are diagonal matrices Ω1, l2, A1, A2 with positive diagonal entries such that (St Jah)/// (L22is are pairwise different and
x
(20)
for 1,2 and some X. Then, X DP for diagonal matrix D and permodation matrie P.
4

---

Published in Transactions on Machine Learning Research (09/2023)
Thus we conclude that for all points such that equation 17 holds Dh(s) is scaled permutation. Let now investigate under which conditions the general relation equation T6 simplifies to equation 17 There are two cases of interest:
The function h is linear. We observe that if (+) = 0 for all and 8 then equation 16 simplifies to equation 17 The second derivative vanishes iff h is s linear. Thus we recover the well known identifiability result for linear ICA, but no more general result.
The vector 0 vanishes. When == which happens whenever Vq(h(s)) = 0 the relation equation also holds. In particular, when q is Gaussian density this is true exactly for the mean of the Gaussian. So
from the results in Yang et al. [2022): Zheng et al. (2022a) we can conclude that in both considered settings the gradient Dh(s) of the mixing function at the mean of the Gaussian prior is a scaled permutation which is a non-trivial observation. However, we cannot conclude anything from the true relation equation 16 soon as V = Vg(h(s)) + d when we do not assume additional restrictions on D2h.
as
We note that when not assuming that h is lincar or away from the points where Vq(h(s)) vanishes, we cannot infer any information on Dh(s) from equation 16 because we have not derived any restriction on the term involving the Hessian of h. Next, restriction on D²h. Indeed, in the settings considered in Yang et al. (2022); Zheng et al. (022a) Doh cannot be arbitrary since we assume h E F where F is characterized by Dh E M for some manifold M, e.g,, Dh satisfies DhT T Dh E Diag(d) in Zheng et al. (2022a). Then we can conclude that 0, Dh(s) E TDR(s) M, the tangent space of the manifold M at Dh(ss and this the tightest restriction we can obtain by looking just at s.
show that this remains true even when carefully using all available
In Appendix A we discuss that even when using the additional condition 0 D))s) (DMCs) M for all i it is not possible to extract any useful restriction from the relation equation 16 for points with * 0. This clarifies that any identifiability proof based on the relation equation 16 cannot just argue locally by considering a fixed point 8 but it has to exploit the global partial differential equation that the relation entails.
3.3 Takeaway
We conclude that for settings of interest it is not possible to show identifiability based on exploiting the relation in Lemma I in a single point. Instead, we need to exploit this relation globally, i.e., the equation must be viewed as a Partial Differential Equation (PDE) and identifiability amounts to showing certain properties of all solutions of this PDE. This approach can provide additional insights into the considered function class and in Appendix B we briefly review general results in this direction. Note that we do not rule out that Dh can be restricted by considering higher order derivatives of the log density, e.g., by considering 0,0,0,q(h(s)) In fact, when restricting to analytic functions it is clear that it is (in theory) sufficient to consider derivatives of all orders at single point. In addition, interesting relations for higher order derivatives were shown Yang 2022), c.g., for i + j and ==0
h 0.
(21)
However, we think it is unlikely that this is promising road, for the same reason as above, the highest order derivatives h have sufficiently many free parameters (O(d++1)) to satisfy the resulting (O(d*)) equations for every value of Dh.
4 Counterexamples for identifiability based on flows
In this section we show how flows can be used to construct spurious solutions in ICA settings and we use this to provide a counter-example to Theorem in Yang et al. (2022). This extends and elaborates on results in Buchholz al.] (2022) where it was shown that ICA with volume prescrving maps is not identifiable. The main idea is given a data generating mechanism x f(s) where P we try to construct a family
Note that considering constant. density 9 does not help. While this ensures v this also implies 0 as this collects the second derivative of if, so the relation equation T6 is trivially satisfied.
5

---

Published in Transactions on Machine Learning Research (09/2023)
Φ I2 - IR such that (0t).P fo th E F and o, is the flow of possibly timo-dependent vector field X, [0,T) Rd > I2d such that
(, = (s()
(22)
This procedure allows to construct spurious solutions certain cases. It blends particularly well with volume preserving transformations because the evolution of the density pt of (ft).. evolves according the equation
O.p. + Div(Xepe) 0.
(23)
In particular the flow of X, preserves the measure P if and only if Div(X,p) = To illustrate this construc- tion, we consider the case of general nonlincar ICA. Suppose P is the uniform mcasure on [0, 1] Then any divergence free vector field X with compact support in [0, 1] will construct a family of spurious solutions fo. Note that there are many such divergence free vector fields, one construction was given in Hyvärinen & Pajunen (1999) basod on radius dependent rotations. This construction can also be phrasod based on flows of suitable vector fields. A simple general construction of divergence free vector fields with compact support is to take any smooth function with compact support yp and then consider the vector field X such that X1 024, X2 -0pp and X, for i > 2. Then it is easy to see that X is divergence free. This also shows that it is possible to just mix 2 arbitrary factors of variation.
In Buchholz et al. (2022) it was also discussed how this type of construction leads to spurious solutions for the class of volume preserving maps. This requires us to construct a divergence free vector field (ensuring that f @ is volume preserving) orthogonal Vp ensuring that (f.).P because for divergence free vector fields
Div(Xp) Div(X) + XVp.
(24)
Then it easy to show (see Buchholz et al. (2022) that the vector field given by
X1 = -2p.. X2 = Sip, X, = 0, i i>
(25)
is divergence free and orthogonal Vp. We now show that this construction can be generalized the case of two auxiliary variables.
4.1 A counterexample to Theorem 1 Yang et al. (2022)
We now show that a similar construction can be extended the auxiliary variable setting. This gives a countercxample to the main result, Thcorem 1, in Yang et al. (2022). Consider the following two densities
Pi = p))|u()) ~ (0,
(26)
P2(s) = (() ~(()u())~((011)
(27)
where N' denotes normal variable and Σ Di oz)). We use P. and P2 to denote the measures oz² We assume that and thus b, are
with densities P1 and p2. We write 0, - b 02 pairwise different. It can be checked that all assumptions of Theorem 1 in Yang et al. (022) are satisfied. Thus the theorem claims that if f1, fa E Fvol and (fi).P, (fe).P, for j = 1,2 then f fu E Smax: i.e. is a concatenation of a permutation and a coordinate-wise transformation. We now construct. a flow $; such that (Pt).P, = P, for j = 1,2 and Q, is volume preserving and &, & Smax- This provides a generic countercxample the thcorem as f 4 generates the same observational distribution as f.
The map 4 will be the flow of the following vector field (denoting (81, (2, 82)8))
(c b)8283
X(s) = (a S1
(28)
(b 0.)8... 81
This means Φ: (0,)) x R + R3 is characterized by
(.4; = (+)
Lo(s) = 8.
(29)
An illustration of the resulting map i can be found in Figure I note that the standard unit vectors are fixed points of &, but otherwise the dynamics is highly mixing.

---

Published in Transactions on Machine Learning Research (09/2023)
(a)
(b)
(c)
(d)
Figure 1: Illustration of the flow 4, with a = b= 2, c = 3 by a frontal view on the sphere {s |s| - =
(a) color encoding by 33 coordinate for = 0, (b) color map of i1=1, (c) color map of P1=10: (d) color map of 1t=100-
Lemma 3. For the flow De defined above, the following results hold:
1. The flow erists for all times and all
2. The flow is analytic.
3. The map v, is volume preserving for any l.
4. The map o preserves p1 and p2 for all t, i.c., (Rt).P, P,
5. The map 4 mices the coordinates.
The proof of this Lemma can be found in Appendix C
4.2 A general construction
Finally, consider more general construction that allows to construct local deformations but requires some differential geometry. Let first give some inluition about the following result. We want to create a vector X field such that its flow 4, is volume preserving and satisfies (Lt).P, for a collection of measures P,, The equivalent conditions for X are Div(X) = 0 and XVP,. = 0. In other words the voctor field must be divergence free and must preserve the level sets of each Pa. As each level set a codimension 1 submanifold the intersection of i such level sets is a submanifold of dimension d (under some generecity condition). Tf k we can pick a divergence free vector field with compact support on those submanifolds consistently thus showing that identifiability fails. The following lemma makes those statements precise.
Lemma Consider i smooth densilies pı,..., Pk and S 2.2. Suppose 8n is a point such that (pc(so) are linearly independent. Then there is a non-vanishing vector field X with compact support such that
Div(X) Div(p, = 0.
(30)
The proof is based on standard techniques from differential geometry and can be found in Appendix This lemma has the following corollary.
Corollary 1. Consider ICA of volume preserving functions with k < d - 2 auriliary variables and densities Pi with < sisk k. Assume that there is point so such that Vp,(so) are independent vectors. Then ICA is not identifiable given the source distribution.
Proof. This a direct consoquence of the lemma above. Note that X cannot be aligned with one coordinate axis, i.c., of the form X e because is divergence free with compact support. Thus its flow mixes the coordinates, i.e. $ g Smax-

---

Published in Transactions on Machine Learning Research (09/2023)
4.3 Takeaway
Let us briefly put this result into context. using some heuristics. Assume we know the densities Pa. and the mixing is volume preserving. Parameter counting suggests that if there are d values of u we can infer from the vector (s)......, Pa(s)) Then the relation qu (a) = qu (f() = Pu implics that volume preserving ICA with d auxiliary variables should be identifialle given the source distribution. Similarly, if is not assumed to be volume preserving, we can use that In(q,, (x)) -n(( (x)) In(pn, (*)) (1))-In((pe) (n((p (s)) because the Jacobian determinant cancels. Thus ICA with d ++1 auxiliary variables should be identifiable for arbitrary nonlinear mixings. This suggests that the bound above is rather tight, and restricting mixings to be volume preserving does not. lead to substant ially stronger identifiability results than for general nonlinear functions. Since volume preserving maps are characterized by single pointwisc condition (dim(fA Det A| = d2 -11 this is not very surprising. Those heuristics can be compared to the rigorous results in Hyvärinen et al. (2019) where it was shown that under some -on-degenrracy nonlinear ICA is identifiable given 2d values of the auxiliary variable and Khemakhem al. (2020) a similar result with again O(d) scaling was shown for parametric family of source distributions. 1t currently an open question whether combining the auxiliary variable setting with stronger restrictions on the function class gives new identifiability results. In particular, results for Foct in the auxiliary variable setting of great interest.
5 Conclusion
The present work tried to clarify some aspects of identifiability proofs for ICA. The main messages are that using a pointwise lincar algebraic reasoning is not sufficient to show identifiability results for restricted function classes. Instead, it appears to be necessary to exploit the global uniqueness properties of certain differential equations associated to the problem. This contrasts with the identifiability results for the auxiliary variable setting, which allow for mostly algebraic arguments when the number of auxiliary variables is sufficiently large in comparison to the number of free parameters. In addition, we described new construction of suitable flows, which are powerful technique to show non-identifiability results.
Acknowledgements We thank Alexander Hägele for helpful comments and suggestions. This work supported by the Tübingen AT Center.
References
Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable nonlinear inde- pendent component analysis, 2022. URL https://arxiv..ot www.aawwww....raaaaaaoaae www./bs///////////. 222000066600 06406
Philippe G. Ciarlet. Mathematical Elasticity, Volume 1: Dhrre-Dimensionall Elasticity. Classics in Applied Mathematics Scries. Society for Industrial and Applied Mathematics, 2021. ISBN 9781611976779.
Pierre Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287 314, 1994. ISSN 0165-1684. doi: hutp:://dd..og//1..00//00.... httpa//ccccttccooco/ccccc/////ea
1684(94)90029-99 URL httpss:/www. Higher Order Statistics.
Harley Flanders. Liouville's theorem on conformal mapping. Journal of Mathematics and Mechanics, 15(1):
157-161, 1966. ISSN 00959057, 19435274. URL httt:://ww.jsto..rrgg//ee/e. www.oor.rrrsstaaeeee000..1
Luigi Gresele, Julius von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve, Independent mechanism analysis, a new concept? In Marc' Aurelio Ranzato, Alina Beygekzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-
14, 2021, virtual, pp. 28233 28248, 2021. URL http:p/p/ppooeet......oop.cccppp
Bac77119cc344bbb22lcccccc44433 http:p/peeeccceeeeeeeeeeeaaaaata Abstract. html
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, (2(3):429 439, 1999. doi:
URL https:::: //oii.
org/ (10p:/0000000/00022---0000088000
8

---

Published in Transactions on Machine Learning Research (09/2023)
Aapo Hyvärinen, Hiroaki Sasaki, and Richard E. Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), The 22nd In- ternational Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 859 559-868. PMLR, 2019. URL http://proceedingg mlr. press/
www.smllr....ereeeeeeeeeere www.:perss///ppeeeeee vttes//v9////y// w88/hy/arinenn9aa .html
Aapo Hyvärinen, Tyes Khemakhem, and Ricardo Monti. Identifability of latent-variable and structural- equation models: from linear to nonlinear, 2023.
Tyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In Silvia Chiappa and Roberto Calandra (eds.), The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Raly), volume 108 of Proceedings of Machine Learning Research, pp. 2207-2217. PMLR, 2020. URL http://proceedinngs mlr. wwwpppeeesss.eeep watpas//1100/hheeaaeeee/00000 wwwmakhen2aa..aaaa. htmml
Xiaojiang Yang. Personal communication. 2022.
Xiaojiang Yang, Yi Wang, Jiacheng Sun, Xing Zhang, Shifeng Zhang, Zhenguo Li, and Junchi Yan. Nonlinear ICA using volume-preserving transformations. In International Conference on Learning Representations, 2022. URL httpss:/ htt.:///cw....eeeoeoooooaaaaaaaa
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA with unconditional priors. In ICIR2022 Workshop on the Elements of Reasoning: Objecls, Struclure and Causalily, 2022a. URL
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advonces in Neural Informa- tion Processing Systems, 2022b. URL
A Obstacles to Pointwise Proofs of Identifiability
The purpose of this appendix is to show that the relation equation 16 point-wise cannot be sufficient to show identifiability even when we use all restriction available for D2h. While this section is slightly technical, we think that it is of interest to everyone who wants to gain a deeper understanding of the difficulties of identifiability proofs in restricted function classes. Here, we focus on the setting of identifiability in some restricted function class FF but similar reasoning can be applied to settings with few auxiliary variable values as in Yang et al.] (2022]. We assume that the function class F is characterized by the condition that Dh M pointwise for some (nonlinear) manifold M. In addition, we assume that the directional derivatives 0,DM(s) DD M are in the langent space of M which is implied by Dh(s) M for all 8. We now show that these assumptions combined with equation I6 are not sufficient to restrict Dh substantially for many typical manifolds M as soon as (s') is not the null-vector.
To make this precise, we assume that the reference distribution and the points 8, 8 = [((s) are fixed and try to investigate the implications the relation equation 16 for the Jacobian Dh(s). Above we discussed how vÿ(h(s))
implics that Dh(s) is a scaled permutation matrix. We now show that similarly strong conclusions are not possible in the general case. We introduce the notation 0,Dh(s) A E dddd for
i S d. Then equation 16 can be expressed as
(DhTDDD)s +
Σ An Asj- Aij-
(31)
We now assume that u of 0 is a fixed vector and show that under very mild conditions on Dh there are matrices A E TDaM such that the equation above holds. This implies that we cannot restrict Dh substantially based on equation [6 Thus we assume for now that Dh E M is a fixed matrix, and we derive under which conditions on Dh the system equation 31 has a solution for A For this it is convenient to rewrite the condition for the matrix A that follows from equation 31 We assume that is fixed. We consider the

---

Published in Transactions on Machine Learning Research (09/2023)
vector 3 with entries S3 (Dh (0hh) ij ie., 8 denote the columns of the matrix. Morcover, we write X Au I. Then the equation equation 31 simplifies to
v A = (8) +
(32)
To illustrate and clarify this approach, we first consider two concrete examples where the restriction on Dh can be characterized explicitly. Later we give more general argument based on parameter counting.
Example I: M = O(d) First, we assume that M (dd (below we will discuss that this is not particular interesting function class). Then
TDaM = {A gddd (Dh)* A+ A+ Dh D 0}, 00,
(33)
i.e., AD for tangent directions the relation (Dh+eA) (Dh+eA)= = =d++((e)) holds, which implies Dh T A A Dh 0. We can show the following lemma.
Dh is skew-symmetric. For readers not so familiar we note that this can be made clear by noting that
Lemma 5. Let M O(d)- Let E TRd and Dh E Rdddd such that all entries of (Dh)-Lv has all entries different from O. Then the system equation 32 has solution (+) for any 3
X²)
Remark 2. Note that 3 in equation 32 depends on Dh. However, we show that under the stated assumptions, there is solution for any vector B. We also remark that for v # and uniformly random Dh E SO(d) (Dh)-iv has all entries different from 0 with probability 1. This approach thus allows us to erclude only a subset of mcasure 0 of all possible values Dh(s). In contrast, for e = 0 it is possible to show that Dh(s) is a scaled permulation, so we constrain Dh(s) to a subset of measure 0 of all possible rotations.
The proof can be found in Appendix C
To summarize, we have established that for M = Odd) all we can infer from the relation equation 16 is that for a fixed v # 0 the only restriction on Dh is that (Dh)-1 - has no zero entry.
Example II: M = (A E Radxd A E Diag(d), (A) 1} Now we consider the case where M {A E Rdxdd A E Diag(d), (ct(A) = 1} which corresponds to Fo = Foct n Fvol- Let us add a remark regarding this choice.
We emphasize that E Fo is not the right condition we want to prove identifiability for this function class because FoOT is not a group, so we cannot conclude that h = gg f is in FoOT if g, f E FoeT and actually, we can only conclude that Dh in some larger space than FOOt (this is misconception in Zheng et al. ) based on too restrictive an estimation model).
Remark 3.
Let us nevertheless investigate the implications of h E F= FocT n Fvoll In this case we can even assume that the right-hand side of equation 16 is a fixed given matrix (i.c., we assume that we also know the ground truth source density p). Then we can absorb the now fixed vector Ne, 3 = 3 Xee, So the equation we now consider is
v =
(34)
We note that the langent space TDAM M is characterized by
ToaM {A Raxd A + (DA)T A Diag(d), Tr((Dh)-' AA 0}.
(35)
To clarify the second condition, we note that Det(Dh + eA) = Det(Dh) Det + (Dh) = 1 + For the next proof helpful to use that for any matrix B E -M there is a unique de- composition B OD where E Diag(d) with all diagonal entrics positive and O O(d). For the existence,
e ((((D))--A)+O0))
(O'D') (O'D") = D We just set D = / BT B and for uniqueness note that D = (OD)T(OD) (OD) = BT = (O'D')
can show the following result.
Lemma 6. Assume that v + 0. There a subset O c O(d) of measure zero (w.r.f. the Haar measure) such that for O O(d) with O & O and Dh OD for D E Diag(d) the equation equation 34 has a solution A M.
10

---

Published in Transactions on Machine Learning Research (09/2023)
Remark 4. The approach again only allows excluding at most set of measure zero of all possible values of Dh. So no useful restriction on Dh can be extracted based on this approach.
The proof of the Lemma can be found in Appendix C
Now analyze the equation equation 32 more generally based parameter counting. Note that it is typically under determined linear equation for A' and X Indeed, the condition A' E M enforces that A' is contained in a dim(M) dimensional subspace, i.e., A' has dim(M) free parameters. As soon as dim(M) 2 d-1 the equation equation 32 genericalll has a solution for a fixed v and any 3 because then there are d equations and at least d degrees of freedom (A' is also a variable). This suggests that this approach is not powerful enough to prove identifiabiltyy except for very low dimensional M. For example the orthogonal group considered above satisfies dim (SO(d) d(d- 1 and we already saw in Section that this function space is too small to be of interest.
Note that in the auxiliary variable setting we get U equations like equation 32 for each A, Then typically those U equation will have a solution if dim M) 2 U(d-1). For unrestricted function classes (i.e., dim (M) de) we then obtain the condition that > (d+1) to ensure identifiability. Note that Theorem 1 in Hyvärinen jtt al. (2019) proves identifiability for 2d+1 auxiliary variables under a non-degeneracy condition. This up to constant factor, the same scaling as parameter counting suggested. It is not clear whether their result is optimal, and it is also not clear that the strategy discussed here can be extended to an identifiability proof.
B Rigidity as an Alternative Path to Identifiability
We here briefly review results showing that certain function classes characterized by a restriction the Jacobian are substantially more restricted than directly apparent from the local condition. This observation was named rigidity and is crucial in the mathematical treatment of elasticity, where it is natural to consider pointwise restrictions on the gradient as this encodes the local deformability of materials (see, e.g., Ciarlet (2021) for an overview). Rigidity results provide additional information beyond the local gradient structure, e.g., they might provide additional restrictions on D2h beyond the trivial inclusion 0,Dh E TD and this information can be used to prove identifiability results.
To clarily this, we consider simple and well-known example that dates back at least to Liouville who showed more general result for conformal maps in 1850. Suppose that f: f - Rd is C2 function defined on an open connected set Ω C Rd such that Df E (O(d) for all s E Ω. Naively, this only implics that (s) E SO(d). So we can conclude that A, = 0,D satisfies the equation A, Df(s) + DJ(s) A, = O, e.g., if (s(s)) = Td then A, is skew-symmetric. However, Df is also the gradient of a function and this might restrict the values A, further than just being in the tangent space. In fact, in this example, it can be shown that f(s) = As + b for some A E SO(d). For clarity, we state this as theorem.
Theorem 1 (Liouville). Assume f : Ω Rd is C² function and Ω C Rd is open and connected. If Df(s) SO(d) for all s E Q then
((s) As
(36)
for some AE SOdd), bE ER.
So we conclude that f is necessarily affine, Df is constant, and A =.=0, O. In this case, the rigidity result implies a strong restriction on f (it vanishos). This additional restriction might be helpful when analyzing equation 16 and imply identifiability. Let us also emphasize that while the additional restrictions help to prove identifiability they also show that the the function class is not as expressive as naively expected.
A simple proof of this result, can be found in the recent review Hyvärinen el al. (0023 3
To the best of our knowledge, it is currently unknown whether there are rigidity results for larger function classes than conformal maps, in particular no results for the function class Foet seem to be known. Nev- ertheless, one possible road to identifiability is to show that for a certain function class FF characterized by
3 AAttualyy this proof shows that in this simple case is sufficient carcfully consider higher order derivatives but this does not extend more general function classes, e.g. conformal maps where (very simple) PDE has be integrated Flanders (9666).
11

---

Published in Transactions on Machine Learning Research (09/2023)
the pointwise condition (s) E M we can, in fact, conclude that 0,Df(s) U C TDS where U is small subset and then use this additional restriction in equation T6 Alternatively, one could try to directly consider the system of PDEs given by
DDD + Σ De A,
(37)
Dh E M.
(38)
This is essentially the road taken in Buchholz et al. (022 to show partial identifiability results in FoCT.
C Proofs
C.1 Proof for Section 3.1
Proof of Lemma
7 The standard transformation formula for densities reads
p(s) q(((s))) det Dh(s))-
(39)
We now apply equation
14 to this equation to conclude that (denoting q = In(g))
0 = 0,0, (Im(q(h(s)) + In det Dh(s)))
= 0: Σ (O.0) + 0,,, In det Dh(s)
(40)
Σ Σ (..0,)(((())) (s) a,1((s) + [(00))(((())))0)))6G))) + 0,0, In det Dh(s)).
Since is volume preserving det Dh(s) constant and the last term in the display above vanishes. Using that q(s') = Σ (s() for some functions the off diagonal terms of the first sum vanish. Thus, we end up with the condition.
0= 0-Σ - (o(T)((s)) asha(s) (she(s) + Σ (00000(()))000001(00)
(41)
We now fix point so and set sig h(s))- Then we write Diag( Of (8))))))))))))))))) with e Ōxg(s)). Then equation 41 can be written concisely (dropping arguments s)
and v e gd
DATADh + Σ De Diag(d).
(42)
The matrix A has diagonal entries A; 077())) This ends the proof. Note that the auxiliary variable case with a finite number of auxiliary variables um with 1 s m S U we get equations of this type that must be satisfied simultancously.
C.2 Proofs for Appendix
A
Proof of Lemma 5 Let O be an invertible matrix such that
O(Dh)-'v
(43)
(such O exists if v # 0). Now show that the first entry of O O-Ter Te, is non-zero, ie., eTo-Te Te, # 0. Note that
To- ---T=-ah -
(44)
by assumption. Thus can find X E such that
el (0-T3) -
(45)
12

---

Published in Transactions on Machine Learning Research (09/2023)
Now we set
(00--T0++0000 T T= T= + XO (0) - - -00-33+xX0- +x00-Tes).
((0-33'
(46)
Clearly T E Skew(d). Moreover,
0 -
(47)
by equation 45
Finally we set.
A' = OO
(48)
Then A' E M because
(Dh) A' + (A') - - -7
(49)
We calculate (using T T -T)
NO-Te, = 3 + Aeei.
(50)
This ends the proof.
Proof of Lemma 0 We first show the following fact. Suppose w E Rd is a vector such that not all its entries
have equal absolute value and E Rd is any vector, then there is T' Diag(d) + Skew(d) with Tr 0 such that
Tw b.
(51)
It is sufficient to show the result for d = as the general case follows by linearity and the d = 2 case applied to suitable subblocks of T. Note that the conditions on T imply that we can write it as
t t2 -t2
T =
E Diag(2) S2ew(2)
(52)
- t
and all such matrices satisfy all the requirements for T. With this we get
Tw t1
to
wi
W1
w2 w1
t1
(53)
t/
t
w22
12
If [wil + Wzl then the determinant of the matrix on the RHS is w3 w # 0. Thus we can find a solution (4,42)) of the linear equation Tw = b for any vector b.
We now define c Odd) to be the set of orthogonal matrices such that OT has all entries of equal absolute value. For + 0 this indoed a set of mcasure because by applying suitable rotation we can assume e₁ and then there are only 2d possible first columns of O namely d d-1/2(#11, #1,.,,, Suppose now that O e O(d) satisfies O O and Dh OD for some D E Diag(d). Let T Diag(d) Skew(d) with T = 0 be a solution of the equation
10
(54)
By assumption on O and o such a solution T exists. We now define
(A¹) DTOT
(55)
so that equation 34 holds. It remains to be shown that A E M. We find
(A') Dh (D6) A' DTO
proToD+L OD + DO or D D(T + +T ) E Diag(d).
(56)
Morcover, we have
Tr((D)) Tr(D-' (TDD) = Tr = 0.
(57)
We conclude that A E M. This ends the proof.
13

---

Published in Transactions on Machine Learning Research (09/2023)
C.3 Proofs for Section 4
Proof of Lemma
To show the first point, we note that the flow exists locally by standard ODE results.
For global existence, we note that
0 (0)
= (2()))12((2-)))))) ()0, + +,(222 (2(a c)),((s)) +c(s)s (2(b - ()) (58) =.
This implics that blow-up occurs. The flow is analytic because the vector field is analytic by the Kauchy-Kowalevsse theorem. To show the next statement, note that by equation 24 it is sufficient to show that Div(X) = Div(p X) = Div( = 0 (the first condition ensures that the Lebesgue measure is preserved, i.e. the flow is volume preserving). We calculate
(c b)5233
Div X Div
(a c)s183
=]c(cc - b)828s) -22((a + as((b a)s182) =0
(59)
[b a)8182
and similarly (using 0ipi- - 8121; O2p1 -82p1 0zpi -8311)
Div(Xpi) -(c b)818283 pi (s) (a c) Pi (8) -b- (b-a)s18283 ps())
(60)
and (using 01p2 - -181p2 0ap2 - bb, and 0ap2 = -cspp2)
Div(Xp2) (cc b) 8352as1 (8) - (a c)s381 bs2 p2(s) -(b (b0-ass18c33) p2(s)
(61)
-199233 (ab - ca - bc ab + ac - bc) = 0.
To show the last point note that the first order expansion for small gives
81 + E(c - b)8888
(s)
$2 -
(62)
83 e(b 182
which cannot be written as a coordinate-wise transformation as a, b, and a are pairwise different.
Proof of Lemma 4 It is convenient to use the language of differential geometry in the proof. We con- sider the manifold M Rd equipped with the standard volume form day A... dsg. The bulk of the proof consists of the construction of suitable new coordinates on M. Pick vectors U1,-..,Ud-1 such that together with Vpi(so) they form basis of Rd. Consider the functions y:(s) u.s. Then the map .k
(bs) (s), (s) (s),...... PM (s)) has an invertible Jacobian at 8o which we can assume to have
positive determinant by changing the sign of Un. Thus v defines a chart locally, i.e., in neighborhood U C M of sn- There a function h U IR with h > 0 such that
h A-dy A... dya- & dpi A dpp dsi A... dsa-
(63)
By shrinking U we can assume that the image vDU) is convex. We define new coordinate function & : (UU) + IR for (y1...., P E (U) by
1 in
z ( ... Va-k PI.... ..... pk)
1(1b- (t, .2......... ........ Na ...... Pk)) dt.
(64)
This implies in particular that
(x) h(2)-((2)
(65)
for I Rdd, We define 21 U + R by
= = -z((ss))
(66)
14

---

Published in Transactions on Machine Learning Research (09/2023)
Note that
k
021 dut
021 dp:. Op:
dæ1
(67)
Oy
We now calculate
0z1
dzp A dyz A... A dya- A dpi A... A dpu
dyI A... A dya- A dpi A... A dpk-
(68)
Note that all further terms involving dp; dy vanish by antisymmetry. We evaluate using the definition of directional derivatives with respect to tangent vector and equation 65
0zi
- (v())) = ( (0= 21 (w(+)) = (h o ((b(s)) = h(s).
(8)
(69)
dy1
0y
Byn
We therefore conclude
dæj A dyz A... A dya- A dpi A... A dpp = 0z1 dyı A... dya- A dpi A... A dpp
θyl
(70)
= h A... A dya-k A dp1 A... A dps
= =ds1 A... dsa-
Now the map : U + Rd given by (ps) = we define 21 as above and x; yi for > (it is easy to check from the definition of 2 that is injective but one could also shrink the domain for the following argument). Let U + R be given by the root of the determinant, of the matrix representation of the (standard) metric tensor for the coordinates sp. The relation
Ed (s)...., (s)) defines local chart where
gdz dyz A... A dya-k k A dp1 A... dpk dsi A... dsa
(71)
implies = 1. Let (UU) + Rd be a non-zero divergence free vector field with compact support such that X, = for i 2. Consider the vector field X M + TM = Rd defined by
X(s) = (p(s))
+ {2(((s))
(72)
2is
for U and extended by for s & U. Then we calculate using the coordinate formula for the divergence and using g
Div X= = Σ (x((xx))= = θ
X1 o
X = 0, + 02X₂ =Divga X 0.
(73)
dz
Ozg
Similarly we also get
DivM((pil Oz1 ((x, ((x10 p))))) + (8204)p)) = p, + p:02X, = p Divga x= 0.
(74)
02g
Here we used that
0z p; = 0
(75)
which follows from the relation Pt (x) Td-k+i(z) Zr where म, denotes the projection on the r-th coordinate and 0, dir-
15

---

Published in Transactions on Machine Learning Research (09/2023)
D Detailed comment on Lemma 1 in Yang et al. (2022)
Here we discuss in bit more detail the errors in Lemma 1 in Yang et al. (2022. We adopt their notation but simplify it slightly by removing u which is not necessary in the context of the lemma. We assume that f : Rd - Rd is volume preserving and consider
d p(s) = II (si)
(76)
i=1
(())
0,,1 9,.23/2
(77)
i=1
Note that
0.1 = -M/00, 0222 o;²
(78)
where Mr and of are the mean and variance of the Gaussian distribution. Lemma 1 in Yang et al. (002
claims that if q({(s)) p(s) then for + (see Eq. (15))
S (s) + 01, 0s,
0f: (s)
Of: Osk
E..
0.
(79)
0s,0sk
Moreover they claim in Equation (20) below the lemma that O21 0 be assumed giving the simpler relation
Of (s)
Of. Osk
Σ
01.2 Ds; 2s,
(8) = 0.
(80)
The proof strategy is to consider In(p(s)) = in(g(f(s)) and apply -0,0x to this relation, ie.. they consider
o,0 Σ (.) + 0c2s(s)²
(81)
The error is that below (17) they state that f(0) can assumed without loss of generality which is not true. Without this assumption, we obtain an additional term when both derivatives hit the same factor of [s)), i.e., we get
Sf: (s) 0,2 0s,
Of 0f: (s)
S (s)f((s) 0s
0 Σ 0rn Osk
Oss
(82)
S... (8)
of
0f
Σ (0.1 + (2))0
022 0s,
(s).
0s
Osk
Now we see that the claimed relation equation 80 only holds 0., + f. (s)8 which in light of equation 78
is true iff (x) = μs for all This finding agrees with the relation equation 16 because for Gaussian densitie ies
g the mean has maximal density, which implies that Vq(u) and thus e 0 in the notation of Section 3.1
16


Reviews:
[{"review_id": "J9sRr7izuS", "reviewer": "Reviewer_ejAJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Stengths\n\nThis paper provides a more thorough discussion on the identifiability issue of nonlinear ICA. \n\n### Weaknesses\n\nThis paper does not provide essentially new ideas but a more thorough discussion based on the existing framework. \n\nThe correction to Yang et al. (2022) does not require any novel ideas and is based on direct calculations, which makes the submitted paper appear weak from the beginning. It might be better to present the counterexamples before Section 3. Providing some numerical evidence would significantly improve the paper.\n\nAs a layperson in regard to the identifiability issue in nonlinear PCA, I found it difficult to understand the problem formulation and definitions. This submission became readable to me only after I skimmed over the work of Buchholz et al. I suggest the authors find some laypeople to read the submission and revise the paper according to their feedback.\n\nIsn't $\\Lambda$ in (24) simply zero? Is $s = s_0$ in the definition of $\\Lambda_{i i}$ after (24)? \n\nThere are some missing citations or proofs and a number of typos. See below. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper points out that the proof in Yang et al. (2022) used an assumption that does not hold in general ($f_i ( s_0 ) = 0$) and hence missed a term in their Lemma 1. Then, this paper points out that the result of Zheng et al. (2022b) is restrictive. Finally, the paper provides counterexamples to Yang et al. (2022). ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Missing citations or proofs: \n- A citation or proof is needed for Theorem 1.   \n- A citation or proof is needed for the claim that only liner Mobius transformations are volume preserving.\n- A citation or proof is needed for Footnote 3. \n\nMissing definitions: \n- The probability distribution $p$ is not defined after (1). \n- The definition of $v_k$ does not follow (20) but is hidden in the proof. \n- 2nd line, Section 3.2: Lemma 5 isn't presented \"above.\"\n- (38) contains a typo. \n\nTypos: \n- 3rd line, 1st paragraph: independent component analysis -> ICA\n- 1st line, 3rd paragraph: proof-strategy -> proof strategy\n- 2nd line, 1st paragraph, Sec. 2: independent component analysis -> ICA\n- p. 3: ... $f \\in \\mathcal{F}_{\\text{OCT}}$ *implies that* the relation...\n- p. 3: We denote *the* orthogonal and special group*s* as usual... \n- p. 3: We structure this *section* in two parts. \n- p. 6: ... *at* a single point...\n- p. 6: ... *necessarily* affine... \n- p. 7: ... rather small*.* *In* this case, it is just a subclass of *affine* functions. \n- p. 7: Louiville -> Liouville", "broader_impact_concerns": "N/A. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kyIe9rMm6u", "reviewer": "Reviewer_vvSK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n1. The manuscript is well-written, and the reference to relevant work is useful.\n\n2. The author provides a strategy for constructing distinct flows as a type of spurious solution for nonlinear ICA, assuming volume preservation.\n\nComments:\n\n1. The scope of this paper seems a little bit narrow. Its main focus is on the construction of a specific spurious solution for nonlinear ICA, a situation that seems to refute a theorem in Yang et al.'s previous work. It could be more beneficial if the paper offered more on potential alternative techniques or assumptions that could be instrumental in resolving the problem. At present, the author merely notes the potential necessity of considering global uniqueness properties, whose solutions could be intriguing.\n\n2. The author notes that the volume-preserving orthogonal coordinate transformation mentioned in Zheng et al. 2022a may require more complex techniques. To ensure clarity, I suggest the author refers to Zheng et al. 2022a consistently throughout the paper, instead of alternating with Zheng et al. 2022b. The main result of Zheng et al. 2022b, which focuses on proving identifiability through sparsity, is substantially different from that of Zheng et al. 2022a.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript examines a specific proof technique employed for the identifiability of nonlinear ICA, underscoring its limitations. In particular, it has been demonstrated that there may be a need to utilize the global uniqueness attributes of PDEs to address the problem.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the comments. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "i5FZBmeEJZ", "reviewer": "Reviewer_6ko9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Note: I edited my review after seeing the authors’ response and rechecking Yang et al.’s proofs.\n\nStrengths:\n\nThe paper meets the renewed interest in nonlinear ICA, which plays an increasingly important role in generative and latent variable models. The paper provides some useful insights into the problem and suggests the need to exploit global restrictions.\n\nAt first sight, eq. (20) in Lemma 1 of the current paper is quite similar to eq. (19) in Lemma 1 of Yang et al. (2022). The problem in the latter is that the coefficients should also depend on $s$ rather than only on $u$. The problem arises when collecting the cross terms on the right-hand side of (18): those in the remainder term of the Taylor expansion are missing. This error is indeed fatal and affects Lemma 2 and the subsequent proofs.\n\nWeaknesses:\n\nThe main purpose of this “short note” (not really so for a 16-page paper) is to point out an error in Yang et al. (2022). Besides that, the new results and contributions are relatively weak and do not seem to warrant a full-length paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper comments on some recent identifiability results for nonlinear independent component analysis (ICA). It is suggested that the identifiability proofs in Yang et al. (2022) and Zheng et al. (2022) rely on a critical but flawed lemma, and hence are problematic. The paper further discusses an alternative proof strategy and presents a counterexample to the identifiability result in Yang et al. (2022).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I see two ways of rewriting the paper:\n\n1) If the paper is intended as a “Comments” paper, it should address directly the issues of previous work and point out exactly where the error occurs. There is no need to introduce new notation or develop the theory from scratch.\n\n2) As a full-length paper, after briefly pointing out the error, it should make substantial new contributions and concentrate more on those new results.\n\nThe paper contains some typos that might affect understanding. For example, in the statement of Lemma 1, $v_k$ is not defined, and there are extra parentheses in the definitions of $\\Omega$ and $\\Lambda$.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8C5JBIJ1g6", "reviewer": "Reviewer_uWec", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths\n\n1/ The authors discuss the reported errors in Yang et al. [2022, Lemma ] both using their own notation, as well as the notation of Yang et al. [2022] (in Appendix C). This is much appreciated.\n\n2/ The fact that the authors also provide a (pessimistic) assessment of the proof strategy of Yang et al. [2022] is also very valuable for people working in this area.\n\n* Weaknesses\n\n3/ This reviewer is less convinced by the necessity of including Section 4. (See also 4/.)\n\n4/ At first glance, the submitted paper shares some similarities with the work of Buchholz et al. [2022]. \n    Looking closer, the two papers appear to be mostly complementary, \n    but it would be better to add a clear reference for Theorem 1 and a comparison with existing work for Theorem 2 of Section 4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The topic of this paper is Independent Component Analysis (ICA). An invertible mixing function f is applied to sources s with independent components s1, . . . , sd. The problem consists in recovering the sources from observing f (s). For a linear mixing function the problem is solved, while this task is known to be infeasible for general f . The present work is part of a growing literature attempting to analyze properties that the class of functions should possess for the problem to be solvable.\n\n\n1/ As their main contribution, the authors pinpoint a technical error in some previous work in the literature Yang et al. [2022, Lemma 1]. The claim is that both of the works Yang et al. [2022], Zheng et al. [2022] suffer from this inaccuracy, thus the results therein are (at least partially) invalidated. The authors propose a corrected\nversion of the erroneous lemma (Lemma 1 in the text), and highlight some limitations of the approach proposed by Yang et al. [2022].\n\n2/ The authors then build upon tools from rigidity theory developed by Buchholz et al. [2022] to discuss that in some cases, local conditions on the Jacobian can lead to very strong global restrictions on functions, helping to prove identifiability.\n\n3/ Finally, the authors further strenghten their argument by constructing a counter-example to Yang et al. [2022, Theorem 1] using flows.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-- Clarify the distinction between the results presented in Section 4 and that of Buchholz et al. [2022].\n\n-- Below is a list of minor issues.\n\n* After equation (20), the expression of Ω and Λ should be revised. Some parentheses are missing. The same issue arises before (24).\n\n* On p.5, “which is an erroneous version of Lemma 5 above” should read “Lemma 1” instead.\n\n* On p.6, a “well-known example”. Please provide a reference.\n\n* On p.8, check equation (38).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All 4 reviewers agreed that the claim of the paper on the mistake in (Yang et al., 2022) is correct. This, together with the notation clarification and the counterexamples in Section 5, is enough to justify publication in TMLR in my opinion. This discussion is clear and easy to connect to the previous works [Reviewer uWec: "The authors discuss the reported errors in Yang et al. [2022, Lemma ] both using their own notation, as well as the notation of Yang et al. [2022] (in Appendix C). This is much appreciated."] [vvSK: "The manuscript is well-written, and the reference to relevant work is useful."].

A recurring criticism is that in Section 4, the author defends the ideas he introduced in a previous paper (Buchholz et al., 2022), but does not provide any new idea: [uWec: "This reviewer is less convinced by the necessity of including Section 4 (...) At first glance, the submitted paper shares some similarities with the work of Buchholz et al. [2022]. Looking closer, the two papers appear to be mostly complementary, but it would be better to add a clear reference for Theorem 1 and a comparison with existing work for Theorem 2 of Section 4.] [ejAJ: "This paper does not provide essentially new ideas but a more thorough discussion based on the existing framework."] 

In response to this, the author provided adequate citations in Section 4, and shortened it.

All reviewers required some minor clarifications (see the list of minor comments / typos [vvSK, ejAJ]). This was addressed by the author.

However, there remains a strong disagreement among the reviewers on the final decision. [uWec, ejAJ] supported the publication of the paper while [6ko9, vvSK] recommended to reject it, based on the lack of novelty: [6ko9: "The main purpose of this “short note” (not really so for a 16-page paper) is to point out an error in Yang et al. (2022). Besides that, the new results and contributions are relatively weak and do not seem to warrant a full-length paper."] [vvSK: "The scope of this paper seems a little bit narrow."].

From the abstract it is clear that this paper is not meant to introduce any new technique. Rather, it is meant to point out a (propagating) mistake in previous works, and to bring some clarification on the problems that led to this mistake. As such, the paper is kept short (8.5 pages including references, with more technical details in the appendices). The comments from all reviewers clearly show that the paper is successfull in its clarification task. I agree with them. The only debatable point was to defend in Section 4 as an alternative to (Yang et al., 2022) ideas that were already introduced in a previous paper. In the revised version, the author shortened this section and clarified the link to (Buchholz et al., 2022). The recommendation based on the lack of novelty would be appropriate for a top-tier conference. In its current state, the paper brings useful clarifications and new challenges to the ICA community, and as it falls perfectly within the scope of TMLR. I will thus support its publication.
Decision: Accept (poster)