Independent mechanism analysis, a new concept?
Authors: Luigi Gresele
Julius Von Kügelgen
Vincent Stimper
Bernhard Schölkopf
Michel Besserve
Venue: NeurIPS 2021 Poster (2021)
Keywords: causal inference
independent causal mechanisms
nonlinear ICA
independent component analysis
identifiability
blind source separation
unsupervised representation learning

Abstract: Independent component analysis provides a principled framework for unsupervised representation learning, with solid theory on the identifiability of the latent code that generated the data, given only observations of mixtures thereof. Unfortunately, when the mixing is nonlinear, the model is provably nonidentifiable, since statistical independence alone does not sufficiently constrain the problem. Identifiability can be recovered in settings where additional, typically observed variables are included in the generative process. We investigate an alternative path and consider instead including assumptions reflecting the principle of independent causal mechanisms exploited in the field of causality. Specifically, our approach is motivated by thinking of each source as independently influencing the mixing process. This gives rise to a framework which we term independent mechanism analysis. We provide theoretical and empirical evidence that our approach circumvents a number of nonidentifiability issues arising in nonlinear blind source separation.

Independent mechanism analysis, a new concept?
Luigi Gresele"
Julius von Kügelgen" 1,2
Vincent Stimper 1,2
Bernhard Schölkopf
Michel Besserve
Max Planck Institute for Intelligent Systems, Tübingen, Germany University of Cambridge jvk, vincent. stinnterttmpprr bs, besserve) www.etteeeee.eeeeeeeeeeeee .dg.de
Abstract
Independent component analysis provides principled framework for unsupervised representation learning, with solid theory on the identifiability of the latent code that generated the data, given only observations of mixtures thereof. Unfortunately, when the mixing nonlinear, the model is provably nonidentifiable, since statistical independence alone does not sufficiently constrain the problem. Identifiability can be recovered in settings where additional, typically observed variables are included in the generative process. We investigate an alternative path and consider instead including assumptions reflecting the principle of independent causal mechanisms exploited in the field of causality. Specifically, our approach is motivated by thinking of each source independently influencing the mixing process. This gives rise to a framework which we term independent mechanism analysis. We provide theoretical and empirical evidence that our approach circumvents number of nonidentifiability issues arising in nonlinear blind source separation.
1
Introduction
One of the goals of unsupervised learning is to uncover properties of the data generating process, such latent structures giving rise to the observed data. Identifiability [55] formalises this desideratum: under suitable assumptions, a model learnt from observations should match the ground truth, to well-defined ambiguities. Within representation learning, identifiability has been studied mostly in the context of independent component analysis (ICA) [17, 40J, which assumes that the observed data x results from mixing unobserved independent random variables s; referred recover the sources based on the observed mixtures alone, also termed blind source separation (BSS). major obstacle BSS is that, in the nonlinear case, independent component estimation does necessarily correspond to recovering the true sources: it is possible to give counterexamples where the observations transformed into components yı which are independent, yet still mixed with respect to the true sources 8; [20, 39, 98). In other words, nonlinear ICA is not identifiable.
sources. The aim is
In order to achieve identifiability, growing body of research postulates additional supervision structure in the data generating process, often in the form of auxiliary variables [28, 30, 37, 38, 41]. In the present work, we investigate different route to identifiability by drawing inspiration from the field of causal inference [71, 78] which has provided useful insights for number of machine learning tasks, including semi-supervised [87, 103], transfer [6, 23, 27, 31, 61, 72, 84, 85, 97, 102, 107], reinforcement [7, 14, 22, 26, 53, 59, 60, 106], and unsupervised [9, 10, 54, 70, 88, 91, 104, 105] causal process and apply the principle of independent causal mechanisms (ICM) which postulates that the generative process consists of independent modules which do not share information [43, 78, 87]. In this context, "independent" does not refer to statistical independence of random variables, but rather to the notion that the distributions and functions composing the generative process are chosen independently by Nature [43, 48). While a formalisation of ICM [43, 57] in terms of algorithmic (Kolmogorov) complexity [51] exists, it not computable, and hence applying ICM in practice requires assessing such non-statistical independence
learning. To this end, we interprel the ICA mixing
Equal contribution. Code available at:
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

os
0f Za2
1
D8g
8F
[Jrl
of
DE
DE
ds2
De1
Figure 1: (Left) For the cocktail party problem, the ICM principle as traditionally understood would say that the content of speech Ps is independent of the mixing recording process f (microphone placement, room acousties). IMA refines, or extends, this idea at the level of the mixing function by postulating that the contributions af/a., of each source f, as captured by the speakers' positions relative to the recording process, should not be fine-tuned to each other. (Right) We formalise this independence between the of/a.., which are the columns of the Jacobian Jr. as an orthogonality condition: the absolute value of the determinant [Jrl. i.e. the volume of the parallelepiped spanned by 0t ae,,, should decompose as the product of the norms of the 8f/az,.
with suitable domain specific criteria [96]. The goal of our work thus constrain the nonlinear ICA problem, in particular the mixing function, via suitable ICM measures, thereby ruling out common counterexamples to identifiability which intuitively violate the ICM principle.
Traditionally, ICM criteria have been developed for causal discovery, where both cause and effect are obverved [18, 45, 46, I 110]. They enforce an independence between (i) the cause (source) distribution and (ii) the conditional or mechanism (mixing function) generating the effect (observations), and thus rely on the fact that the observed cause distribution is informative. As we will show, this renders them insufficient for nonlinear ICA, since the constraints they impose are satisfied by common counterexamples to identifiability. With this in mind, we introduce new way to characterise or refine the ICM principle for unsupervised representation learning tasks such as nonlinear ICA.
Motivating example. To build intuition, we turn to a famous example of ICA and BSS: the cocktail party problem, illustrated in Fig. (Left). Here, number of conversations are happening in parallel, and the task is to recover the individual voices 8; from the recorded mixtures r;- The mixing recording process f is primarily determined by the room acoustics and the locations at which microphones are placed. Moreover, each speaker influences the recording through their positioning in the room, and may think of this influence as af/as,. Our independence postulate then amounts to stating that the speakers' positions are not fine-tuned to the room acoustics and microphone placement, or to each other, i.e., the contributions 0f/as, should be independent (in non- non-statistical sense)."
Our approach. We formalise this notion of independence between the contributions at/os. of each source to the mixing process (i.c., the columns of the Jacobian matrix Jr of partial derivatives) as an orthogonality condition, see Fig. (Right). Specifically, the absolute value of the determinant Jrl. which describes the local change in infinitesimal volume induced by mixing the sources, should factorise or decompose the product of the norms of its columns. This can be seen as a decoupling of the local influence of each partial derivative in the pushforward operation (mixing function) mapping the source distribution independent mechanism analysis (IMA). IMA can be understood as a refinement of the ICM principle that applies the idea of independence of mechanisms the level of the mixing function.
the observed one, and gives rise to a novel framework which term
Contributions. The structure and contributions of this paper can be summarised as follows:
we review well-known obstacles to identifiability of nonlincar ICA ($ 2)1), as well as existing ICM criteria ($ 2.2), and show that the latter do not sufficiently constrain nonlinear ICA 3);
we propose more suitable ICM criterion for unsupervised representation learning which gives rise to a new framework that we term independent mechanism analysis (IMA) ($ 4); we provide geomet- ric and commaaioon-thooeeti interpretations of IMA (S 4.1), introduce an IMA contrast function which is invariant to the inherent ambiguities of nonlinear ICA (S 4.2), and show that rules out a large class of counterexamples and is consistent with existing identifiability results ($ 4.3);
we experimentally validate our theoretical claims and propose regularised maximum maximum-likelihood learning approach based on the IMA constrast which outperforms the unregularised baseline ($ 5); additionally, we introduce a method to learn nonlinear ICA solutions with triangular Jacobian and a metric to assess BSS which can be of independent interest for the nonlinear ICA community.
For additional intuition and possible violations in the context of the cocktail party problem, Appendix B.4.

---

2 Background and preliminaries
Our work builds on and connects related literature from the fields of independent component analy- sis (8 2. and causal inference ($ 2.2). We review the most important concepts below.
2.1
Independent component analysis (ICA)
Assume the following metaageneraiing process for independent component analysis (ICA)
f(8):
(s()) = II= Ps, (8))
(1)
where the observed mixtures Rn result from applying smooth and invertible mixing function f: IR" - Rn to a set of unobserved, independent signals or sources s E RR" with smooth, factorised density Ps with connected support (see illustration Fig. 2b). The goal of ICA is learn unmixing function g IR" -> RR" such that y g(x) has independent components. Blind source separation (BSS), the other hand, aims to recover the true unmixing f-1 and thus the true sources (up to tolerable ambiguities, see below). Whether performing ICA corresponds to solving BSS related to the concept of identifiability of the model class. Intuitively, identifiability is the desirable property that all models which give rise to the same mixture distribution should be "equivalent" up to certain ambiguities, formally defined follows.
Definition 2.1 (>-iidnntiiabiility) Let F be the set of all smooth, invertible functions f R" + R" and P be the set of all smooth, factorised densities ps with connected support on R" Let M S FxP be a subspace of models and let ~ be an equivalence relation on M. Denote by f.ps the push-forward density of pa via Then the generative process (1) is said to be ~-iieetifiable on M if
v(f, (f, ps) E M:
f.px f.ps
(f, (f, pz).
(2)
If the true model belongs to the model class M, then ~-identiiiaabiitty ensures that any model M learnt from (infinite amounts of) data will be ~-equivalent to the true one. An example is linear ICA which is identifiable up to permutation and rescaling of the sources on the subspace MuN of pairs of (i) invertible matrices (constraint on F) and (ii) factorizing densities for which at most one 8; is Gaussian (constraint on P)) [17, 21, 93], see Appendix A for more detailed account.
In the nonlinear case (i.e., without constraints on F), identifiability is much more challenging. If 8; and 8j are independent, then are (s:) and hy(8;) for any functions h; and hy- In addition to permutation-ambiguig such element-wise h(s) = (hi (s1)...... hip (sn)) can therefore not be resolved either. We thus define the desired form of identifiability for nonlinear BSS as follows.
Definition 2.2 (~uss). The equivalence relation ~Bss F P defined as in Defn. 2.1 is given by
(f, p&) ~ass p&) > P. h s.t. (f, = (f o h o p-1, h).ps)
(3)
where P is a permutation and h(s) (st). (.. is an invertible, element-wise function.
fundamental obstacle-and crucial difference the linear problem- -s that in the nonlinear case, different mixtures of 8; and 8j can be independent, i.e., solving ICA is nol equivalent to solving BSS. A prominent example of this is given by the Darmois construction [20, 39].
Definition 2.3 (Darmois construction). The Darmois construction gD R > (0, 1)" is obtained by recursively applying the conditional cumulative distribution function (CDF) transform:
(P(x..) == P(X, S x. x1(x = f p(z) x14=1)da
(i= I,....n).
(4)
The resulting estimated sources yD = gx)) are mutually- inndpenll- uniform r.v.s by construction, see Fig. 2a for an illustration. However, they need not be meaningfully related to the true sources 8, and will, in general, still be a nonlinear mixing thereof [39]." Denoting the mixing function corresponding to (4) by pD = (gD)- and the uniform density on (0. 1)" by Pu, the Darmois solution (FD, .Pu) thus allows construction of counterexamples to res "uss-identifiabiiity on F x P.
Remark 2.4. gD has lower-triangular Jacobian, i.e., 8g. 8/P/ea, = 0 for i < j. Since the order of the J.; is arbitrary, applying gD after a permutation yields a different Darmois solution. Moreover, (4) yields independent components DD even if the sources 8; were not independent to begin with.*
*Consider, e.g., a mixing f with full Jacobian which yields a contradiction to Defn. 2.2, due to Remark 2.4. 'By applying a change of variables, can see that the transformed variables in (4) are uniformly distributed the open unit cube, thereby corresponding to independent components [69, $ 2.22]. "This has broad implications for unsupervised learning, as it shows that, for i.i.d. observations. not only factorised priors, but any unconditional prior is insufficient for identifiability (see. e.g.. [49]. Appendix D.2).
3

---

(-g-tty)
F1
T2
X
(a) CDF transform
(b) ICA
(c) ICM
(d) IMA
Figure (a) Any observed density pa can be mapped to uniform Ps via the CDF transform g(x) P(X S #): Darmois solutions (f", pp)) constructed from (4) therefore automatically satisfy the independence postulated by IGCI (6). (b) ICA setting with n 2 sources (shaded nodes are observed, white ones unobserved). (c) Existing ICM criteria typically enforce independence between observed input cause distribution Pe and mechanism Pe (independent objects are highlighted in blue and red). (d) IMA enforces independence between the contributions of different sources s, to the mixing function as captured by 6t/ōss.
Another well-known obstacle to identifiability are measure- meeasure-preserving antomorphisms (MPAs) of the source distribution Ps: these are functions a which map the source space to itself without affecting its distribution, i.e., a.P² Ps [39]. A particularly instructive class of MPAs is the following [49, 58).
Definition 2.5 ("Rotated-Gaausian" MPA). Let R E (n) be an orthogonal matrix, and denote by (s (Fs, (81),.- ..., Fs,, (8x)) and (z) (#) ",,,,, ... ((zz)) the element-wise CDFs of smooth, factorised density Ps and of Gaussian, respectively. Then the "aatted-auussian" MPA a (ps) is
a ap()) = F.' + R + 1...d-FFF. F...
(pa)
(5)
a (p=) first maps to the (rotationally invariant) standard isotropic Gaussian (via T- F.). then applies a rotation, and finally maps back, without affecting the distribution of the estimated sources. Hence, if (f. ps) is valid solution, then so (f a (ps), Pb) for any E (n)). Unless is permutation, this constitutes another common counterexample to "uss-identifiability on F x P.
Identifiability results for nonlinear ICA have recently been established for settings where auxiliary variable u (c.g., environment index, time stamp, class label) renders the sources conditionally inde- pendent [37, 38, 41, 49]. The assumption on Ps in (1) is replaced with Ps is (s/uu) II Ps. (3; u), thus restricting P in Defn. 1. In most cases, u is assumed to be observed, though [30] is notable exception. Similar results exist given access to a second noisy view X [28].
2.2 Causal inference and the principle of independent causal mechanisms (ICM)
Rather than relying only on additional assumptions on P (e.g., via auxiliary variables), we seek to further constrain (1) by also placing assumptions on the set F of mixing functions f. To this end, we draw inspiration from the field of causal inference [71, 78]. Of central importance to our approach is the Principle of Independent Causal Mechanisms (ICM) [43, 56, 87].
Principle 2.6 (ICM principle [78]). The causal generative process of a system's 's variables is composed of autonoomuus modules that do not inform or influence each other.
These "modules" are typically thought of as the conditional distributions of each variable given its direct causes. Intuitively, the principle then states that these causal conditionals correspond to independent mechanisms of nature which do not share information. Crucially, here "independent" does not refer to statistical independence underlying distributions algorithmic objects. For bivariate system comprising a cause and an effect e, this idea reduces to an independence of cause and mechanism, see Fig. 2c. One way to formalise ICM uses Kolmogorov complexity K(-) [51] as measure of algorithmic information [43].
random variables, but rather to independence of the
However, since Kolmogorov complexity is is not computable, using ICM in practice requires as- sessing Principle 2.6 with other suitable proxy criteria [9, 11, 34, 42, 45, 65, 75-78, 90, 110]. Allowing for deterministic relations between cause (sources) and effect (observations), the criterion which is most closely related to the ICA setting in (1) is information- -eeomtiinggemotriicc causal inference (IGCI) [18, 46). IGCI assumes nonlinear relation
f(c) and formulates notion of indepen-
SuThis can seen as an algorithmic analog of replacing the empirically undecidable question statistical in- dependence with practical independence tests that are based on assumptions on underlying distribution" [43]. Forr a similar criterion which assumes linearity [45, 110] and its relation to linear ICA, see Appendix B.I.

---

dence between the cause distribution Pe and the deterministic mechanism f (which we think of as degenerate conditional Pete) via the following condition (in practice, assumed hold approximately).
Ciaci (f,, Pe) := f log Jy(c) Pe(c)dc - f log [J((c)] dc = O,
(6)
where (Jr(c)),, = 0 (c) the Jacobian matrix and | the absolute value of the determinant.
Ciaci can be understood as the covariance between Pe and log JJpl (viewed as r.v.s on the unit cube w.r.t. the Lebesgue measure), so that Cieei rules out a form of fine-tuning between Pe and [Jr]- As its name suggests, IGCI can, from an information-geomee geometric perspective, also be seen as an orthogonality condition between cause and mechanism in the space of probability distributions [46], see Appendix B.2, particularly eq. (19) for further details.
3 Existing ICM measures are insufficient for nonlinear ICA
Our aim is to use the ICM Principle 2.6 to further constrain the space of models MCF and rule out common counterexamples identifiability such as those presented in $ 2.1. Intuitively, both the Darmois construction (4) and the rotated Gaussian MPA (5) give rise to "non- monngeneric" solutions which should violate ICM: the former, (fD, Pu). due the triangular Jacobian of pD (see Remark 2.4), meaning that each observation I; = f (y (yis) only depends on a subset of the inferred independent components Yi:s, and the latter, (f o a Rt (pa), Ps). , due to the dependence of f a (ps) on ps (5).
However, the ICM criteria described in $ 2.2 were developed for the task of cause-effect inference where both variables are observed. In contrast, in this work, we consider an unsupervised representa- tion learning task where only the effects (mixtures x) are observed, but the causes (sources s) are not, turns out that this renders existing ICM criteria insufficient for BSS: they can easily be satisfied by spurious solutions which are not equivalent to the true one. We can show this for IGCL. Denote by Miscı ps) E F x P : Ciaci px) 0} J P the class of nonlinear ICA models satisfying IGCI (6). Then the following negative result holds.
Proposition 3.1 (IGCI is insufficient for Mess-ideniifabiiity). (1) is not "sss-identiffable on MiacI
Proof. IGCI (6) is satisfied when Ps is uniform. However, the Darmois construction (4) yields uniform sources, see Fig. 2a. This means that (fD o a a (Pu),PP Pu) E MMIsCI. so IGCI can be satisfied by solutions which do not separate the sources in the sense of Defn. 2.2, see footnote 2 and [39]. D
As illustrated in Fig. 2c, condition (6) and other similar criteria enforce notion of "genericity" or "decoupling" of the mechanism w.r.t. the observed input distribution." They thus rely on the fact that the cause (source) distribution is informative, and are generally not invariant to reparametrisation of the cause variables. In the (nonlinear) ICA setting, on the other hand, the learns source distribution may be fairly uninformative. This poses challenge for existing ICM criteria since any mechanism is generic w.r.t. an uninformative (uniform) input distribution.
4 Independent mechanism analysis (IMA)
As argued in $ 3, enforcing independence between the input distribution and the mechanism (Fig. 2c), as existing ICM criteria do, is insufficient for ruling out spurious solutions to nonlinear ICA. We therefore propose a new ICM-inspired framework which is more suitable for BSS and which we term independent mechanism analysis (IMA). All proofs are provided in Appendix C.
4.1 Intuition behind IMA
As motivated using the cocktail party example in $ 1 and Fig. 1 (Left), our main idea is to enforce notion of independence between the contributions or influences of the different sources s; on the observations x = f(s) as illustrated in Fig. 2d-as as opposed to between the source distribution and mixing function, cf. Fig. 2c. These contributions or influences are captured by the vectors of partial derivatives öf 0f/a... IMA can thus be understood as a refinement of ICM at the level of the mixing f: in addition to statistically independent components s;, we look for mixing with contributions ar/as, which are independent, in a non-statistical sense which we formalise as follows.
Principle 4.1 (IMA). The mechanisms by which each source influences the observed distribution, as captured by the partial derivatives öt /0,e,, are independent of each other in the sense that for all
Of
log Jr Jr(()) = Σ log
SS)
a8:
II fact, many ICM criteria can he phrased as special cases of unifying group-invariance framework [9]. 'The title of the present work is thus reverence Pierre Comon's seminal 1994 paper [17].
5

---

Geometric interpretation. Geometrically, the IMA principle can be understood as an orthogonality condition, as illustrated for n in Fig. (Right). First, the vectors of partial derivatives of of/as.. for which the IMA principle postulates independence, are the columns of Jr. [Jrl thus measures the volume of the n didmennsional parallclepiped spanned by these columns, as shown on the right. The product of their norms, on the other hand, corresponds to the volume of an n n-dimensional box, or rectangular parallelepiped with side lengths || as shown on the left. The two volumes are equal if and only if' all columns of/De, of Jr are orthogonal. Note that (7) is trivially satisfied for n i.e., if there no mixing, further highlighting its difference from ICM for causal discovery.
Independent influences and orthogonality. In a high dimensional setting (large n), this orthogonality can be intuitively interpreted from the ICM perspective as Nature choosing the direction of the influence of each source component in the observation space independently and from an isotropic prior. Indeed, it can be shown that the scalar product of two independent isotropic random vectors in Ra vanishes as the dimensionality n increases (equivalently: two high-dimensional isotropic vectors are typically orthogonal). This property was previously exploited in other linear ICM-based criteria (see [44, Lemma 5] and [45, Lemma 1 & Thm. 1))." The principle in (7) can be seen as a constraint on the function space, enforcing such orthogonality between the columns of the Jacobian of f at all points in the source domain, thus approximating the high-dimensional behavior described above. 10
cnermatiio-geomeeri interpretation and comparison to IGCL. The additive contribution of the sources' influences 0f/ in (7) suggests their local decoupling at the level of the mechanism f. Note that IGCI (6), on the other hand, postulates a different type of decoupling: one between log [Jrl and Ps. There, dependence between cause and mechanism can be conceived as a fine tuning between the derivative of the mechanism and the input density. The IMA principle leads to a complementary, non-statistical measure of independence between the influences 8f/Be, of the individual sources on the vector of observations. Both the IGCI and IMA postulates have an information information-geomerric -enomerric interpre- tation related to the influence of ("non-statistically") independent modules the observations: both lead to an additive decomposition of a KL-divergence between the effect distribution and a reference distribution. For IGCI, independent modules correspond to the cause distribution and the mechanism mapping the cause to the effect (see (19) in Appendix B.2). For IMA, on the other hand, these are the influences of each source component on the observations in an interventional setting (under soft interventions on individual sources), as measured by the KL-ddieereeness between the original and intervened distributions. See Appendix B.3, and especially (22), for a more detailed account.
We finally remark that while recent work based on the ICM principle has mostly used the term "mechanism" to refer to causal Markov kernels p(X, P.A,) or structural equations [78], we employ it in line with the broader use of this concept in the philosophical literature." !I To highlight just two examples, [86] states that "Causal processes, causal interactions, and causal laws provide the mechanisms by which the world works; to understand why certain things happen, we need to see how they are produced by these mechanisms"" ; and [99] states that " "Mechanisms are events that alter relations among some specified set of elements". ". Following this perspective, we argue that a causal mechanism can more generally denote any process that describes the way in which causes influence their effects: the partial derivative of/as. thus reflects a causal mechanism in the sense that it describes the infinitesimal changes in the observations x, when an infinitesimal perturbation is applied to s;.
4.2 Definition and useful properties of the IMA contrast
We now introduce contrast function based on the IMA principle (7) and show that it possesses several desirable properties in the context of nonlinear ICA. First, we define a local contrast as the difference between the two integrands of (7) for particular value of the sources s.
Definition 4.2 (Local IMA contrast). The local IMA contrast CIMA (f, s) of f at point s is given by
12
Of as; (s)
CaMA (f,
log Σ
log [r((s)
(8)
Remark 4.3. This corresponds to the left KL measure of diagonality [2] for J Jr(s).
This has also been used as "leading intuition" [sic] to interpret IGCI in [46].
1o provide additional intuition on how IMA differs from existing principles of independence of cause and mechanism, we give examples, both technical and pictorial, of violations of both in Appendix B.4. See Table in [62] for a long list of definitions from the literature.

---

The local IMA contrast CIca (f. s) quantifies the extent to which the IMA principle is violated at given point s. We summarise some of its properties the following proposition.
Proposition 4.4 (Properties of CiaA s)). The local IMA contrast CIMA (f, s) defined in (8) satisfies:
(i) CIMA (f, s) 2 O, with equality if and only if all columns (f/a..() of (s)) are orthogonal.
(ii) Cpaa (f. is invariant to left multiplication of Jr(s) by an orthogonal matrix and right multiplication by permutation and diagonal matrices.
Property (i) formalises the geometric interpretation of IMA as an orthogonality condition on the columns of the Jacobian from $ 4.1, and property (ii) intuitively states that changes of orthonormal basis and permutations or rescalings of the columns of Jr do not affect their orthogonality. Next, we define a global IMA contrast w.r.t. a source distribution ps the expected local IMA contrast.
Definition 4.5 (Global IMA contrast). The global IMA contrast CIMA (f, Ps) of f w.r.t Ps is given by
Ciaa (f, Ps) Esrop. [CIMa (f.s)) / Caaa s) (s)
The global IMA contrast CIMA (f,ps) ps) thus quantifies the extent to which the IMA principle is violated for partieular solution (f, to the nonlinear ICA problem. We summarise its properties as follows. Proposition 4.6 (Properties of Ciaa pt)). The global IMA contrast Caaa Ps) from (9) satisfies:
(i) CIaa (f. Ps) 2 O, with equality iff. Jr(s) = ((s) almost surely w.r.t. Ps- where (s), D(s) Rax= are orthogonal and diagonal matrices, respectively;
(ii CIMA Pf) = CIMA (f, (f,Pa)),p.pp))))pp for any f f h- o P-1 and s = Ph(s), where P E R is permutation and h(s) - (11(),), ..., hn.(s))) an invertible element-wise functionn.
Property (i) is the distribution- analogue to (i) of Prop- 4.4 and only allows for orthogonality vi-
18-
0.4
olations on sets of measure zero w.r.t Ps- This
means that CImA can only be zero if f is an or- thogonal coordinate transformation almost every- where [19, 52, 66]. see Fig. 3 for an example. We par- ticularly stress property (ii), as precisely matches the inherent indeterminacy of nonlinear ICA: CIRa is blind to reparametrisation of the sources by permu- tation and element wise transformation.
-02
-06-
0.0
5.13
100
3.50
3.0
Figure 3: An example of (non-conformal) or- thogonal coordinate transformation from polar (left) to Cartesian (right) coordinates.
4.3 Theoretical analysis and justification of CIma
We now show that, under suitable assumptions on the generative model (1), large class of spurious solutions such as those based on the Darmois construction (4) or measure preserving automorphisms such as a from (5) as described in $ 2.1 exhibiit nonzero IMA contrast. Denote the class of non- linear ICA models satisfying (7) (IMA) by MIMA = {(f, p=) E F x P CiMa (f, Ps) 0} C F x P. Our first main theoretical result that, under mild assumptions on the observations, Darmois solutions will have strictly positive CIMA, making them distinguishable from those in MIMA-
Theorem 4.7. Assume the data generating process in (1) and assume that z; 1 2j for some i + j. Then any Darmois solution (f" Pu)) based on g" as defined in (4) satisfies CfMA(FD, Pu)) > 0. Thus can be distinguished from (f" PPu) based on the contrast CIMA- The proof is based on the fact that the Jacobian of gD is triangular (see Remark 2.4) and on the specific form of (4). A specific example of mixing process satisfying the IMA assumption is the case where f is a conformal sangle-preserving map.
solution satisfying CiMs (f, Pb)
Definition 4.8 (Conformal map). A smooth map f: RR" + R" is conformal if Jr = (ss){(s) Vs. where X: R + IR is a scalar field, and O E O(n) is orthogonal matrix.
Corollary 4.9. Under assumptions of Thm. 4.7, if additionally f is a conformal map, then (f,P%) MIMa for any Ps E P due to Prop. 4.6 (i), see Defn. 4.8. Based on Thm. 4.7, (f,Ps) is thus distinguishable from Darmois solutions (fD Pu))
This is consistent with a result that proves identifiability of conformal maps for n = 2 and conjectures it in general [39]. 12 However, conformal maps are only a small subset of all maps for which CIMA = O, as is apparent from the more flexible condition of Prop. 4.6 (i), compared to the stricter Defn. 4.8.
Note that Corollary 4.9 holds for any dimensionality
7

---

Ground
observatioss
Darmols:
MRA
Darmo MPA gr4
MLE,
144
Earns
10
IEU
Nambee meeddes
10
1.0
30a
Jos
40
01
Carmee
(a)
00
01
0.2
0.2
2.4
(b)
00
5
(c)
0.0 602
3df (d)
Namber layers
Figure 4: Top. Visual comparison of different nonlinear ICA solutions for r. = 2: (left to right) true sources; ob-
served mixtures; Darmois solution; true unmixing, composed with the mcasure preserving automorphism (MPA) from (5) (with rotation by */4); Darmois solution composed with the same MPA, maximum likelihood (A 0); and Cmm-rreuulrried approach
1). Bottom. Quantitative comparison of Cru.. for different spurious solutions: learnt Darmois solutions for (a) n 2, and (b) E (2, 3, 5, 10) dimensions; (c) composition of the
MPA (5) in n = 2 dim. with the true solution (blue) and a Darmois solution (red) for different angles. (d) Cina distribution for true MI. mixing (red) vs. Darmois solution (blue) for n = 5 dim., J. E {2, 22,3,44 43 layers.
Example 4.10 (Polar to Cartesian coordinate transform). Consider the non-conformal transformation
from polar Cartesian coordinates (see Fig. 3), defined as (z, y) f(r, 9) (r cos (0), sin) with independent sources s = (r, f), with r ~ UO0 R) and O ~ (O, 2 2.r). Then, CIMA (f, Px) = 0 and CIa (fD, Pu) > 0 for any Darmois solution (fD,Pu)
Pu) see Appendix D for details.
Finally, for the case in which the true mixing is linear, obtain the following result.
Corollary 4.11. Consider a linear ICA model, x = As, with E(s s] = L, and A E O(n) an orthog- onal, non-trivial mixing matrix, i.e., not the product of diagonal and a permutation matrix DP. If at most one of the 8; is Gaussian, then CIaA (A,
(A,,(A,)) Ps) ) = and CIMA (f2,
(fP, .Pu) > O.
In a "blind" setting, we may not know a priori whether the true mixing is linear or not, and thus choose learn nonlinear unmixing. Corollary 4.11 shows that, in this case, Darmois solutions still distinguishable from the true mixing via CIMA- Note that unlike in Corollary 4.9, the assumption that x; HK 2y for some i + j is not required for Corollary 4.11. In fact, due to Theorem 11 of [17), it follows from the assumed linear ICA model with non- Gaussian sources, and the fact that the mixing matrix is not the product of a diagonal and permutation matrix (see also Appendix A).
Having shown that the IMA principle allows distinguish a class of models (including, but not limited to conformal maps) from Darmois solutions, we next turn to a second well- known counterexample to identifiability: the "rotated- -aaussd-G MPA R (Pa) (5) from Defn. 2.5. Our second main theoretical result is that, under suitable assumptions, this class of MPAs can also be ruled out for "non-trivial R.
Theorem 4.12. Lel (f, Ps) E MiMa and assume that is a conformal map. Given R E On)), assume additionally that 3 al least one non Mon-GGassian 8; whose associated canonical basis vector e, is not transformed by R-1 R. inlo another canonical basis vector ej. Then Cisa (f a R (px), Ps) O.
Thm. 4.12 states that for conformal maps, applying the a (ps) transformation at the level of the sources leads to an increase in CIMA except for very specific rotations R that are "fine-tuned" to Ps in the sense that they permute all non- Gaussian sources with another sy. Interestingly, as for the linear case, non- Gaussiaanyyy again plays an important role in the proof of Thm. 4.12.
Experiments
Our theoretical results from $ 4 suggest that CIMA is a promising contrast function for nonlinear blind source separation. We test this empirically by evaluating the CIsA of spurious nonlinear ICA solutions ($ 5.1), and using it as a learning objective recover the true solution ($ 5.2).
We sample the ground truth sources from a uniform distribution in [0, 1|"; the reconstructed sources reference measure via the CDF transform. Unless
also mapped to the uniform hypercube
For different (a, y) can be made to have independent Gaussian components (198). IL.B), and CIrA" identifiability is lost; this shows that the assumption of Thm. 4.7 that 2; JL æ; for some i * j is crucial.
8

---

otherwise specified, the ground truth mixing f is a Möbius transformation [81] (i.e., a conformal map) with randomly sampled parameters, thereby satisfying Principle 4.1. In all of our experiments. we use JAX [12] and Distrax [13]. For additional technical details, equations and plots see Appendix E. The code to reproduce our experiments is available at this link.
5.1 Numerical evaluation of the CIMa contrast for spurious nonlinear ICA solutions
Learning the Darmois construction. To learn the Darmois construction from data, we use normalis- ing flows, see [35, 69]. Since Darmois solutions have triangular Jacobian (Remark 2.4), we use an architecture based on residual flows [16] which we constrain such that the Jacobian of the full model is triangular. This yields an expressive model which we train effectively via maximum likelihood.
CIma of Darmois solutions. To check whether Darmois solutions (learnt from finite data) can be distinguished from the true one, as predicted by Thm. 4.7, we generate 1000 random mixing functions for n = 2, compute the CIMA values of learnt solutions, and find that all values are indeed significantly larger than zero, see Fig. (a). The same holds for higher dimensions, see Fig. 4 (b) for results with 50 random mixings for n E {2, 3, 5, 10}: with higher dimensionality, both the mean and variance of the CIbA distribution for the learnt Darmois solutions generally attain higher values. 14 We confirmed these findings for mappings which not conformal, while still satisfying (7), in Appendix E.5.
Cima of MPAs. We also investigate the effect on CIsA of applying an MPA (-) from (5) to the true solution or a learnt Darmois solution. Results for n = 2 dim. for different rotation matrices R (parametrised by the angle () are shown in Fig. 4 (c). As expected, the behavior is periodic in and vanishes for the true solution (blue) at multiples of f/2, i... when R is a permutation matrix, as predicted by Thm. 4.. For the learnt Darmois solution (red, dashed) CImA remains larger than zero.
CIMA values for random MLPs. Lastly, we study the behavior of spurious solutions based on the Darmois construction under deviations from our assumption of CIMA 0 for the true mixing function. To this end, we use invertible MLPs with orthogonal weight initalisation and leaky_ tanh activations [29] as mixing functions; the more layers L are added to the mixing MLP, the larger a deviation from our assumptions is expected. We compare the true mixing and learnt Darmois solutions over 20 realisations for each L E {2, 3, 4J, n 5. Results are shown in figure Fig. 4 (d): the CIMA of the mixing MLPs grows with L; still, the one of the Darmois solution is typically higher.
Summary. We verify that spurious solutions can be distinguished from the true one based on CIMA-
5.2 Learning nonlinear ICA solutions with -eguulagler maximum likelihood
Experimental setup. To use CIMA as a learning signal, we consider a regularised maximum- likelihood approach, with the following objective: L(g) Ex X CIMA (g " ' Pyy, where g denotes the learnt unmixing, y = g(x) the reconstructed sources, and À 2 0 a Lagrange multiplier. For X = O, this corresponds to standard maximum likelihood estimation, whereas for À > O, L lower-bounds the likelihood, and recovers it exactly iff. "._t.,y) E MIMA. We train a residual flow g (with full Jacobian) to maximise L. For evaluation, we compute (i) the KL divergence to the true data likelihood, as a measure of goodness of for the learnt flow model; and (ii) the mean correlation coefficient (MCC) between ground truth and reconstructed sources [37, 49). We also introduce (iii) nonlinear extension of the Amari distance [5] between the true mixing and the learnt unmixing, which is larger than or equal to zero, with equality iff. the learnt model belongs to the BSS equivalence class (Defn. 2.2) of the true solution, see Appendix E.5 for details.
,y)),
Results. In Fig. 4 (Top), we show an example of the distortion induced by different spurious solutions forn = 2, and contrast it with solution learnt using our proposed objective (rightmost plot). Visually, we find that the -rguulalar solution (with = 1) recovers the true sources most faithfully. Quantitative results for 50 learnt models for each À f0.0, 0.5, 1.0} and in Fig. 5 (see Appendix E for additional plots) As indicated by the KL divergence values (left), most trained models achieve good fit to the data across all values of X. We observe that using CiaA (i.... À > 0) is beneficial for BSS, both in terms of our nonlincar Amari distance (center, lower is better) and MCC (right. higher is better), though we do not observe substantial difference between = 0.5 and À = 1.
(5,7) summarised
Summary: CImA can be a useful learning signal to recover the true solution.
"te latter possibly due to the increased difficulty of the learning task for larger
15 models with 7 have high outlier KL. values, scemingly less pronounced for nonzero values of In Appendix E.5, we also show that method is superior to linear ICA baseline, FastCAA [36].

---

1.8
1.0
15
AnMe disaanee
14-
0.8
04
6.5
10
066
0.2
0.5
0.5
3.2
63
0.5 1.6
1.0 0.0
Figure BSS Cua. Cegullauulrieed MLE for, side by side, n (blue) and (red) dim. with E 0.5, 1.0). (Left) KL-divergence between ground truth likelihood and Icarnt model; (center) nonlinear Amari distance given true mixing and learnt unmixing; (right) MCC between true and reconstructed sources.
6 Discussion
Assumptions on the mixing function. Instead of relying weak supervision in the form of auxiliary variables [28, 30, 37, 38, 41, 49], our IMA approach places additional constraints on the functional form of the mixing process. In similar vein, the minimal nonlinear distortion principle [108] pro- poses to favor solutions that are as close to linear as possible. Another example is the post-nonlinear model [98, 109], which assumes an element-wise nonlinearity applied after a linear mixing. IMA is different in that still allows for strongly nonlinear mixings (see, e.g., Fig. 3) provided that the columns of their Jacobians are (close to) orthogonal. In the related field of disentanglement [8, 58), line of work that focuses on image generation with adversarial networks [24] similarly proposes con- strain the "generator" function via regularisation of its Jacobian [82] or Hessian [74], though mostly from an empirically-driven. rather than from an identifiability perspective as the present work.
Towards identifiability with CIMA- The IMA principle rules out a large class of spurious solutions to nonlinear ICA. While we do not present full identifiability result, experiments show that CIma can be used to recover the BSS equivalence class, suggesting that identifiability might indeed hold, possibly under additional assumptions e.g., for conformal maps [39].
IMA and independence cause and mechanism. While inspired by measures of independence of cause and mechanism as traditionally used for cause-effect inference [18, 45, 46, 110], we view the IMA principle as addressing different question, the sense that they evaluate independence between different elements of the causal model. Any nonlinear ICA solution that satisfies the IMA Principle 4.1 can be turned into one with uniform reconstructed sources- tths satisfying IGCI as argued in 3 through composition with an element- -wiee transformation which. according to Prop. 4.6 (ii), leaves the Cima value unchanged. Both IGCI (6) and IMA (7) can therefore be fulfilled simultaneosly, while the former on its own is inconsequential for BSS as shown in Prop. 3.1.
BSS through algorithmic information. Algorithmic information theory has previously been pro- posed as a unifying framework for identifiable approaches to linear BSS [67, 68], in the sense that commonly-used contrast functions could, under suitable assumptions, be interpreted as proxies for the total complexity of the mixing and the reconstructed sources. However, to the best of our knowledge, the problem of specifying suitable proxies for the complexity of nonlinear mixing functions has not yet been addressed. We conjecture that our framework could be linked to this view, based on the additional assumption of algorithmic independence of causal mechanisms [43], thus potentially representing an approach to nonlineur BSS by minimisation of algorithmic complexity.
ICA for causal inference & causality for ICA. Past advances in ICA have inspired novel causal discovery methods [50, 64, 92]. The present work constitutes, to the best of our knowledge, the first. effort to use ideas from causality (specifically ICM) for BSS. An application of the IMA principle to causal discovery or causal representation learning [88] is an interesting direction for future work.
Conclusion. We introduce IMA, a path to nonlinear BSS inspired by concepts from causality. We postulate that the influences of different sources on the observed distribution should be approximately independent, and formalise this as an orthogonality condition on the columns of the Jacobian. We prove that this constraint is generally violated by well-known spurious nonlinear ICA solutions, and propose a regularised maximum likelihood approach which we empirically demonstrate to be effective in recovering the true solution. Our IMA principle holds exactly for orthogonal coordinate transformations, and is thus of potential interest for learning spatial representations [33], robot dynamics [63], or physics problems where orthogonal reference frames are common [66].
10

---

Acknowledgements
The authors thank Aapo Hyvärinen, Adrián Javaloy Bornás, Dominik Janzing, Giambattista Paras- candolo, Giancarlo Fissore, Nasim Rahaman, Patrick Burauel, Patrik Reizinger, Paul Rubenstein, Shubhangi Ghosh, and the anonymous reviewers for helpful comments and discussions.
Funding Transparency Statement
This work was supported by the German Federal Ministry of Education and Research (BMBF); Tübingen AI Center, FKZ: OIIS (IS18039B; and by the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645.
References
[1] Pierre Ablin, Lean-François Cardoso, and Alexandre Gramfort. Faster independent component analysis by preconditioning with hessian approximations. IEEE Transactions on Signal Processing, 66(15);
4040-4049, 2018.
[2] Khaled Alyani, Marco Congedo, and Maher Moakher. Diagonality measures of Hermitian positive- definite matrices with application the approximate joint diagonalization problem. Linear Algebra and its Applications, 528:290-320, 2017.
[3] Shun-ichi Amari. Information geometry. Japanese Journal of Mathematics, 16(1):1-48, 2021.
[4] Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. American
Mathematical Soc.. 2007.
[5] Shun-ichi Amari, Andrzej Cichocki, Howard Hua Yang, et al. A new learning algorithm for blind signal separation. In Advances in Neural Information Processing Systems, pages 757-763, 1996.
[6] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pazz Invariant risk minimization. arXiv preprint arXiv: 1907.02833, 2019.
[7] Elias Bareinboim. Andrew Foreyy and Judea Pearl. Bandits with unobserved confounders: A causal approach. Advances in Neural Information Processing Systems, 28:1342-113 1350, 2015.
[8] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspectives. IEEE Transactions on Puttern Analysis and Machine Intelligence. 35(8):1798 1828, 2013.
[9] M Besserve, N Shajarisales, B Schölkopf, and D Janzing. Group invariance principles for causal generative models. In 21 International Conference on Artificial Intelligence and Statistics (AISTATS 2018), pages 557-565. International Machine Learning Society, 2018.
[10] M Besserve, A Mehrjou, R Sun, and Schölkopf. Counterfactuals uncover the modular structure of deep generative models. In Eighth International Conference on Learning Representations (ICLR 2020), 2020.
[11] Patrick Blöbaum, Dominik Janzing, Takashi Washio, Shohei Shimizu, and Bernhard Schölkopf. Cause- effect inference by comparing regression errors. In International Conference on Artificial Intelligence and Statistics, pages 900-909. PMLR, 2018.
[12] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Miler JAX: composable transformations of Python+NumPy programs, 2018.
[13] Jake Bruce, David Budden, Mattco Hessel, George Papamakarios, and Francisco Ruiz. Distrax: Probabil-
ity distributions in JAX, 2021.
[14] Lars Buesing, Theophane Weber, Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean Captittt Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Counterfactualll- policy search, arXiv preprint arXiv: 1.11.2211 006222, 2018
[15] Jean-François Cardoso. The three easy routes to independent component analysis; contrasts and geometry. In Proc. ICA, volume 2001, 2001.
[16] Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and Jömn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances Neural Information Processing Systems, 2019.
[17] Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314, 1994.
[18] Povilas Daniušis, Dominik Janzing, Joris Mooij. Jakob Zscheischler. Bastian Steudel, Kun Zhang, and Bernhard Schölkopf. Inferring deterministic causal relations. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, pages 143-150, 2010.
[19] Gaston Darboux. Leçons sur les systemes orthogonaux et les coordonnées curvilignes. Sauthier-Villare 1910.
[20] George Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat, Conferences 1947, page 231, 1951.

---

[21] George Darmois. Analyse générale des liaisons stochastiques: etude particulière de I' l'analyse factorielle linéaire. Revue de I'Instiul international de statistique, pages 2-8, 1953.
[22]
Andrew Forney, Judea Pearl, and Elias Bareinboim. Counterfactual data-fusion for online reinforcement learners. In International Conference on Machine Learning, pages 1156-1166 PMLR, 2017.
[23] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transfcrable components. In International Conference on Machine Learning, pages 2839-2848. PMLR, 2016.
[24] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014.
[25] Alexander N Gorban and Ivan Yu Tyukin. Blessing of dimensionality: mathematical foundations of the statistical physies of data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(21 18):211022207 2018.
[26] Anirudh Goyal. Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In 9th International Conference on Learning Representations, 2021.
[27] Daniel Greenfeld and Uri Shalit. Robust leamnngg with the hilbert-schmidt independence criterion. In International Conference on Machine Learning, pages 3759-376550 -366. PMLR, 2020.
[28] Luigi Gresele, Paul K Rubenstein, Arash Mchrjou, Francesco Locatello, and Bernhard Schölkopf. The Incomplete Rosetta Stone problem: Identifiability results for multi-view nonlinear ICA. In Uncertainty in Artificial Intelligence, pages 217-227. PMLR, 2019.
[29] Luigi Gresele, Giancarlo Fissore, Adrián Javaloy, Bernhard Schölkopf, and Aapo Hyvarinen. Relative gradient optimization of the Jacobian term in unsupervised deep learning. Advances in Neural Information Processing Systems, 33, 2020.
[30] Hermanni Hälvä and Aapo Hyvärinen. Hidden Markow nonlinear ICA: nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939-948. PMLR, 2020.
Unsupervised learning from
[31] Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. Machine Learning, 10022)303-334, 2021.
[32] Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.
[33] Geoffrey E Hinton and Lawrence M Parsons. Frames of reference and mental imagery. Attention and
performance IX, pages 261-277, 1981.
[34] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in Neural Information Processing Systems, 21 21:689-696, 2008.
[35] Chin- Huang, David Krueger, Alexandre Lacoste, and Aaron C. Courville. Neural autoregressive flows. Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 2083-2092. PMLR, 2018
[36] Aapo Hyvärinen. Fast and robust fixed point algorithms for independent component analysis. IEEE transactions on Neural Newworks, 10(3):626-334 1999.
[37] Aapo Hyvärinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlincar ICA. In Advances in Neural Information Processing Systems, pages
2016.
[38] Aapo Hyvärinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
[39] Aapo Hyvärinen and Petteri Pajunen. Nonlincar independent component analysis: Existence and unique- ness results. Neural Networks, 12(3):429-43) 1999.
[40] Aapo Hyvärinen, Juha Karhunen, and Erkki Oja. Independent Component Analysis. John Wiley Sons,
Ltd, 2001.
[41] Aapo Hyvärinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859 868. PMLR, 2019.
[42] Dominik Janzing. Causal version of principle of insufficient reason and maxent. arXiv preprint arXiv:2102.039000 2021.
[43] Dominik Janzing and Bernhard Schölkopf. Causal inference using the algorithmic Markov condition. IEEE Transactions on Information Theory, 56 56(10):518-5199) 2010.
[44] Dominik J Janzing and Bernhard Schölkopf. Detecting confounding in multivariate linear models via spectral analysis. Journal of Causal Inference, 6(1), 2018
12

---

[45] Dominik Janzing, Patrik O Hoyer, and Bernhard Schölkopf. Telling cause from effect based on high- dimensional observations. In International Conference on Machine Learning, 2010.
[46] Dominik Janzing, Joris Mooij, Kun Zhang, Jan Lemeire, Jakob Zscheischler, Povilas Daniušis, Bastian Steudel, and Bernhard Schölkopf. information-gcomen approach to inferring causal directions. Artificial Intelligence, 182:1-31, 2012.
[47] Dominik Janzing, Bastian Steudel, Naji Shajarisales, and Berhhard Schölkopf. Justifying information- geometric causal inference. In Measures of complexity, pages 253-265. Springer, 2015.
[48] Dominik Janzing. Raphael Chaves, and Bernhard Schölkopf. Algorithmic independence of initial condition and dynamical law in thermodynamies and causal inference. New Journal of Physics, 18(9):
093052, 2016.
[49] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217 PMLR, 2020.
[50] Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International Conference on Artificial Intelligence and Statistics, pages 3520-3528. PMIR, 2021.
[51] Andrei N Kolmegorov. On tables of random numbers. Sankhyā: The Indian Journal of Statistics, Series A. pages 369-376, 1963.
[52] Gabriel Lamé. Leçons sur les coordonnées curvilignes et leurs diverses applications. Mallet-Bachelies 1859.
[53] Sanghack Lee and Elias Barcinboim. Structural causal bandits: where to intervene? Advances in Neural Information Processing Systems 31, 31, 2018
[54] Felix Leeb, Yashas Annadani, Stefan Bauer, and Bernhard Schölkopf. Structural autoenenders improve representations for generation and transfer. arXiv preprint arXiv:2006.07790 2020.
[55] Erich L Lehmann and George Casella. Theory of point extimation. Springer Science & Business Media, 2006.
[56] Jan Lemeire and Erik Dirkx. Causal models as minimal descriptions of multivariate systems, 2006.
[57] Jan Lemeire and Dominik Janzing. Replacing causal faithfulness with algorithmic independence of conditionals. Minds and Machines, 23(2):227-2453 2013.
[58] Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning disentangled representations. In International Conference on Machine Learning, pages 4114-4124. PMLR, 2019.
[59] Chaochao Lu, Bernhard Schölkopf, and José Miguel Hernández- Deconfounding reinforcement learning in observational settings. arXiv preprint arXiv: 1812. 2018.
[60] Chaochao Lu, Biwei Huang, Ke Wang, José Miguel Hernández- Kun Zhang, and Berhhard Schölkopf. Sample-efficient reinforcement learning via counterfactual-bass data augmentation. arXiv preprint arXiv:2012.0900 09092, 2020.
[61] Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 10869-10879, 2018.
[62] James Mahoney. Beyond correlational analysis: Recent innovations in theory and method. In Sociological forum, pages 575-593. JSTOR, 2001.
[63] Michael Mistry, Jonas Buchli, and Stefan Schaal. Inverse dynamics control of floating base systems using orthogonal decomposition. In 2010 IEEE International Conference on Robotics and Automation, pages 3406-3412. IEEE, 2010.
[64] Ricardo Pio Monti, Kun Zhang, and Aapo Hyvärinen. Causal discovery with general non-linear re- lationships using non-linear ICA. In Uncertainty in Artificial Intelligence, pages 185-195, PMLR, 2020.
[65] Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Berhhard Schölkopf. Distin- guishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17()):100312044 2016.
[66] Parry Moon and Domina Eberle Spencer. Field theory handbook, including coordinate systems, differential equations and their solutions. Springer, 1971.
[67] Petteri Pajunen. Blind source separation using algorithmic information theory. Neurocomputing, 22(1-3): 35-48, 1998.
13

---

[68] Petteri Pajunen. Blind source separation of natural signals based on approximate complexity minimization. In Proc. Int. Workshop on Independent Component Analyxis and Signal Separation (ICA '99), page 267. Citeseer, 1999.
[69] George Papamakarios, Erie Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 1-64, 2021.
[70] Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulay and Bernhard Schölkopf. Learning independent causal mechanisms. In International Conference on Machine Learning, pages 4036-4044.
PMLR, 2018.
[71] Judea Pearl. Causality. Cambridge university press, 2009,
[72] Judea Pearl and Elias Bareinboim. External validity: From do-calculus to transportability across popula- tions. Statistical Science, 29(4)557--595, 2014.
[73] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-lcam: Machine learning in python. The Journal of Machine Learning Research, 12:2825-2830, 2011.
[74] William S. Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, and Antonio Torralba. The Hessian penalty: A weak prior for unsupervised disentanglement. In ECCV, volume 12351, pages 581-597. Springer, 2020.
[75] Jonas Peters and Peter Bühlmann. Identifiability of Gaussian structural equation models with equal error variances. Biometrika, 101(1):219- 228, 2014.
[76] Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15:2009-2053, 2014.
[77] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction:
identification and confidence intervals. Journal of the Royal Statistical Society. Series B (Statistical Methodology), pages 947-1012, 2016.
[78] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[79] Dinh- TuanTTunnPe Pham and J-F Cardoso. Blind separation of instantancous mixtures of nonstationary sources. IEEE Transactions on Signal Processing, 49(9):1837-1848, 2001.
[80] Dinh Tuan Pham and Philippe Garat. Blind separation of mixture of independent sources through quasi-maximum likelihood approach. IEEE Transactions on Signal Processing, 45(7):1112-1725, 1997.
[81] Robert Phillips. Liouville's 's thcorem. Pacific Journal of Mathematics, 28(2):397-400) 405, 1969.
[82] Aditya Ramesh, Youngduck Choi, and Yann LeCun. spectral regularizer for unsupervised disentangle- ment. arXiv preprint arXiv:1812.011611 2018.
[83] Danilo Jimenez Rezende. Short notes on divergence measures, 2018.
[84] Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309 1342, 2018
[85] Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regression: Heterogencous data meet causality. Journal of the Royal Statistical Sociery: Series B (Statistical Methodologyy, 83(2):215-246, 2021.
[86] Wesley C Salmon. Scientific explanation and the causal structure of the world. Princeton University Press, 2020.
[87] Schölkopf, D Janzing, J Peters, Sgouritsa, Zhang, and Mooij. On causal and anticausal learning. 29th International Conference on Machine Learning (ICML 2012), pages 1255-1262. International Machine Learning Society, 2012.
[88] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEF, 2021.
[89] Manfred R Schroeder. Listening with two ears. Music perception, 10(3):255-280, 1993.
[90] Naji Shajarisales. Dominik Janzing, Bernhard Schoelkopf, and Michel Besserve. Telling cause from effect in deterministic linear dynamical systems. In International Conference on Machine Learning, pages 285-294, PMLR, 2015.
[91] Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Disentangled generative causal representation learning. arXiv preprint 27X:v22010.02637, 2020.
[92] Shohei Shimizu, Patrik Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non Gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
[93] VP Skitovié. On a property of normal distribution. In Doklady Akad. Nauk, 1953.
14

---

[94] James R Smart. Modem geometries. Brooks/Cole Pacific Grove, CA, 1998.
[95] Mirjam Soeten. Conformal maps and the theorem of Liouville, PhD thesis, Faculty of Science and Engineering, 2011.
[96] B. Steudel, D. Janzing. and B. Schölkopf. Causal Markov condition for submodular information measures.
In A. Kalai and M. Mohri, editors, Conference on Learning Theory (COL.T), pages 464-476, Madison, WI, USA, 2010. OmniPress.
[97] Adarsh Subbaswamy. Peter Schulam, and Suchi Saria. Preventing failures due to dataset shift: Learning predictive models that transport. In The 22nd International Conference Artificial Intelligence and Statistics, pages 3118 3127. PMLR, 2019.
[98] Anisse Taleb and Christian Jutten. Source separation in post-nonlinear mixtures. IEEE Transactions on Signal Processing, 47( (110)22877-28000 1999.
[99] Charles Tilly. Historical analysis political processes. Handbook of sociological theory, pages
567-588. Springer, 2001.
[100] Ruy Tojeiro. Liouville's theorem revisited. Enseignement Mathematique, 53112))667 2007.
[101] Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournapcau, Evgeni Burovski, Pcaru Peterson, Warren Weckesser, Jonathan Bright, ct al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):211-72,, 2020.
[102] Julius von Kügelgen, Alexander Mey, and Marco Loog. Semi- geni-generetive modelling: Covariate-shii adaptation with cause and effect features. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1361-1369. PMLR, 2019.
[103] Julius von Kügelgen, Alexander Mey, Marco Loog, and Bernhard Schölkopf. Semi-supervised learning, causality, and the conditional cluster assumption. In Conference on Uncertainty in Artificial Intelligence. pages 1-10. PMLR, 2020.
[104] Julius Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. In ICLR 2020 Workshop on "Causal Learning for Decision Making", 2020.
[105] Julius von Kügelgenn Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances Neural Information Processing Systems, 2021.
[106] Junzhe Zhang and Elias Bareinboim. Near-optimal reinforcement learning in dynamic treatment regimes. Advances in Neural Information Processing Systems, 32:13401-13411, 2019.
[107] Zhang. B. Schölkopf, K. Muandet, and Z. Wang. Domain adaptation under target and conditional shift.
In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of JMLR Workshop and Conference Proceedings, pages 819-827, 2013.
[108] Kun Zhang and Laiwan Chan. Minimal nonlinear distortion principle for nonlinear independent compo- nent analysis. Journal of Machine Learning Research, 9N45):2455-2488, 2008
[109] Kun Zhang and Aapo Hyvärinen. On the identifiability of the post-nonlinear causal model. In Proceedings the 25th Conference on Uncertainty in Artificial Intelligence, pages 647-655. AUAI Press, 2009.
[110) Jakob Zscheischler, Dominik Janzing, and Kun Zhang. Testing whether linear equations are causal: free probability theory approach. In 27th Conference on Uncertainty in Artificial Intelligence, pages 839-847, 2011.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See $ 6, where we discuss limitations of our theory (e.g. lines 354-357) and open questions.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work is mainly theoretical, and we believe does not bear immediate negative societal impacts.
(d) Have you read the ethies review guidelines and ensured that your paper conforms them?
[Yes]
you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] We formally define the problem setting in $ and $ 3, and transparently state and discuss our assumptions in $ 4. (b) Did you include complete proofs of all theoretical results? [Yes] Due to space constraints, full proof's and detailed explanations are mainly reported in appendix C and appendix D; the proof of Prop. 3.1 given the main text.
you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (cither in the supplemental material or as a URL)? [Yes] The code, data, and the configuration files are included in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters. how they were chosen)? [Yes] All details, including hyperparameters, seed for random number generators, etc. are specified in the configuration files which will be included in the supplemental.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments mul- tiple times)? [Ys]] We visualized the distribution of the considered quantities via histograms and violin plots, see e.g. Figure 4 and Figure 5.
(d) Did you include the total amount compute and type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] They are specified appendix E.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the Python libraries JAX and Distrax and cited the creators in the article.
(b) Did you mention the license of the assets? [Yes] Both packages have Apache License 2.0; we report this in the appendices.
(c) Did you include any new assets either in the supplemental material Implementations our proposed methods and metrics will be provided in the supplemental. (d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
URL? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "fVC2Kp-5ubB", "reviewer": "Reviewer_ovUQ", "summary": "The paper proposes a criterion for non-linear ICA as applied to Blind Source Separation inspired by Independence of Causal Mechanisms in the causal discovery literature. The general idea is to constrain the gradients of the output with respect to the input sources to be orthogonal to each other; this is linked to the notion that the mixing mechanism is in some sense independent (or not \"too fine-tuned\") to the input sources. The authors show that this constraint eliminates a number of known counterexamples to identifiability in non-linear ICA, and test it experimentally to show that helps to characterize true solutions (both as an evaluation metric, and as a regularizer). The authors conclude with a discussion putting their contributions into context.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is clearly written and the technical properties and limitations of the method are well-discussed. The experiments are well-designed to support the claims in the paper.\n\nMy main concern with the paper is that the discussion of ICM as motivation for IMA is a bit long and imprecise. While I do appreciate the review of ICM ideas, the notion of \"algorithmic independence\" is still rather vague (this is, IMO, often a problem in a lot of the ICM literature). Although two different measurements of the concept are given (in terms of Kolmogorov complexity and IGCI), a concrete counterexample of what it would mean to have \"fine-tuning\" between the cause and the conditional would be helpful. In particular, is there an example that could be given (e.g., a modification of the cocktail party example) where this independence would not hold? Perhaps, an operator might have changed some properties of the recording to better capture certain parts of the input sources?\n\nA concrete example of this type would also give a better sense of when this functional restriction may or may not be appropriate in practice. When specifying identifying assumptions, this kind of grounding is generally useful because, by definition, there is no way to test these assumptions in practice.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do a good job of noting exactly what they establish regarding IMA in the paper---that their measure eliminates certain counterexamples---and what they do not---full identifiability under the IMA restriction.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "egpnK-VX8Rt", "reviewer": "Reviewer_VVZG", "summary": "The paper proposes applying the theory of independent causal mechanisms (ICM), from causal discovery, to nonlinear ICA approaches to the blind source separation problem (BSS).\n\nICM is based on the idea that variables in a system are algorithmically independent or do not share information with each other. The paper shows, however, that ICM is not sufficient for BSS as spurious solutions exist which are not equivalent to the truth.\n\nTo address these difficulties, the paper proposes a new condition for identifiability which places an orthogonality condition on the columns of the Jacobian and provides information theoretic and geometric interpretations. The paper then shows theoretically that a large class of spurious solutions are not admitted under the IMA formulation. \n\nExperiments on toy examples demonstrate that the model is identifiable under these spurious solutions.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general the paper is well written (though a bit difficult to parse for a non-specialist) and the theoretical contributions are rigorous. I did not check proofs. \n\nI am familiar with the related causal inference literature but less so the ICA literature so it is difficult for me to assess the significance of the class of spurious solutions references in the paper and whether these correspond to real world obstacles or are more of general theoretical interest. If the authors could clarify this, it may influence my final score as it seems this is sort of the crux of whether the paper constitutes a significant advance since IMA is not compared to other ICA approaches in any other way.\n\nI found the experimental results section to be particularly weak as no baselines are included, only the spurious cases are investigated and there are no results on real data. This leaves the reader unclear on several points: (i) whether IMA works similarly in practice to existing ICA approaches (despite the theoretical results), (ii) whether IMA is competitive enough with other ICA approaches on the non-spurious cases to justify the support for the spurious cases, (iii) whether the spurious cases are of practical significance in any real world datasets.\n\nThe paper would be significantly improved if other ICA baselines were included as well as experiments on the non spurious cases.\n\n===\n\nI increased my review to 7 based on the discussions and other reviews. See discussion below.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses limitations", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "E5iWBHEs7rH", "reviewer": "Reviewer_tDZV", "summary": "This work proposes a new regularization scheme to improve identifiability in the _nonlinear blind source separation problem_ (nonlinear ICA) inspired by the notion of _independence of mechanism_ coming from causal inference. The idea is to consider only mixing functions which have a Jacobian matrix with orthogonal columns everywhere. They show how this extra assumption allows to exclude two classical types of degeneracies in nonlinear ICA, although without showing identifiability. The theoretical claims are validated by synthetic experiments. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nRecent works on nonlinear ICA have focused on imposing structure on the distribution of the source via auxiliary variables. Instead, this work proposes to put restrictions on the mixing function f. Although different assumptions on f have been explored in the past (cited by the authors), their independence of mechanism assumption (IMA) hasn’t (to the best of my knowledge). \n\nQuality:\n\nI have not reviewed the proofs, however the various theoretical claims seem reasonable and are clearly presented. I was happy to read about new findings in that interesting direction. However, I have some concerns with the overall framing of the paper.\n\nThe inspiration for the authors' approach is the principle of independent causal mechanism (ICM). However, it is not clear to me what is the link between ICM and the orthogonality of the columns of the Jacobian of the mixing function. The authors seem to suggest a connection, but it felt somewhat vague. ICM is about algorithmic independence of mechanisms in a structural causal model (SCM), and here we regularize for orthogonality of the columns of the Jacobian of the mixing function.  \n\nIn causality, the term “mechanism” is typically reserved to the conditional distribution of a variable given its direct causal parents (or its associated structural equation). In this work, “mechanism” is used to denote a partial derivative of f w.r.t. some source s_i. This is an unnecessary and confusing clash of terminology in my opinion. Consequently, I do not think independent mechanism analysis is a good name for this contribution.\n\nThe authors also seem to suggest a connection between the orthogonality of partial derivatives and their algorithmic independence (lines 192-200), but again, this is a bit vague and unjustified (or maybe I missed an important point).  \n\nThe above points can be summarized as follows: I am not convinced that ICM and algorithmic independence is the correct language to describe the authors’ contribution, unless more justification is added to the paper.\n\nSection 3 shows how “regularizing” for algorithmic independence between p(s) and f via a proxy (IGCI) is not sufficient for identifiability. It is always interesting to have negative results, but, following up on the above paragraph, the connection to ICM feels a bit forced.\n\nInstead of showing identifiability for the model class they consider, the authors show how it excludes two types of degeneracies, namely the Darmois construction and the “rotated-Gaussian measuring preserving automorphisms”. This is not sufficient to show the model class is identifiable, although it is a good step in that direction. The authors acknowledge this fact at the very end of the paper (in the discussion section), I believe this should’ve been stressed earlier in the paper as well as near the theorem statements, since I believe it is not a typical thing to do in this literature (usually, there’s a theorem showing identifiability).\n\nIn corollary 4.11, it would be nice to have an intuition for why we need non-Gaussianity, can’t we just apply corollary 4.9 directly (since A is a conformal map)? What am I missing?\nIt would be good to have experiments with non-conformal maps satisfying the IMA assumption (I didn’t see any).\n\nMinor: \nWould be great to have an intuition for why the Darmois construction yields independent sources.\n\nThe presentation of information geometric causal inference is very fast and I could not understand its meaning. \n\nLine 332: Would be good to have a brief description of the MCC and the Amari metrics.\n\nClarity:\n\nI enjoyed reading the paper overall. \n\nSignificance:\n\nI believe this work has average significance, given how strong the conditions are on the mixing function and the fact that we do not know if it holds in practice (even partially) due to lack of real world experiments. \n\nThat being said, I believe imposing constraints on the mixing function to improve identifiability is an interesting direction with potential significance.\n\n-------\nAfter reading the author's rebuttal, I decided to raise my score from 5 to 6. See my comment above.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "CVew-UL_SY", "reviewer": "Reviewer_hbzz", "summary": "The authors propose to borrow concepts from causality, in particular \"independent causal mechanisms\" , to propose a novel framework termed \"independent mechanism analysis\" (IMA), which can provide non-spurious solutions to nonlinear blind source separation. IMA can be seen as a restricted nonlinear ICA model, where the contributions of the latent variables $z_i$ to the nonlinear mixing $\\mathbf{f}$ are independent, i.e. the columns of the Jacobian $J_\\mathbf{f}$ are orthogonal.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed IMA framework is novel, and provides an alternative solution to the nonlinear ICA (equivalently nonlinear BSS) problem. It is by now known that nonlinear ICA is not possible when dealing with general nonlinear mixing functions by only assuming independence of the latent components. Recent approaches usually involve using inductive biases in the form of auxiliary information to condition the latent distribution, without constraining the mixing. The authors propose to take a different path, by constraining the mixing instead.\nThe paper is very well written, and was a joy to read. Section 2 does a very good job of laying the ground for the IMA framework. Readers without knowledge about nonlinear ICA or the ICM framework will find this section very helpful.\nThe theoretical results are sound, and the assumptions are well justified. \nThe new IMA contrast functions is very useful, and can be used in practice to rule out a lot of spurious solutions to the nonlinear ICA problem.\n\nOverall, this work is original, and of great significance. Learning identifiable representations is an important topic in unsupervised learning, and the provided framework is an important step in this direction.\nI don't have much to criticize about the paper. I would have liked to see a full identifiability theory, but I believe it will follow in future work, and I am satisfied with the quality and quantity of the current contribution.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the paper are well explained and discussed in section 6.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "9euLLDmhvB8", "reviewer": "Reviewer_989y", "summary": "The paper aims at the identifiability of nonlinear ICA. It proposes a principle as a constraint of mixture functions based on the modified independent causal mechanism principle in the causal discovery literature; investigates the identifiability of nonlinear ICA under such an assumption; then applies it to nonlinear ICA. It also shows that the identifiability results can cover some non-identifiable cases in the existed literature.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The idea of the work is interesting: it adjusted the ICM principle in the related field of causal discovery for nonlinear ICA.\n2. The paper is in general well written and properly introduces the method. The theoretical analysis and examples are appreciated.\n\nMy minor concern is how restrictive is the proposed IMA in line 199 which imposes constraints on the determinant of the Jacobin matrix. Although the information-theoretic interpretation and geometric interpretation were trying to further explain IMA, I didn't get help from the given information without further details and explanation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel method for non-linear ICA. The idea is to constrain partial derivatives of the mixing function with respect to each source to be orthogonal. The authors connect this idea to the principle of independent mechanisms. The authors show that the constraint allows to deal with known counterexamples to identifiability in non-linear ICA. The method is evaluated on simulated toy examples. Reviewer hbzz praised the originality and significance of the work. Several reviewers (tDZV, ovUQ, 989y) criticized that the connection of IMA to other principles (such as algorithmic independence and information geometric causal inference) is vague or difficult to understand. In the responses, the authors expanded on the connections between the principles and proposed to de-emphasise the connection to algorithmic independence, while emphasizing the connection to information geometric causal inference. Several reviewers (VVZG, ovUQ, hbzz) agree that the paper is well-written. Overall, the authors have addressed the main concerns in the discussion. 
Decision: Accept (Poster)