Independent mechanism analysis, a new concept?

1 Introduction . One of the goals of unsupervised learning is to uncover properties of the data generating process , such as latent structures giving rise to the observed data . Identifiability [ 55 ] formalises this desideratum : under suitable assumptions , a model learnt from observations should match the ground truth , up to well-defined ambiguities . Within representation learning , identifiability has been studied mostly in the context of independent component analysis ( ICA ) [ 17 , 40 ] , which assumes that the observed data x results from mixing unobserved independent random variables si referred to as sources . The aim is to recover the sources based on the observed mixtures alone , also termed blind source separation ( BSS ) . A major obstacle to BSS is that , in the nonlinear case , independent component estimation does not necessarily correspond to recovering the true sources : it is possible to give counterexamples where the observations are transformed into components yi which are independent , yet still mixed with respect to the true sources si [ 20 , 39 , 98 ] . In other words , nonlinear ICA is not identifiable . In order to achieve identifiability , a growing body of research postulates additional supervision or structure in the data generating process , often in the form of auxiliary variables [ 28 , 30 , 37 , 38 , 41 ] . In the present work , we investigate a different route to identifiability by drawing inspiration from the field of causal inference [ 71 , 78 ] which has provided useful insights for a number of machine learning tasks , including semi-supervised [ 87 , 103 ] , transfer [ 6 , 23 , 27 , 31 , 61 , 72 , 84 , 85 , 97 , 102 , 107 ] , reinforcement [ 7 , 14 , 22 , 26 , 53 , 59 , 60 , 106 ] , and unsupervised [ 9 , 10 , 54 , 70 , 88 , 91 , 104 , 105 ] learning . To this end , we interpret the ICA mixing as a causal process and apply the principle of independent causal mechanisms ( ICM ) which postulates that the generative process consists of independent modules which do not share information [ 43 , 78 , 87 ] . In this context , “ independent ” does not refer to statistical independence of random variables , but rather to the notion that the distributions and functions composing the generative process are chosen independently by Nature [ 43 , 48 ] . While a formalisation of ICM [ 43 , 57 ] in terms of algorithmic ( Kolmogorov ) complexity [ 51 ] exists , it is not computable , and hence applying ICM in practice requires assessing such non-statistical independence ∗Equal contribution . Code available at : https : //github.com/lgresele/independent-mechanism-analysis 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with suitable domain specific criteria [ 96 ] . The goal of our work is thus to constrain the nonlinear ICA problem , in particular the mixing function , via suitable ICM measures , thereby ruling out common counterexamples to identifiability which intuitively violate the ICM principle . Traditionally , ICM criteria have been developed for causal discovery , where both cause and effect are observed [ 18 , 45 , 46 , 110 ] . They enforce an independence between ( i ) the cause ( source ) distribution and ( ii ) the conditional or mechanism ( mixing function ) generating the effect ( observations ) , and thus rely on the fact that the observed cause distribution is informative . As we will show , this renders them insufficient for nonlinear ICA , since the constraints they impose are satisfied by common counterexamples to identifiability . With this in mind , we introduce a new way to characterise or refine the ICM principle for unsupervised representation learning tasks such as nonlinear ICA . Motivating example . To build intuition , we turn to a famous example of ICA and BSS : the cocktail party problem , illustrated in Fig . 1 ( Left ) . Here , a number of conversations are happening in parallel , and the task is to recover the individual voices si from the recorded mixtures xi . The mixing or recording process f is primarily determined by the room acoustics and the locations at which microphones are placed . Moreover , each speaker influences the recording through their positioning in the room , and we may think of this influence as ∂f/∂si . Our independence postulate then amounts to stating that the speakers ’ positions are not fine-tuned to the room acoustics and microphone placement , or to each other , i.e. , the contributions ∂f/∂si should be independent ( in a non-statistical sense ) .1 Our approach . We formalise this notion of independence between the contributions ∂f/∂si of each source to the mixing process ( i.e. , the columns of the Jacobian matrix Jf of partial derivatives ) as an orthogonality condition , see Fig . 1 ( Right ) . Specifically , the absolute value of the determinant |Jf | , which describes the local change in infinitesimal volume induced by mixing the sources , should factorise or decompose as the product of the norms of its columns . This can be seen as a decoupling of the local influence of each partial derivative in the pushforward operation ( mixing function ) mapping the source distribution to the observed one , and gives rise to a novel framework which we term independent mechanism analysis ( IMA ) . IMA can be understood as a refinement of the ICM principle that applies the idea of independence of mechanisms at the level of the mixing function . Contributions . The structure and contributions of this paper can be summarised as follows : • we review well-known obstacles to identifiability of nonlinear ICA ( § 2.1 ) , as well as existing ICM criteria ( § 2.2 ) , and show that the latter do not sufficiently constrain nonlinear ICA ( § 3 ) ; • we propose a more suitable ICM criterion for unsupervised representation learning which gives rise to a new framework that we term independent mechanism analysis ( IMA ) ( § 4 ) ; we provide geometric and information-theoretic interpretations of IMA ( § 4.1 ) , introduce an IMA contrast function which is invariant to the inherent ambiguities of nonlinear ICA ( § 4.2 ) , and show that it rules out a large class of counterexamples and is consistent with existing identifiability results ( § 4.3 ) ; • we experimentally validate our theoretical claims and propose a regularised maximum-likelihood learning approach based on the IMA constrast which outperforms the unregularised baseline ( § 5 ) ; additionally , we introduce a method to learn nonlinear ICA solutions with triangular Jacobian and a metric to assess BSS which can be of independent interest for the nonlinear ICA community . 1For additional intuition and possible violations in the context of the cocktail party problem , see Appendix B.4 . 2 Background and preliminaries . Our work builds on and connects related literature from the fields of independent component analysis ( § 2.1 ) and causal inference ( § 2.2 ) . We review the most important concepts below . 2.1 Independent component analysis ( ICA ) . Assume the following data-generating process for independent component analysis ( ICA ) x = f ( s ) , ps ( s ) = ∏n i=1 psi ( si ) , ( 1 ) where the observed mixtures x ∈ Rn result from applying a smooth and invertible mixing function f : Rn → Rn to a set of unobserved , independent signals or sources s ∈ Rn with smooth , factorised density ps with connected support ( see illustration Fig . 2b ) . The goal of ICA is to learn an unmixing function g : Rn → Rn such that y = g ( x ) has independent components . Blind source separation ( BSS ) , on the other hand , aims to recover the true unmixing f−1 and thus the true sources s ( up to tolerable ambiguities , see below ) . Whether performing ICA corresponds to solving BSS is related to the concept of identifiability of the model class . Intuitively , identifiability is the desirable property that all models which give rise to the same mixture distribution should be “ equivalent ” up to certain ambiguities , formally defined as follows . Definition 2.1 ( ∼-identifiability ) . Let F be the set of all smooth , invertible functions f : Rn → Rn , and P be the set of all smooth , factorised densities ps with connected support on Rn . Let M ⊆ F×P be a subspace of models and let ∼ be an equivalence relation on M. Denote by f∗ps the push-forward density of ps via f . Then the generative process ( 1 ) is said to be ∼-identifiable on M if ∀ ( f , ps ) , ( f̃ , ps̃ ) ∈ M : f∗ps = f̃∗ps̃ =⇒ ( f , ps ) ∼ ( f̃ , ps̃ ) . ( 2 ) If the true model belongs to the model class M , then ∼-identifiability ensures that any model in M learnt from ( infinite amounts of ) data will be ∼-equivalent to the true one . An example is linear ICA which is identifiable up to permutation and rescaling of the sources on the subspace MLIN of pairs of ( i ) invertible matrices ( constraint on F ) and ( ii ) factorizing densities for which at most one si is Gaussian ( constraint on P ) [ 17 , 21 , 93 ] , see Appendix A for a more detailed account . In the nonlinear case ( i.e. , without constraints on F ) , identifiability is much more challenging . If si and sj are independent , then so are hi ( si ) and hj ( sj ) for any functions hi and hj . In addition to permutation-ambiguity , such element-wise h ( s ) = ( h1 ( s1 ) , ... , hn ( sn ) ) can therefore not be resolved either . We thus define the desired form of identifiability for nonlinear BSS as follows . Definition 2.2 ( ∼BSS ) . The equivalence relation ∼BSS on F × P defined as in Defn . 2.1 is given by ( f , ps ) ∼BSS ( f̃ , ps̃ ) ⇐⇒ ∃P , h s.t . ( f , ps ) = ( f̃ ◦ h−1 ◦P−1 , ( P ◦ h ) ∗ps̃ ) ( 3 ) where P is a permutation and h ( s ) = ( h1 ( s1 ) , ... , hn ( sn ) ) is an invertible , element-wise function . A fundamental obstacle—and a crucial difference to the linear problem—is that in the nonlinear case , different mixtures of si and sj can be independent , i.e. , solving ICA is not equivalent to solving BSS . A prominent example of this is given by the Darmois construction [ 20 , 39 ] . Definition 2.3 ( Darmois construction ) . The Darmois construction gD : Rn → ( 0 , 1 ) n is obtained by recursively applying the conditional cumulative distribution function ( CDF ) transform : gDi ( x1 : i ) : = P ( Xi ≤ xi|x1 : i−1 ) = ∫ xi −∞ p ( x ′ i|x1 : i−1 ) dx′i ( i = 1 , ... , n ) . ( 4 ) The resulting estimated sources yD = gD ( x ) are mutually-independent uniform r.v.s by construction , see Fig . 2a for an illustration . However , they need not be meaningfully related to the true sources s , and will , in general , still be a nonlinear mixing thereof [ 39 ] .2 Denoting the mixing function corresponding to ( 4 ) by fD = ( gD ) −1 and the uniform density on ( 0 , 1 ) n by pu , the Darmois solution ( fD , pu ) thus allows construction of counterexamples to ∼BSS-identifiability on F × P .3 Remark 2.4. gD has lower-triangular Jacobian , i.e. , ∂gDi /∂xj = 0 for i < j . Since the order of the xi is arbitrary , applying gD after a permutation yields a different Darmois solution . Moreover , ( 4 ) yields independent components yD even if the sources si were not independent to begin with.4 2Consider , e.g. , a mixing f with full Jacobian which yields a contradiction to Defn . 2.2 , due to Remark 2.4 . 3By applying a change of variables , we can see that the transformed variables in ( 4 ) are uniformly distributed in the open unit cube , thereby corresponding to independent components [ 69 , § 2.2 ] . 4This has broad implications for unsupervised learning , as it shows that , for i.i.d . observations , not only factorised priors , but any unconditional prior is insufficient for identifiability ( see , e.g. , [ 49 ] , Appendix D.2 ) . Another well-known obstacle to identifiability are measure-preserving automorphisms ( MPAs ) of the source distribution ps : these are functions a which map the source space to itself without affecting its distribution , i.e. , a∗ps = ps [ 39 ] . A particularly instructive class of MPAs is the following [ 49 , 58 ] . Definition 2.5 ( “ Rotated-Gaussian ” MPA ) . Let R ∈ O ( n ) be an orthogonal matrix , and denote by Fs ( s ) = ( Fs1 ( s1 ) , ... , Fsn ( sn ) ) and Φ ( z ) = ( Φ ( z1 ) , ... , Φ ( zn ) ) the element-wise CDFs of a smooth , factorised density ps and of a Gaussian , respectively . Then the “ rotated-Gaussian ” MPA aR ( ps ) is aR ( ps ) = F −1 s ◦Φ ◦R ◦Φ−1 ◦ Fs . ( 5 ) aR ( ps ) first maps to the ( rotationally invariant ) standard isotropic Gaussian ( via Φ−1 ◦ Fs ) , then applies a rotation , and finally maps back , without affecting the distribution of the estimated sources . Hence , if ( f̃ , ps̃ ) is a valid solution , then so is ( f̃ ◦ aR ( ps̃ ) , ps̃ ) for any R ∈ O ( n ) . Unless R is a permutation , this constitutes another common counterexample to ∼BSS-identifiability on F × P . Identifiability results for nonlinear ICA have recently been established for settings where an auxiliary variable u ( e.g. , environment index , time stamp , class label ) renders the sources conditionally independent [ 37 , 38 , 41 , 49 ] . The assumption on ps in ( 1 ) is replaced with ps|u ( s|u ) = ∏n i=1 psi|u ( si|u ) , thus restricting P in Defn . 2.1 . In most cases , u is assumed to be observed , though [ 30 ] is a notable exception . Similar results exist given access to a second noisy view x̃ [ 28 ] .

Summary: The authors propose to borrow concepts from causality, in particular "independent causal mechanisms" , to propose a novel framework termed "independent mechanism analysis" (IMA), which can provide non-spurious solutions to nonlinear blind source separation. IMA can be seen as a restricted nonlinear ICA model, where the contributions of the latent variables $z_i$ to the nonlinear mixing $\mathbf{f}$ are independent, i.e. the columns of the Jacobian $J_\mathbf{f}$ are orthogonal.