Independent mechanism analysis, a new concept?

1 Introduction . One of the goals of unsupervised learning is to uncover properties of the data generating process , such as latent structures giving rise to the observed data . Identifiability [ 55 ] formalises this desideratum : under suitable assumptions , a model learnt from observations should match the ground truth , up to well-defined ambiguities . Within representation learning , identifiability has been studied mostly in the context of independent component analysis ( ICA ) [ 17 , 40 ] , which assumes that the observed data x results from mixing unobserved independent random variables si referred to as sources . The aim is to recover the sources based on the observed mixtures alone , also termed blind source separation ( BSS ) . A major obstacle to BSS is that , in the nonlinear case , independent component estimation does not necessarily correspond to recovering the true sources : it is possible to give counterexamples where the observations are transformed into components yi which are independent , yet still mixed with respect to the true sources si [ 20 , 39 , 98 ] . In other words , nonlinear ICA is not identifiable . In order to achieve identifiability , a growing body of research postulates additional supervision or structure in the data generating process , often in the form of auxiliary variables [ 28 , 30 , 37 , 38 , 41 ] . In the present work , we investigate a different route to identifiability by drawing inspiration from the field of causal inference [ 71 , 78 ] which has provided useful insights for a number of machine learning tasks , including semi-supervised [ 87 , 103 ] , transfer [ 6 , 23 , 27 , 31 , 61 , 72 , 84 , 85 , 97 , 102 , 107 ] , reinforcement [ 7 , 14 , 22 , 26 , 53 , 59 , 60 , 106 ] , and unsupervised [ 9 , 10 , 54 , 70 , 88 , 91 , 104 , 105 ] learning . To this end , we interpret the ICA mixing as a causal process and apply the principle of independent causal mechanisms ( ICM ) which postulates that the generative process consists of independent modules which do not share information [ 43 , 78 , 87 ] . In this context , “ independent ” does not refer to statistical independence of random variables , but rather to the notion that the distributions and functions composing the generative process are chosen independently by Nature [ 43 , 48 ] . While a formalisation of ICM [ 43 , 57 ] in terms of algorithmic ( Kolmogorov ) complexity [ 51 ] exists , it is not computable , and hence applying ICM in practice requires assessing such non-statistical independence ∗Equal contribution . Code available at : https : //github.com/lgresele/independent-mechanism-analysis 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . with suitable domain specific criteria [ 96 ] . The goal of our work is thus to constrain the nonlinear ICA problem , in particular the mixing function , via suitable ICM measures , thereby ruling out common counterexamples to identifiability which intuitively violate the ICM principle . Traditionally , ICM criteria have been developed for causal discovery , where both cause and effect are observed [ 18 , 45 , 46 , 110 ] . They enforce an independence between ( i ) the cause ( source ) distribution and ( ii ) the conditional or mechanism ( mixing function ) generating the effect ( observations ) , and thus rely on the fact that the observed cause distribution is informative . As we will show , this renders them insufficient for nonlinear ICA , since the constraints they impose are satisfied by common counterexamples to identifiability . With this in mind , we introduce a new way to characterise or refine the ICM principle for unsupervised representation learning tasks such as nonlinear ICA . Motivating example . To build intuition , we turn to a famous example of ICA and BSS : the cocktail party problem , illustrated in Fig . 1 ( Left ) . Here , a number of conversations are happening in parallel , and the task is to recover the individual voices si from the recorded mixtures xi . The mixing or recording process f is primarily determined by the room acoustics and the locations at which microphones are placed . Moreover , each speaker influences the recording through their positioning in the room , and we may think of this influence as ∂f/∂si . Our independence postulate then amounts to stating that the speakers ’ positions are not fine-tuned to the room acoustics and microphone placement , or to each other , i.e. , the contributions ∂f/∂si should be independent ( in a non-statistical sense ) .1 Our approach . We formalise this notion of independence between the contributions ∂f/∂si of each source to the mixing process ( i.e. , the columns of the Jacobian matrix Jf of partial derivatives ) as an orthogonality condition , see Fig . 1 ( Right ) . Specifically , the absolute value of the determinant |Jf | , which describes the local change in infinitesimal volume induced by mixing the sources , should factorise or decompose as the product of the norms of its columns . This can be seen as a decoupling of the local influence of each partial derivative in the pushforward operation ( mixing function ) mapping the source distribution to the observed one , and gives rise to a novel framework which we term independent mechanism analysis ( IMA ) . IMA can be understood as a refinement of the ICM principle that applies the idea of independence of mechanisms at the level of the mixing function . Contributions . The structure and contributions of this paper can be summarised as follows : • we review well-known obstacles to identifiability of nonlinear ICA ( § 2.1 ) , as well as existing ICM criteria ( § 2.2 ) , and show that the latter do not sufficiently constrain nonlinear ICA ( § 3 ) ; • we propose a more suitable ICM criterion for unsupervised representation learning which gives rise to a new framework that we term independent mechanism analysis ( IMA ) ( § 4 ) ; we provide geometric and information-theoretic interpretations of IMA ( § 4.1 ) , introduce an IMA contrast function which is invariant to the inherent ambiguities of nonlinear ICA ( § 4.2 ) , and show that it rules out a large class of counterexamples and is consistent with existing identifiability results ( § 4.3 ) ; • we experimentally validate our theoretical claims and propose a regularised maximum-likelihood learning approach based on the IMA constrast which outperforms the unregularised baseline ( § 5 ) ; additionally , we introduce a method to learn nonlinear ICA solutions with triangular Jacobian and a metric to assess BSS which can be of independent interest for the nonlinear ICA community . 1For additional intuition and possible violations in the context of the cocktail party problem , see Appendix B.4 . 2 Background and preliminaries . Our work builds on and connects related literature from the fields of independent component analysis ( § 2.1 ) and causal inference ( § 2.2 ) . We review the most important concepts below . 2.1 Independent component analysis ( ICA ) . Assume the following data-generating process for independent component analysis ( ICA ) x = f ( s ) , ps ( s ) = ∏n i=1 psi ( si ) , ( 1 ) where the observed mixtures x ∈ Rn result from applying a smooth and invertible mixing function f : Rn → Rn to a set of unobserved , independent signals or sources s ∈ Rn with smooth , factorised density ps with connected support ( see illustration Fig . 2b ) . The goal of ICA is to learn an unmixing function g : Rn → Rn such that y = g ( x ) has independent components . Blind source separation ( BSS ) , on the other hand , aims to recover the true unmixing f−1 and thus the true sources s ( up to tolerable ambiguities , see below ) . Whether performing ICA corresponds to solving BSS is related to the concept of identifiability of the model class . Intuitively , identifiability is the desirable property that all models which give rise to the same mixture distribution should be “ equivalent ” up to certain ambiguities , formally defined as follows . Definition 2.1 ( ∼-identifiability ) . Let F be the set of all smooth , invertible functions f : Rn → Rn , and P be the set of all smooth , factorised densities ps with connected support on Rn . Let M ⊆ F×P be a subspace of models and let ∼ be an equivalence relation on M. Denote by f∗ps the push-forward density of ps via f . Then the generative process ( 1 ) is said to be ∼-identifiable on M if ∀ ( f , ps ) , ( f̃ , ps̃ ) ∈ M : f∗ps = f̃∗ps̃ =⇒ ( f , ps ) ∼ ( f̃ , ps̃ ) . ( 2 ) If the true model belongs to the model class M , then ∼-identifiability ensures that any model in M learnt from ( infinite amounts of ) data will be ∼-equivalent to the true one . An example is linear ICA which is identifiable up to permutation and rescaling of the sources on the subspace MLIN of pairs of ( i ) invertible matrices ( constraint on F ) and ( ii ) factorizing densities for which at most one si is Gaussian ( constraint on P ) [ 17 , 21 , 93 ] , see Appendix A for a more detailed account . In the nonlinear case ( i.e. , without constraints on F ) , identifiability is much more challenging . If si and sj are independent , then so are hi ( si ) and hj ( sj ) for any functions hi and hj . In addition to permutation-ambiguity , such element-wise h ( s ) = ( h1 ( s1 ) , ... , hn ( sn ) ) can therefore not be resolved either . We thus define the desired form of identifiability for nonlinear BSS as follows . Definition 2.2 ( ∼BSS ) . The equivalence relation ∼BSS on F × P defined as in Defn . 2.1 is given by ( f , ps ) ∼BSS ( f̃ , ps̃ ) ⇐⇒ ∃P , h s.t . ( f , ps ) = ( f̃ ◦ h−1 ◦P−1 , ( P ◦ h ) ∗ps̃ ) ( 3 ) where P is a permutation and h ( s ) = ( h1 ( s1 ) , ... , hn ( sn ) ) is an invertible , element-wise function . A fundamental obstacle—and a crucial difference to the linear problem—is that in the nonlinear case , different mixtures of si and sj can be independent , i.e. , solving ICA is not equivalent to solving BSS . A prominent example of this is given by the Darmois construction [ 20 , 39 ] . Definition 2.3 ( Darmois construction ) . The Darmois construction gD : Rn → ( 0 , 1 ) n is obtained by recursively applying the conditional cumulative distribution function ( CDF ) transform : gDi ( x1 : i ) : = P ( Xi ≤ xi|x1 : i−1 ) = ∫ xi −∞ p ( x ′ i|x1 : i−1 ) dx′i ( i = 1 , ... , n ) . ( 4 ) The resulting estimated sources yD = gD ( x ) are mutually-independent uniform r.v.s by construction , see Fig . 2a for an illustration . However , they need not be meaningfully related to the true sources s , and will , in general , still be a nonlinear mixing thereof [ 39 ] .2 Denoting the mixing function corresponding to ( 4 ) by fD = ( gD ) −1 and the uniform density on ( 0 , 1 ) n by pu , the Darmois solution ( fD , pu ) thus allows construction of counterexamples to ∼BSS-identifiability on F × P .3 Remark 2.4. gD has lower-triangular Jacobian , i.e. , ∂gDi /∂xj = 0 for i < j . Since the order of the xi is arbitrary , applying gD after a permutation yields a different Darmois solution . Moreover , ( 4 ) yields independent components yD even if the sources si were not independent to begin with.4 2Consider , e.g. , a mixing f with full Jacobian which yields a contradiction to Defn . 2.2 , due to Remark 2.4 . 3By applying a change of variables , we can see that the transformed variables in ( 4 ) are uniformly distributed in the open unit cube , thereby corresponding to independent components [ 69 , § 2.2 ] . 4This has broad implications for unsupervised learning , as it shows that , for i.i.d . observations , not only factorised priors , but any unconditional prior is insufficient for identifiability ( see , e.g. , [ 49 ] , Appendix D.2 ) . Another well-known obstacle to identifiability are measure-preserving automorphisms ( MPAs ) of the source distribution ps : these are functions a which map the source space to itself without affecting its distribution , i.e. , a∗ps = ps [ 39 ] . A particularly instructive class of MPAs is the following [ 49 , 58 ] . Definition 2.5 ( “ Rotated-Gaussian ” MPA ) . Let R ∈ O ( n ) be an orthogonal matrix , and denote by Fs ( s ) = ( Fs1 ( s1 ) , ... , Fsn ( sn ) ) and Φ ( z ) = ( Φ ( z1 ) , ... , Φ ( zn ) ) the element-wise CDFs of a smooth , factorised density ps and of a Gaussian , respectively . Then the “ rotated-Gaussian ” MPA aR ( ps ) is aR ( ps ) = F −1 s ◦Φ ◦R ◦Φ−1 ◦ Fs . ( 5 ) aR ( ps ) first maps to the ( rotationally invariant ) standard isotropic Gaussian ( via Φ−1 ◦ Fs ) , then applies a rotation , and finally maps back , without affecting the distribution of the estimated sources . Hence , if ( f̃ , ps̃ ) is a valid solution , then so is ( f̃ ◦ aR ( ps̃ ) , ps̃ ) for any R ∈ O ( n ) . Unless R is a permutation , this constitutes another common counterexample to ∼BSS-identifiability on F × P . Identifiability results for nonlinear ICA have recently been established for settings where an auxiliary variable u ( e.g. , environment index , time stamp , class label ) renders the sources conditionally independent [ 37 , 38 , 41 , 49 ] . The assumption on ps in ( 1 ) is replaced with ps|u ( s|u ) = ∏n i=1 psi|u ( si|u ) , thus restricting P in Defn . 2.1 . In most cases , u is assumed to be observed , though [ 30 ] is a notable exception . Similar results exist given access to a second noisy view x̃ [ 28 ] .

Summary: The paper proposes applying the theory of independent causal mechanisms (ICM), from causal discovery, to nonlinear ICA approaches to the blind source separation problem (BSS).  ICM is based on the idea that variables in a system are algorithmically independent or do not share information with each other. The paper shows, however, that ICM is not sufficient for BSS as spurious solutions exist which are not equivalent to the truth.  To address these difficulties, the paper proposes a new condition for identifiability which places an orthogonality condition on the columns of the Jacobian and provides information theoretic and geometric interpretations. The paper then shows theoretically that a large class of spurious solutions are not admitted under the IMA formulation.   Experiments on toy examples demonstrate that the model is identifiable under these spurious solutions.