Nonlinear ICA Using Volume-Preserving Transformations

Nonlinear ICA is a fundamental problem in machine learning , aiming to identify the underlying independent components ( sources ) from data which is assumed to be a nonlinear function ( mixing function ) of these sources . Recent works prove that if the sources have some particular structures ( e.g . temporal structure ) , they are theoretically identifiable even if the mixing function is arbitrary . However , in many cases such restrictions on the sources are difficult to satisfy or even verify , hence it inhibits the applicability of the proposed methods . Different from these works , we propose a general framework for nonlinear ICA , in which the mixing function is assumed to be a volume-preserving transformation , and meanwhile the conditions on the sources can be much looser . We provide an insightful proof of the identifiability of the proposed framework . We implement the framework by volume-preserving flow-based models , and verify our theory by experiments on artificial data and synthesized images . Moreover , results on real-world images indicate that our framework can disentangle interpretable features . 1 INTRODUCTION . Independent component analysis ( ICA ) is one of the most fundamental problems in machine learning . The earlier works concentrate on linear ICA ( Comon , 1994 ) , in which the observed data is assumed to be a linear and invertible transformation ( called “ mixing function ” ) of several independent components ( called “ sources ” ) , and the goal is to identify the independent components from data points . Recently , there are increasing interests in nonlinear ICA ( Hyvärinen & Pajunen , 1999 ) , in which the mixing function is generalized to be nonlinear . This nonlinear problem is crucial , as it is the theoretical foundation of many important tasks . For example , one task is disentanglement ( Locatello et al. , 2019 ; Sorrenson et al. , 2020 ; Locatello et al. , 2020 ) , which aims at learning some explanatory factors of variation from data ( Bengio et al. , 2013 ) and hence facilitates the interpretability of representation learning as well as its downstream tasks ( Peters et al. , 2017 ; Lake et al. , 2017 ) . Another example is controlling the generation process in generative models using semantic factors ( Karras et al. , 2019 ; Abdal et al. , 2021 ) . These tasks essentially require their latent variables to be identifiable in highly nonlinear latent models . Otherwise the latent variables can be mixing functions of explanatory / semantic factors , hence are probably not explanatory or semantic . The central problem in nonlinear ICA is the unidentifiability . Specifically , if the observed data points are independent and identically distributed ( i.i.d . in short ) , i.e . there is no temporal or similar structure in the data , then the sources are not identifiable essentially ( Hyvärinen & Pajunen , 1999 ) . This motivates many works to involve some structures to data for identifiability guarantee . To deal with the unidentifiability problem , most existing works of nonliear ICA involve structures to data by restricting the sources in some plausible ways . One popular solution is to assume the sources have some kinds of temporal structures ( Hyvarinen & Morioka , 2016 ; 2017 ) . These works almost have no restriction on the mixing functions , but limit the data in the form of time series . Recent works ( Hyvarinen et al. , 2019 ; Khemakhem et al. , 2020 ) extend the applicability of nonlinear ICA by involving auxiliary variables like labels of classes , and assume that the priors of sources are members of exponential family . However , their works require many classes in a dataset and each class should not be isolated in data space ( Sorrenson et al. , 2020 ) , which is difficult to be satisfied . To see this , note that most datasets ( e.g . CIFAR-10 ( Krizhevsky et al. , 2009 ) ) just have very few classes , and usually each class is isolated in data space . Another direction of involving structures to data is to restrict the mixing function . One way is to assume that the mixing function is post-nonlinear ( Taleb & Jutten , 1999 ) , i.e . each data variable is a nonlinear function of a linear combination of the sources . This is obviously overly restrictive , and hence is impractical . Some works provide a few heuristic settings on learning algorithms to facilitate identifiability ( Watters et al. , 2019 ; Sorrenson et al. , 2020 ) , e.g . Sorrenson et al . ( 2020 ) empirically report that volume preservation of flow-based models promotes disentanglement . In this work , we explore a balancing direction : involve some natural structures to the mixing function , and meanwhile loosen the restrictions on the sources . Specifically , motivated by the empirical evidences by ( Sorrenson et al. , 2020 ) , we assume the mixing function to be volume-preserving ( and a weaker condition is provided in Appendix B ) , and seek some natural restrictions on the sources for identifiability guarantee . With mild conditions , we establish novel and fundamental identifiability theorems , which mainly show that if there exist two distinct classes in the datasets , the true sources are identifiable . The proofs are essentially different to existing works , and provide some insights for nonlinear ICA : the main indeterminacy of a nonlinear ICA framework using volume-preserving mixing functions is the rotation of latent variables . Based on our identifiability theorems , we implement the proposed nonlinear ICA framework using a volume-preserving flow-based model ( Dinh et al. , 2014 ) , which requires merely two classes of data according to our theory . First , we empirically show that with the most essential conditions on sources ( two distinct classes with non-zero overlap ) and mixing functions , the sources can be well identified , which implies that there might exist some extensions of our theorems with weaker conditions . Then we compare our implementation with the state-of-the-art identifiable model iVAE ( Khemakhem et al. , 2020 ) on artificial and synthetic image datasets , and show that our framework remarkably better results in terms of mean correlation coefficient . More importantly , we empirically show that besides the given two classes , involving more classes can not improve identifiability . Moreover , experiments on MNIST ( LeCun et al. , 1998 ) and CelebA ( Liu et al. , 2015 ) indicate that the implementation is able to disentangle interpretable features using only two classes of data , which demonstrates the applicability of our theory . Our contributions are as follows : 1 ) We present a new solution to the unidentifiability problem of nonlinear ICA by using the general volume-preserving transformation , as widely used in flow-based models . Specifically , our theoretical results lessen the requirement by existing works on the sources by reducing the number of required classes to 2 , at the slight cost of adding some natural conditions on the mixing function . Accordingly , all the restrictions are moderate , which facilitates the applicability of nonlinear ICA . 2 ) We establish two novel identifiability theorems . Specifically , by using the friendly and general volume-preserving property , they guarantee the sources can be identified up to a point-wise nonlinearity and a point-wise linearity , respectively . The proofs suggest that the main indeterminacy of a nonlinear ICA framework with some natural conditions on mixing functions ( volume-preservation or a weaker condition , see Appendix B ) is simply the rotation of latent variables . The hope is to lay the foundation for a new path to identifiability . 3 ) We conduct experiment on both synthetic and real-world data to verify our theory , and point out that as long as the most essential conditions on sources ( two distinct classes with non-zero overlap ) and mixing functions are satisfied , our framework can well identify the true sources . This means that our theory can be further extended . Moreover , our framework remarkably excels the state-of-the-art nonlinear ICA method , which indicates that our framework is more powerful to identify the true sources . 2 BACKGROUNDS . In this section , we provide some backgrounds on nonlinear ICA theory . We first give a general definition of nonlinear ICA , and then explain its central problem , namely unidentifiability . Based on this , we discuss some recent proposed solutions , including their advantages and disadvantages . 2.1 DEFINITION AND THE KEY PROBLEM OF NONLINEAR ICA . Consider an observed random vector x ∈ Rd , we assume it is generated by an invertible nonlinear transformation f ( called mixing function ) using n independent latent variables s = ( s1 , · · · , sn ) ( called independent components or sources ) as x = f ( s ) , ( 1 ) where n ≤ d ( Khemakhem et al. , 2020 ) . In earlier ICA theory it is commonly assumed that n = d , i.e . the number of independent components should be the same as the number of observed variables . As it is commonly believed that n d for real world data ( Cayton , 2005 ; Narayanan & Mitter , 2010 ; Rifai et al. , 2011 ) , the definition above is much more practical . The goal of nonlinear ICA theory is to identify ( or recover ) the independent components si based on the observations of x using a estimating function g. Given the data distribution p ( x ) , if there exists a transformation g such that for arbitrary f , g ◦ f is a point-wise linear transformation , then the mixing model ( Eq . 1 ) is said to be identifiable . Unfortunately , if there are no more restrictions on the mixing model ( Eq . 1 ) , it is seriously unidentifiable ( Hyvärinen & Pajunen , 1999 ; Locatello et al. , 2019 ) . Specifically , even if the components of the estimated latent variables z ≡ g◦ f ( s ) are independent , zi can be the mixing of { si } ni=1 . In other words , independence of components does not guarantee identifiability of the mixing model . E.g . suppose the prior p ( s ) is a factorial multivariate Gaussian , take a point-wise scaling on s such that p ( s ) becomes an isotropic multivariate Gaussian , then take a rotation around the center , the independence of components is unchanged , but the obtained components are the mixing of the original sources . As arbitrary density functions can be transformed to a Gaussian ( Locatello et al. , 2019 ) , this example shows that for any prior , the independence of components is insufficient for identifiability . 2.2 EXISTING SOLUTIONS FOR UNIDENTIFIABILITY PROBLEM . To obtain identifiability , it is necessary to restrict the mixing model . There are two possible ways to involve restrictions : 1 ) restrict the sources s ; 2 ) restrict the mixing function f . In the first direction , one idea is to involve temporal structure to the sources s , i.e . assuming the sources are time series s ( t ) with time index t ( Harmeling et al. , 2003 ; Sprekeler et al. , 2014 ; Hyvarinen & Morioka , 2016 ; 2017 ) . This solution guarantees identifiability of the mixing model with few restrictions on the mixing function , but is limited in the setting of time series . A more general way is to involve an auxiliary variable u and assume that conditioned on them , the prior of sources s is a factorial member of exponential family ( Hyvarinen et al. , 2019 ; Khemakhem et al. , 2020 ) : p ( s|u ) = n∏ i=1 Qi ( si ) Zi ( u ) exp k∑ j=1 Ti , j ( si ) λi , j ( u ) , ( 2 ) where Ti , j , λi , j , Zi are the so-called sufficient statistics , coefficients and normalizing constant , respectively . Qi is the so-called base measure , and is simply set to be 1 in many cases . k is the order of the distribution , and distributions with higher k are more flexible . The auxiliary variable u is additionally observed , which can be time index , class label , etc . This solution seems general , but it requires nk + 1 distinct values of u to guarantee identifiability , which is impossible in most datasets . For example , if we assume that in MNIST ( LeCun et al. , 1998 ) , the prior of s is a factorial multivarite Gaussian ( k = 2 ) , and attempt to identify the sources using the 10 labels of the dataset based on the theory above , then we can identify 4 sources at most . However , the intrinsic dimensions is obviously far more than 4 ( Pope et al. , 2021 ) , and hence we can not identify the true sources . In the second direction , there are very few works to obtain identifiability by restricting the mixing function f in nonlinear ICA theory . One solution is to assume f to be post-nonlinear , i.e . { xj } dj=1 are nonlinear functions of the linear combination of sources , then the mixing model is identifiable ( Taleb & Jutten , 1999 ) . This condition is too restrictive ( Hyvarinen et al. , 2019 ) in practice .

Summary: [Sorrenson et al 2020] proposed to do nonlinear ICA via volume preserving mixing functions, when a variable u is observed that makes the latent variables conditionally independent. Via maximum likelihood optimization, the ICA model uses a volume-preserving encoder and then maximises a latent distribution, which is a factorized Gaussian conditional on the observed variable u.  The submitted paper does a theoretical identifiability analysis on that procedure. It assumes that the data is generated by an independent distribution, given observed u, then mixed by a volume-preserving mixing function. Furthermore, it assumes that both the mixer and the encoders are twice differentiable, there exist two observed u variables, u^1 and u^2 such that that the two resulting distributions of encoded latent variables have the same mean and that the set of ratios of the two standard deviations for each variable is distinct. If these assumptions hold, the paper proves that the original sources are identifiable up to a permutation and dimension-wise diffeomorphism.  The authors do three experiments: 2D mixture of Gaussians, triangle images and MNIST. For the first two, the authors quantitatively estimate the source identification and for all datasets, qualitative results on identification are given.