Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)

1 INTRODUCTION . Deep latent-variable models promise to unlock the key factors of variation within a dataset , opening a window to interpretation and granting the power to manipulate data in an intuitive fashion . The theory of identifiability in linear independent component analysis ( ICA ) ( Comon , 1994 ) tells us when this is possible , if we restrict the model to a linear transformation , but until recently there was no corresponding theory for the highly nonlinear models needed to manipulate complex data . This changed with the recent breakthrough work by Khemakhem et al . ( 2019 ) , which showed that under relatively mild conditions , it is possible to recover the joint data and latent space distribution , up to a simple transformation in the latent space . The key requirement is that the generating process is conditioned on a variable which is observed along with the data . This condition could be a class label , time index of a time series , or any other piece of information additional to the data . They interpret their theory as a nonlinear version of ICA . This work extends this theory in a direction relevant for application to real-world data . The existing theory assumes knowledge of the intrinsic problem dimension , but this is unrealistic for anything but artificially generated datasets . Here , we show that in the special case of Gaussian latent space distributions , the intrinsic problem dimension can be discovered . The important latent variables are organically separated from noise variables by the estimating model . Furthermore , the variables discovered correspond to the true generating latent variables , up to a trivial component-wise translation and scaling . Very similar results exist for other members of the exponential family with two parameters , such as the beta and gamma distributions . We introduce a variant of the RealNVP ( Dinh et al. , 2016 ) invertible neural network : the General Incompressible-flow Network ( GIN ) . The flow is called incompressible in reference to fluid dynamics , since it preserves volumes : the Jacobian determinant is simply unity . We emphasise its generality and increased expressive power in comparison to previous volume-preserving flows , such as NICE ( Dinh et al. , 2014 ) . As already noted in Khemakhem et al . ( 2019 ) , flow-based generative models are a natural fit for the theory of nonlinear ICA , as are the variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) used in that work . For us , major advantages of invertible architectures over VAEs are the ability to specify volume preservation and directly optimize the likelihood , and freedom from the requirement to specify the dimension of the model ’ s latent space . An INN always has a latent space of the same dimension as the data . In addition , the forward and backward models share parameters , saving the effort of learning separate models for each direction . In summary , our work makes the following contributions : • We extend the theory of nonlinear ICA to allow for unknown intrinsic problem dimension . Doing so , we find that this dimension can be discovered and a one-to-one correspondence between generating and estimated latent variables established . • We propose as an implementation an invertible neural network obtained by modifying the RealNVP architecture . We call our new architecture GIN : the General Incompressible-flow Network . • We demonstrate the viability of the model on artificial data and the EMNIST dataset . We extract 22 meaningful variables from EMNIST , encoding both global and local features . 2 RELATED WORK . The basic goals of nonlinear ICA stem from the original work on linear ICA . An influential formulation , as well as the first identifiability results , were given in Comon ( 1994 ) . These stated the conditions which allow the generating latent variables to be discovered , when the mixing function is a linear transformation . However , it was shown in Hyvärinen & Pajunen ( 1999 ) that this approach to identifiability does not extend to general nonlinear functions . The first identifiability results in nonlinear ICA came in Hyvärinen & Morioka ( 2016 ) and Hyvärinen & Morioka ( 2017 ) , applied to time series , and implemented via a discriminative model and semisupervised learning . A more general formulation , valid for other forms of data , was given in Hyvärinen et al . ( 2018 ) and the theory was extended to generative models in Khemakhem et al . ( 2019 ) , where experiments were implemented by a VAE . Many authors have addressed the general problem of disentanglement , and proposed models to learn disentangled features . Prominent among these is β-VAE ( Higgins et al. , 2017 ) and its variations ( e.g . Chen et al. , 2018 ) which augment the standard ELBO loss with tunable hyperparameters to encourage disentanglement . There are also attempts to modify the GAN framework ( Goodfellow et al. , 2014 ) such as InfoGAN ( Chen et al. , 2016 ) , which tries to maximize the mutual information between some dimensions of the latent space and the observed data . Many of these approaches are unsupervised . However , as pointed out and empirically demonstrated in Locatello et al . ( 2018 ) , unsupervised models without conditioning in the latent space are in general unidentifiable . Several unsupervised VAE models implement conditioning in the latent space by means of Gaussian mixtures ( Johnson et al. , 2016 ; Dilokthanakul et al. , 2016 ; Zhao et al. , 2019 ) . Our work differs mainly by ( i ) only considering supervised tasks , therefore being safely covered by the theory of Khemakhem et al . ( 2019 ) , and ( ii ) enforcing volume-preservation , not possible in a VAE . The invertible neural networks in this work build upon the NICE framework ( Dinh et al. , 2014 ) and its extension in RealNVP ( Dinh et al. , 2016 ) . A similar network design to ours is Ardizzone et al . ( 2019 ) . This is a conditioned INN based on RealNVP , however the conditioning information is applied as a parameter of the network . The authors find in experiments with MNIST that meaningful variables are present in their latent space , but are rotated such that they are not aligned with the axes of the space . In this work , the conditioning is only present as a parameter of the latent space distributions . As a result , it is covered by the theory of Khemakhem et al . ( 2019 ) and its extension here , which results in non-rotated , meaningful latent variables . 3 THEORY . 3.1 EXISTING THEORY OF NONLINEAR ICA . This section adapts theoretical results from Khemakhem et al . ( 2019 ) to the context of invertible neural networks . Suppose the existence of the following three random variables : a latent generating variable z ∈ Rn , a condition u ∈ Rm and a data point x ∈ Rd , where n ≤ d. If n < d , also suppose the existence of a noise variable ε ∈ Rd−n . The variables z and ε make up the generating latent space , and are not necessarily known , whereas u and x are observable and therefore always known . The distribution of z is a factorial member of the exponential family with k sufficient statistics , conditioned on u . In its most general form the distribution can be written as pz ( z|u ) = n∏ i=1 Qi ( zi ) Zi ( u ) exp k∑ j=1 Ti , j ( zi ) λi , j ( u ) ( 1 ) where the Ti , j are the sufficient statistics , the λi , j their coefficients and Zi the normalizing constant . Qi is called the base measure , which in many cases is simply 1 . The distribution of εmust not depend on u and its probability density function must be always finite . The variable x is the result of an arbitrarily complex , invertible , deterministic transformation from the generating latent space to the data space : x = f ( z , ε ) . This can alternatively be formulated as an injective , non-deterministic transformation from the lower-dimensional z-space to the higherdimensional x-space : x = fε ( z ) . In general , an observed dataset D will consist only of instances of x and u . The task of nonlinear ICA is to disentangle the data to recover the generating latent variables z , as well as the form of the function f and its inverse . We can try to achieve this with a sufficiently general , invertible function approximator g , which maps from a latent variable w ∈ Rd to the data space : x = g ( w ; θ ) . Here θ denotes the parameters of g. Note that the dimensions of the latent space and data space are the same due to the invertibility of g. We assume that w follows a conditionally independent exponential probability distribution , conditioned on the same condition u as z : pw ( w|u ) = d∏ i=1 Q′i ( wi ) Z ′i ( u ) exp k∑ j=1 T ′i , j ( wi ) λ ′ i , j ( u ) . ( 2 ) The coefficients of the sufficient statistics λ′i , j ( u ) are not restricted , which means that a given variable i of the estimated latent space is allowed to lose its dependence on u . If this occurs , we will consider this variable to encode noise , playing a role equivalent to ε in the generating latent space . In addition to this specification of the generative and estimating models , some conditions are necessary to ensure the latent variables can be recovered . The most important of these concerns the variability of λi , j ( u ) under u . However , as long as the λi , j are randomly and independently generated , and there are at least nk + 1 distinct conditions u , this condition is almost surely fulfilled . See Appendix A.1 for further details . In the limit of infinite data and perfect convergence , the estimating model will give the same conditional likelihood to all data points as the true generating model : pT , λ , f , ε ( x|u ) = pT′ , λ′ , g ( x|u ) . ( 3 ) If this is the case , the vector of sufficient statistics T from the generating latent space will be related to that of the estimated latent space by an affine transformation : T ( z ) = AT′ ( w ) + c , ( 4 ) where A is some constant , full-rank , nk × dk matrix and c ∈ Rnk some constant vector . The relationship holds for all values of z and w. The proof of this result can be found in Appendix A . This relationship is a generalization of that derived in Khemakhem et al . ( 2019 ) . In that version , they assume knowledge of n , the dimension of the generating latent space , and give their estimating model a latent space of the same dimension . In this context the matrix A defines a relationship between latent spaces of the same dimension , making it square and invertible . 3.2 NONLINEAR ICA WITH A GAUSSIAN LATENT SPACE . Given the relationship in equation ( 4 ) , we can ask whether any stronger results hold for particular special cases . We hope in particular to induce sparsity in the matrix A , so that the estimated latent space is related to the true one by as simple a transformation as possible . We show ( proof in Appendix B ) that when both the generating and estimated latent spaces follow a Gaussian distribution , the generating latent space variables are recovered up to a trivial translation and scaling . Furthermore , the dimension of the generating latent space is recovered . More precisely , each generating latent variable zi is related to exactly one estimated latent variable wj , for some j , as zi = aiwj + bi , for some constant ai and bi . Furthermore , each estimated latent variable wj is related to at most one zi . If the estimating latent space has higher dimension than the dimension of the generating latent space , some estimating latent variables are not related to any generating latent variables and so must encode only noise . This is the dimension discovery mechanism , since the estimated latent space organically splits into informative and non-informative parts , with the dimension of the informative part equal to the unknown intrinsic dimension of the generating latent space . Very similar results can be derived for all common continuous two-parameter members of the exponential family , including the gamma and beta distributions . See Appendix C .

Summary: This paper builds upon the recent theoretical framework on nonlinear ICA, put forward in recent work Khemakhem et al. (2019) that draw a lot of attention. The latter work provides an extension of the basic nonlinear ICA that is closely related to a VAE with a conditional factorized prior, essentially introducing side information (with assumed extra observables u) to resolve non-identifiability issues.