When Is Unsupervised Disentanglement Possible?

35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . With the recent success of deep learning approaches , there has been a resurgence of interest in the problem of recovering the independent factors that generate high dimensional data . As a motivating example , consider the cat1 manifold shown in figure 1a . Each image is of size 256× 256× 3 , but is generated as a smooth , nonlinear function of just two independent latent factors : the fur color and pose of the cat . Is it possible to recover these latent factors , given only samples of images ? This problem is closely related , yet different from the problem of learning a deep generative model from data . Given a training set { xi } , deep generative models find parameters θ such that xi = gθ ( zi ) where each latent variable zi is sampled from a base distribution , and the different components of z are independent . Tremendous progress in deep generative models has been achieved by methods including Generative Adversarial Networks ( GAN ) [ 9 , 1 , 16 ] , Variational Auto-Encoders ( VAE ) and their variants [ 18 ] and optimization-based approaches [ 4 ] . While deep generative models have achieved remarkable success in generating realistic images given a random input , the latent variables that they find are often very different from the desired “ independent factors ” [ 19 ] . Returning to our motivating example in figure 1a , a deep generative model may generate perfectly realistic images given random latent vectors z , where some of the coordinates of z have only a weak relationship to the true latent factors , and no single coordinate encodes a latent factor perfectly , as desired . Figure 1 illustrates this issue . In the top of column ( c ) we show the representation learned by a particular autoencoder [ 15 ] for the data on the left ( see figure 5 for additional baseline autoencoders ) . The embedding is shown as a function of the ground truth factors ( z1 : fur color , z2 : cat pose ) . The autoencoder learned two latent dimensions , y1 and y2 , and we plot y1 ( z1 , z2 ) , y2 ( z1 , z2 ) using the `` jet '' colormap that goes from blue to red . If each learned dimension encodes only one factor , then these plots should have contours that are parallel to the axes . For this particular autoencoder , however , this is not the case . Even though it gives very low reconstruction loss ( see supplementary ) , it can be seen that neither fur color nor pose are directly encoded by either of the learned factors . Such a representation is called “ entangled ” [ 3 ] , and there has been much recent work attempting to quantify the extent to which a learned representation is “ disentangled ” [ 8 , 11 , 5 ] . An intriguing result from the recent literature on disentanglement is that the problem is impossible without additional assumptions , even with infinite data [ 19 , 17 ] . This issue is well-understood in the context of linear functions of latent factors , i.e . Independent Component Analysis [ 12 ] . In the linear ICA setting , it is known that while the problem is non-identifiable when the latent factors are Gaussian , it can be solved if the latent factors have a sufficiently non-Gaussian distribution . The recent work of [ 19 , 17 ] , points out that in the nonlinear case , the identifiability problem is much more severe : even if the independent factors are non-Gaussian , there can be many non-linear transformations ẑ = q ( z ) that preserve the distribution of z . By defining ĝ ( ẑ ) = g ( q−1 ( ẑ ) ) , for g ( z ) = x , we can perfectly generate the data from latent variables that have the desired distribution yet are highly entangled ( see figure 2 for a graphical representation of the impossibility result ) . This suggests that disentanglement is impossible without additional strong assumptions on the architecture and the data even with infinite data , and indeed Locatello et al . demonstrated empirically that many deep autoencoder methods fail to learn a disentangled representation . In this paper we show the power of one very general assumption that can enable the learning of disentangled representations . The assumption of local isometry , i.e . that an change in the latent representation corresponds to an change in the observed variables . In the bottom of figure 1b , c we show a scatter plot of the distances d ( xi , xj ) and d ( yi , yj ) for randomly chosen pairs of nearby points . If the embedding satisfies local isometry , these points should lie on a straight line . As can be seen , the highly entangled autoencoder embedding does not satisfy local isometry : there is almost no correlation between the magnitude of change in the embedding space Y and the magnitude of change in the original input space X . Specifically , we prove that as the number of datapoints goes to infinity , a combination of two classical methods : Hessian Eigenmaps [ 7 ] and fastICA [ 12 ] , is guaranteed to find a disentangled representation when the manifold parameterization satisfies local isometry and the base distribution is sufficiently non-Gaussian . Figure 1b shows the result of our proposed method on the same data that the autoencoder was trained on . The contours in the plots are now axis-aligned , indicating that a perfectly disentangled representation was learned . As shown in the bottom plot , there is a strong correlation between the pairwise distances of points in the input space X and their corresponding representations in the embedding space Y . In the remaining sections , we first derive analytical conditions on manifold parameterizations that guarantee that disentanglement will be possible ; and discuss settings in which these conditions hold . We leverage recent advances in deep generative models to construct manifolds of highly realistic images for which the ground truth latent representation is known , and test whether different methods succeed in recovering a disentangled representation . For many different manifolds , we find that methods that explicitly optimize local isometry and non-Gaussianity consistently find the correct latent factors , while baseline deep autoencoders do not . Finally , we discuss ways to encourage deep autoencoders to find encodings that satisfy local isometry , and show that this helps them discover disentangled representations . Overall , our results suggest that in some realistic settings , disentanglement is possible without any domain-specific assumptions . 2 Local Isometry + non-Gaussianity make disentanglement possible . Our main theoretical result shows that a combination of two classical algorithms is guaranteed to find disentangled representations under two classical assumptions : local isometry and non-Gaussianity . We first define the assumptions and the algorithms and then state the result . Local Isometry : Intuitively , a mapping from a latent representation z to an observed vector x satisfies local isometry if an change in z corresponds to an change in x . Formally , a mapping g from z to x is said to satisfy local isometry if for all z , the Jacobian J ( z ) = ∂g∂z satisfies : J ( z ) TJ ( z ) = I As we discuss in section 3 , local isometry has been widely used in manifold learning algorithms and any manifold can be approximated to arbitrarily high accuracy using an isometric parameterization . Non-Gaussianity : Given a distribution over p ( z ) , we say that z is sufficiently non-Gaussian , if a linear ICA algorithm is guaranteed to recover z ( up to permutation ) from a linear mixing Az , given infinite training examples . As mentioned above , this assumption is crucial to enable disentanglement in linear generative models . The two algorithms that we will use are HLLE [ 7 ] and fastICA [ 13 ] . Hessian Eigenmaps ( HLLE ) belongs to a family of manifold learning techniques that first estimate a local parameterization around each datapoint and then align the local parameterizations by solving an eigenvector problem ( e.g . [ 25 , 30 , 28 ] ) . Specifically , it is based on the insight that the local isometry assumption means that local PCA will recover a parameterization that is locally isometric to the true parameterization . Given a dataset { xi } Mi=1 , an integer K that determines the size of the neighborhoods , and d the dimensionality of the manifold ( or equivalently the dimension of the latent code ) , the algorithm proceeds in the following steps : • For each point xi , identify its K nearest neighbors using Euclidean distance . • Perform a local principal components analysis ( PCA ) within each neighborhood and use the principal components in each neighborhood to define a global , M ×M , matrix H . • Calculate the d+1 eigenvectors ofH with smallest eigenvalues . Discard the trivial , constant eigenvector to obtain a d dimensional embedding Y . The embedding of xi , which is a vector of length d denoted yi is then computed from the ith index of the d eigenvectors . As shown in [ 7 ] , given infinite samples from a manifold for which the mapping from z to x satisfies local isometry , the HLLE algorithm is guaranteed to find a representation of x that is the same as z up to a linear transformation : yi = Azi + b for some matrix A . The fastICA algorithm [ 12 ] searches for projections of the data ( u = wTx ) which maximize the deviation between the expectation of a nonlinear function of the projection , G ( u ) , and the same expectation with a Gaussian distribution . It is guaranteed to be consistent “ for most reasonable choices of G , and distributions of the sources . In particular , if G ( u ) = u4 , the condition is fulfilled for any distribution of non-zero kurtosis . In that case , it can also be proven that there are no spurious optima ” [ 13 ] . In other words , if p ( z ) has nonzero kurtosis and we observe infinite samples from x = Az , then fastICA with G ( u ) = u4 is guaranteed to correctly recover A . Finally , we refer to HLLE+ICA as an algorithm that first runs HLLE and then runs ICA on the embedding found by HLLE . We are now ready to state our main result . Definition : A representation y ( x ) is said to be a disentangled representation of the true latent factors z if there exists a permutation π and a pointwise nonlinearity f such that y = π ( f ( z ) ) . Theorem 1 : Let { xi } be samples from a manifold x = g ( z ) . Assume that the support of p ( z ) is connected . If there exists an alternative representation of the manifold x = g2 ( s ) with s = Af ( z ) , for any matrix A and pointwise nonlinearity f , such that g2 satisfies local isometry and f ( z ) is sufficiently non-Gaussian ; then as the number of samples goes to infinity , HLLE+ICA will provably find a disentangled representation of the true latent factors z . Proof : Donoho and Grimes [ 7 ] showed that as the number of examples goes to infinity , the H matrix constructed in the HLLE algorithm converges to the ( continuous ) Hessian operator on the manifold whose d+ 1 eigenfunctions with zero eigenvalue are spanned by the latent factors . They also showed that if the support of p ( z ) is connected , then HLLE will recover the isometric embedding of the manifold , up to a linear transformation . In our setting , the results of Donoho and Grimes guarantee that HLLE will recover s up to a linear transformation . Since s is assumed to be a linear mixing of independent factors , the assumption of non-Gaussianity means that running fastICA on the output of HLLE will provably recover a disentangled representation . Figure 3 illustrates our theorem for a two dimensional manifold created by translating white squares on a black background . The location of each square is sampled from a uniform distribution . The mapping from z ( the horizontal and vertical position ) to x ( the image of the square ) has been shown to satisfy local isometry by [ 6 ] . Thus , HLLE ( left ) is guaranteed to find a representaton that is related to the true positions by a linear transformation . However , this is not a perfectly disentangled representation just yet . Nevertheless , since the distribution is uniform , not Gaussian , applying ICA on the result of HLLE ( middle ) indeed recovers a perfectly disentangled representation . Note that the definition of local isometry is sensitive to rescalings : if we parameterize the squares manifold in terms of horizontal and vertical position , then it will satisfy local isometry . However , we can also parameterize the same manifold using log ( x ) , exp ( y ) and such a parameterization does not satisfy local isometry . Nevertheless , Theorem 1 guarantees that we will still recover a disentangled representation , regardless of the rescaling of the true factors . In the supplementary material , we show that another classical manifold learning algorithm , Laplacian EigenMaps ( LEM ) , followed by fastICA , will also recover disentangled representations under certain conditions , but unlike HLLE+ICA , LEM+ICA is very sensitive to rescalings .

Summary: This paper studies the disentanglement of samples generated by GANs. It argues that the local isometry and non-Gaussian property are the keys to disentanglement. Some intuitive experiments based on GAN latent space discovery are conducted to support this insight.