When Is Unsupervised Disentanglement Possible?

35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . With the recent success of deep learning approaches , there has been a resurgence of interest in the problem of recovering the independent factors that generate high dimensional data . As a motivating example , consider the cat1 manifold shown in figure 1a . Each image is of size 256× 256× 3 , but is generated as a smooth , nonlinear function of just two independent latent factors : the fur color and pose of the cat . Is it possible to recover these latent factors , given only samples of images ? This problem is closely related , yet different from the problem of learning a deep generative model from data . Given a training set { xi } , deep generative models find parameters θ such that xi = gθ ( zi ) where each latent variable zi is sampled from a base distribution , and the different components of z are independent . Tremendous progress in deep generative models has been achieved by methods including Generative Adversarial Networks ( GAN ) [ 9 , 1 , 16 ] , Variational Auto-Encoders ( VAE ) and their variants [ 18 ] and optimization-based approaches [ 4 ] . While deep generative models have achieved remarkable success in generating realistic images given a random input , the latent variables that they find are often very different from the desired “ independent factors ” [ 19 ] . Returning to our motivating example in figure 1a , a deep generative model may generate perfectly realistic images given random latent vectors z , where some of the coordinates of z have only a weak relationship to the true latent factors , and no single coordinate encodes a latent factor perfectly , as desired . Figure 1 illustrates this issue . In the top of column ( c ) we show the representation learned by a particular autoencoder [ 15 ] for the data on the left ( see figure 5 for additional baseline autoencoders ) . The embedding is shown as a function of the ground truth factors ( z1 : fur color , z2 : cat pose ) . The autoencoder learned two latent dimensions , y1 and y2 , and we plot y1 ( z1 , z2 ) , y2 ( z1 , z2 ) using the `` jet '' colormap that goes from blue to red . If each learned dimension encodes only one factor , then these plots should have contours that are parallel to the axes . For this particular autoencoder , however , this is not the case . Even though it gives very low reconstruction loss ( see supplementary ) , it can be seen that neither fur color nor pose are directly encoded by either of the learned factors . Such a representation is called “ entangled ” [ 3 ] , and there has been much recent work attempting to quantify the extent to which a learned representation is “ disentangled ” [ 8 , 11 , 5 ] . An intriguing result from the recent literature on disentanglement is that the problem is impossible without additional assumptions , even with infinite data [ 19 , 17 ] . This issue is well-understood in the context of linear functions of latent factors , i.e . Independent Component Analysis [ 12 ] . In the linear ICA setting , it is known that while the problem is non-identifiable when the latent factors are Gaussian , it can be solved if the latent factors have a sufficiently non-Gaussian distribution . The recent work of [ 19 , 17 ] , points out that in the nonlinear case , the identifiability problem is much more severe : even if the independent factors are non-Gaussian , there can be many non-linear transformations ẑ = q ( z ) that preserve the distribution of z . By defining ĝ ( ẑ ) = g ( q−1 ( ẑ ) ) , for g ( z ) = x , we can perfectly generate the data from latent variables that have the desired distribution yet are highly entangled ( see figure 2 for a graphical representation of the impossibility result ) . This suggests that disentanglement is impossible without additional strong assumptions on the architecture and the data even with infinite data , and indeed Locatello et al . demonstrated empirically that many deep autoencoder methods fail to learn a disentangled representation . In this paper we show the power of one very general assumption that can enable the learning of disentangled representations . The assumption of local isometry , i.e . that an change in the latent representation corresponds to an change in the observed variables . In the bottom of figure 1b , c we show a scatter plot of the distances d ( xi , xj ) and d ( yi , yj ) for randomly chosen pairs of nearby points . If the embedding satisfies local isometry , these points should lie on a straight line . As can be seen , the highly entangled autoencoder embedding does not satisfy local isometry : there is almost no correlation between the magnitude of change in the embedding space Y and the magnitude of change in the original input space X . Specifically , we prove that as the number of datapoints goes to infinity , a combination of two classical methods : Hessian Eigenmaps [ 7 ] and fastICA [ 12 ] , is guaranteed to find a disentangled representation when the manifold parameterization satisfies local isometry and the base distribution is sufficiently non-Gaussian . Figure 1b shows the result of our proposed method on the same data that the autoencoder was trained on . The contours in the plots are now axis-aligned , indicating that a perfectly disentangled representation was learned . As shown in the bottom plot , there is a strong correlation between the pairwise distances of points in the input space X and their corresponding representations in the embedding space Y . In the remaining sections , we first derive analytical conditions on manifold parameterizations that guarantee that disentanglement will be possible ; and discuss settings in which these conditions hold . We leverage recent advances in deep generative models to construct manifolds of highly realistic images for which the ground truth latent representation is known , and test whether different methods succeed in recovering a disentangled representation . For many different manifolds , we find that methods that explicitly optimize local isometry and non-Gaussianity consistently find the correct latent factors , while baseline deep autoencoders do not . Finally , we discuss ways to encourage deep autoencoders to find encodings that satisfy local isometry , and show that this helps them discover disentangled representations . Overall , our results suggest that in some realistic settings , disentanglement is possible without any domain-specific assumptions . 2 Local Isometry + non-Gaussianity make disentanglement possible . Our main theoretical result shows that a combination of two classical algorithms is guaranteed to find disentangled representations under two classical assumptions : local isometry and non-Gaussianity . We first define the assumptions and the algorithms and then state the result . Local Isometry : Intuitively , a mapping from a latent representation z to an observed vector x satisfies local isometry if an change in z corresponds to an change in x . Formally , a mapping g from z to x is said to satisfy local isometry if for all z , the Jacobian J ( z ) = ∂g∂z satisfies : J ( z ) TJ ( z ) = I As we discuss in section 3 , local isometry has been widely used in manifold learning algorithms and any manifold can be approximated to arbitrarily high accuracy using an isometric parameterization . Non-Gaussianity : Given a distribution over p ( z ) , we say that z is sufficiently non-Gaussian , if a linear ICA algorithm is guaranteed to recover z ( up to permutation ) from a linear mixing Az , given infinite training examples . As mentioned above , this assumption is crucial to enable disentanglement in linear generative models . The two algorithms that we will use are HLLE [ 7 ] and fastICA [ 13 ] . Hessian Eigenmaps ( HLLE ) belongs to a family of manifold learning techniques that first estimate a local parameterization around each datapoint and then align the local parameterizations by solving an eigenvector problem ( e.g . [ 25 , 30 , 28 ] ) . Specifically , it is based on the insight that the local isometry assumption means that local PCA will recover a parameterization that is locally isometric to the true parameterization . Given a dataset { xi } Mi=1 , an integer K that determines the size of the neighborhoods , and d the dimensionality of the manifold ( or equivalently the dimension of the latent code ) , the algorithm proceeds in the following steps : • For each point xi , identify its K nearest neighbors using Euclidean distance . • Perform a local principal components analysis ( PCA ) within each neighborhood and use the principal components in each neighborhood to define a global , M ×M , matrix H . • Calculate the d+1 eigenvectors ofH with smallest eigenvalues . Discard the trivial , constant eigenvector to obtain a d dimensional embedding Y . The embedding of xi , which is a vector of length d denoted yi is then computed from the ith index of the d eigenvectors . As shown in [ 7 ] , given infinite samples from a manifold for which the mapping from z to x satisfies local isometry , the HLLE algorithm is guaranteed to find a representation of x that is the same as z up to a linear transformation : yi = Azi + b for some matrix A . The fastICA algorithm [ 12 ] searches for projections of the data ( u = wTx ) which maximize the deviation between the expectation of a nonlinear function of the projection , G ( u ) , and the same expectation with a Gaussian distribution . It is guaranteed to be consistent “ for most reasonable choices of G , and distributions of the sources . In particular , if G ( u ) = u4 , the condition is fulfilled for any distribution of non-zero kurtosis . In that case , it can also be proven that there are no spurious optima ” [ 13 ] . In other words , if p ( z ) has nonzero kurtosis and we observe infinite samples from x = Az , then fastICA with G ( u ) = u4 is guaranteed to correctly recover A . Finally , we refer to HLLE+ICA as an algorithm that first runs HLLE and then runs ICA on the embedding found by HLLE . We are now ready to state our main result . Definition : A representation y ( x ) is said to be a disentangled representation of the true latent factors z if there exists a permutation π and a pointwise nonlinearity f such that y = π ( f ( z ) ) . Theorem 1 : Let { xi } be samples from a manifold x = g ( z ) . Assume that the support of p ( z ) is connected . If there exists an alternative representation of the manifold x = g2 ( s ) with s = Af ( z ) , for any matrix A and pointwise nonlinearity f , such that g2 satisfies local isometry and f ( z ) is sufficiently non-Gaussian ; then as the number of samples goes to infinity , HLLE+ICA will provably find a disentangled representation of the true latent factors z . Proof : Donoho and Grimes [ 7 ] showed that as the number of examples goes to infinity , the H matrix constructed in the HLLE algorithm converges to the ( continuous ) Hessian operator on the manifold whose d+ 1 eigenfunctions with zero eigenvalue are spanned by the latent factors . They also showed that if the support of p ( z ) is connected , then HLLE will recover the isometric embedding of the manifold , up to a linear transformation . In our setting , the results of Donoho and Grimes guarantee that HLLE will recover s up to a linear transformation . Since s is assumed to be a linear mixing of independent factors , the assumption of non-Gaussianity means that running fastICA on the output of HLLE will provably recover a disentangled representation . Figure 3 illustrates our theorem for a two dimensional manifold created by translating white squares on a black background . The location of each square is sampled from a uniform distribution . The mapping from z ( the horizontal and vertical position ) to x ( the image of the square ) has been shown to satisfy local isometry by [ 6 ] . Thus , HLLE ( left ) is guaranteed to find a representaton that is related to the true positions by a linear transformation . However , this is not a perfectly disentangled representation just yet . Nevertheless , since the distribution is uniform , not Gaussian , applying ICA on the result of HLLE ( middle ) indeed recovers a perfectly disentangled representation . Note that the definition of local isometry is sensitive to rescalings : if we parameterize the squares manifold in terms of horizontal and vertical position , then it will satisfy local isometry . However , we can also parameterize the same manifold using log ( x ) , exp ( y ) and such a parameterization does not satisfy local isometry . Nevertheless , Theorem 1 guarantees that we will still recover a disentangled representation , regardless of the rescaling of the true factors . In the supplementary material , we show that another classical manifold learning algorithm , Laplacian EigenMaps ( LEM ) , followed by fastICA , will also recover disentangled representations under certain conditions , but unlike HLLE+ICA , LEM+ICA is very sensitive to rescalings .

Summary: The authors propose a novel method to learn disentangled representations from data. Their approach uses some inductive biases on the transformation between the latent and observed spaces, as well as on the latent distribution. Namely, they assume the transformation satisfies the assumption of the local isometry, and that the latent factors are (non-jointly) non-gaussian. The disentangled representations are learned by first applying a spectral transform to the data, followed by a linear ICA unmixing.  Their method is backed by a theoretical analysis, as well as an empirical validation.