Evaluating the Disentanglement of Deep Generative Models through Manifold Topology

1 INTRODUCTION . Learning disentangled representations is important for a variety of tasks , including adversarial robustness , generalization to novel tasks , and interpretability ( Stutz et al. , 2019 ; Alemi et al. , 2017 ; Ridgeway , 2016 ; Bengio et al. , 2013 ) . Recently , deep generative models have shown marked improvement in disentanglement across an increasing number of datasets and a variety of training objectives ( Chen et al. , 2016 ; Lin et al. , 2020 ; Higgins et al. , 2017 ; Kim and Mnih , 2018 ; Chen et al. , 2018b ; Burgess et al. , 2018 ; Karras et al. , 2019 ) . Nevertheless , quantifying the extent of this disentanglement has remained challenging and inconsistent . As a result , evaluation has often resorted to qualitative inspection for comparisons between models . Existing evaluation metrics are rigid : while some rely on training additional ad-hoc models that depend on the generative model , such as a classifier , regressor , or an encoder ( Eastwood and Williams , 2018 ; Kim and Mnih , 2018 ; Higgins et al. , 2017 ; Chen et al. , 2018b ; Glorot et al. , 2011 ; Grathwohl and Wilson , 2016 ; Karaletsos et al. , 2015 ; Duan et al. , 2020 ) , others are tuned for a particular dataset ( Karras et al. , 2019 ) . These both pose problems to the evaluation metric ’ s reliability , its relevance to different models and tasks , and consequently , its applicable scope . Specifically , reliance on training and tuning external models presents a tendency to be sensitive to additional hyperparameters and introduces partiality for models with particular training objectives , e.g . variational methods ( Chen et al. , 2018b ; Kim and Mnih , 2018 ; Higgins et al. , 2017 ; Burgess et al. , 2018 ) or adversarial methods with an encoder head on the discriminator ( Chen et al. , 2016 ; Lin et al. , 2020 ) . In fact , this reliance may provide an explanation for the frequent fluctuation in model rankings when new evaluation metrics are introduced ( Kim and Mnih , 2018 ; Lin et al. , 2020 ; Chen et al. , 2016 ) . Meanwhile , dataset-specific preprocessing , such as automatically removing background portions from generated portrait images ( Karras et al. , 2019 ) , generally limits the scope of the evaluation metric ’ s applicability because it depends on the preprocessing procedure and may otherwise be unreliable . To address this , we introduce an unsupervised disentanglement evaluation metric that can be applied across different model architectures and datasets without training an ad-hoc model for evaluation or introducing a dataset-specific preprocessing step . We achieve this by using topology , the mathematical discipline which differentiates between shapes based on gross features such as holes , loops , etc. , alongside density analysis of samples . The combination of these two ideas are the basis for functional persistence , which is one of the areas of application of persistent homology ( Cayton , 2005 ; Narayanan and Mitter , 2010 ; Goodfellow et al. , 2016 ) . In discussing topology , we walk a fine line between perfect mathematical rigor on the one hand and concreteness for a more general audience on the other . We hope we have found the right level for the machine learning community . Our method investigates the topology of these low-density regions ( holes ) by estimating homology , a topological invariant that characterizes the distribution of holes on a manifold . We first condition the manifold on each latent dimension and subsequently measure the persistent homology of these conditional submanifolds . By comparing persistent homology , we examine the degree to which conditional submanifolds continuously deform into each other . This provides a notion of topological similarity that is higher across submanifolds conditioned on disentangled dimensions than those conditioned on entangled ones . From this , we construct our evaluation metric using the aggregate topological similarity across data submanifolds conditioned on every latent dimension in the generative model . In this paper , we make several key contributions : • We present an unsupervised metric for evaluating disentanglement that only requires the generative model ( decoder ) and is dataset-agnostic . In order to achieve this , we propose measuring the topology of the learned data manifold with respect to its latent dimensions . Our approach measures the topological dissimilarity measure across latent dimensions , and permits the clustering of submanifolds based on topological similarity . • We also introduce a supervised variant that compares the generated topology to a real reference . • For both variants , we develop a topological similarity criterion based on Wasserstein distance , which defines a metric on barcode space in persistent homology ( Carlsson , 2019 ) . • Empirically , we perform an extensive set of experiments to demonstrate the applicability of our method across 10 models and three datasets using both the supervised and unsupervised variants . We find that our results are consistent with several existing methods . 2 BACKGROUND . Our method draws inspiration from the Manifold Hypothesis ( Cayton , 2005 ; Narayanan and Mitter , 2010 ; Goodfellow et al. , 2016 ) , which posits that there exists a low-dimensional manifoldMdata on which real data lie and pdata ( x ) is supported , and that generative models g : Z → X learn an approximation of that manifold Mmodel . As a result , the true data manifold Mdata contains high-density regions , separated by large expanses of low-density regions.Mmodel approximates the topology ofMdata , and superlevel sets of density withinMdata , through the learning process . A k-manifold is a space X , for example a subset of Rn for some n , which locally looks like an open set in Rk ( formally , for every point x ∈ X , there is a subset can be reparametrized to an open disc in Rk ) . A coordinate chart for the manifold X is an open subset U of Rk together with a continuous parametrization g : U → X of a subset of X . An atlas for X is a collection of coordinate charts that cover X . For example , any open hemisphere in a sphere is a coordinate chart , and the collection of all open hemispheres form an atlas . We say two manifolds are homeomorphic if there is a continuous map from X to Y that has a continuous inverse . Intuitively , two manifolds are homeomorphic if one can be viewed as a continuous reparametrization of the other . If we have a continuous map f from a manifold X to Rn , and are given two nearby points ~x and ~y in Rn , it is often useful to compare the subsets f−1 ( ~x ) and f−1 ( ~y ) , which are manifolds ( where the Jacobian matrix of f is maximal rank ) . They are frequently homeomorphic , and we will be using topological invariants that can distinguish between two non-homeomorphic manifolds . Among the easiest topological invariants to numerically estimate is homology ( Hatcher , 2005 ) , which characterizes the number of k-dimensional holes in a topological space such as a manifold . Intuitively , these holes correspond to low-density regions on the manifold . The field of persistent homology offers several methods for estimating the homology of a topological space from data samples ( Carlsson , 2019 ) . Recent work on Relative Living Times ( RLTs ) ( Khrulkov and Oseledets , 2018 ) has applied persistent homology to generative model data manifolds , also enabling direct comparison of generated data manifolds to real ones . For low-dimensional data ( and images ) , points correspond to their vector representation ( flattened matrices of pixels ) ; for high-dimensional data , particularly images , points correspond to vectorized embeddings from a pretrained VGG16 ( Simonyan and Zisserman , 2015 ) . To obtain RLTs , we first construct a family of simplicial complexes–graph-like structures–from data samples , each starting with a set of vertices representing the data points and no edges ( Figure 2 ) . These are witness complexes that characterize the topology of a set of points , a common method for statistically estimating topological invariants in persistent homology ( Carlsson , 2019 ; Lim et al. , 2020 ) . These simplicial complexes approximate the persistent homology of the data manifold by identifying k-dimensional holes present in the simplices at varying levels of proximity . Proximity is defined with a dissimilarity metric ( Euclidean distance ) between points . It is used to build a simplicial complex in which a collection of points spans a simplex if all points have proximity measure less than some threshold , as shown in Figure 2d . Varying the threshold gives persistent homology . As proximity increases , simplices are added , creating varying numbers of k-dimensional holes , which gives rise to persistence barcodes ( Carlsson , 2019 ; Zomorodian and Carlsson , 2005 ; Ghrist , 2008 ) . RLTs are vectorizations of these persistence barcodes , specifically the discrete distributions over the duration of each k-dimensional hole as it appears and disappears , or their lifetime relative to other holes . This is merely one method to ( partially ) vectorize persistence barcodes efficiently , and we leave it to future work to explore alternate methods ( Adcock et al. , 2013 ; Bubenik , 2015 ) . To measure the topological similarity between data samples representing two generative model manifolds , Khrulkov and Oseledets ( 2018 ) then take the Euclidean mean of several RLTs to produce a discrete probability distribution , called a Mean Relative Living Time ; they propose employing the Euclidean distance between two Mean Relative Living Times as the measure of topological similarity between two sets of data samples , known as the Geometry Score . Additional background is in Appendix B . 3 MANIFOLD INTERPRETATION OF DISENTANGLEMENT . We use prevailing definitions of disentanglement where a disentangled model has a factorized latent space corresponding bijectively to factors of variation ( Shu et al. , 2019 ; Higgins et al. , 2018 ; Duan et al. , 2020 ) . We frame our approach primarily on the definition in Shu et al . ( 2019 ) , with a more formal connection between their approach and our method in Appendix A . From the manifold perspective , disentanglement is an extrinsic property that is dependent on the generative model ’ s atlas . Consider a disentangled generative model g with manifoldM that assumes topology τ . We can define another generative model g′ with the same underlying manifoldM and τ , but it is entangled and has a different atlas . In fact , we can define several alternate disentangled and entangled atlases , provided there are multiple valid factorizations of the space . As a result , we need a method that can detect whether an atlas is disentangled . In this paper , we sliceM into submanifolds Usi=v ⊂M that are conditioned on a factor si at value v. These conditional submanifolds may have different homology from their supermanifoldM . If we observe samples from one factor , e.g . Xs1=v ∼ Us1=v at varying values of v , we find that all samples Xs1=v appear identical , except with respect to that single factor of variation si set to a different value of v. For a generative model , the correspondence between latent dimensions zj and factors si is not known upfront . As a result , we perform this procedure by conditioning on each latent dimension zj . Conditional submanifold topology . For two submanifolds to have the same topology , there needs to be a continuous and invertible mapping between them . First , assume that there exists an invertible mapping , or encoder e : X → Z , and a generative model g : Z → X , where both functions are continuous . Then , for a given z and x = g ( z ) , we can recover z by the composition z = e ( g ( z ) ) . We can also construct a simple linear mapping l : z→ z ’ , which adapts a factor ’ s value , such that z ’ = l ( e ( g ( z ) ) ) remains continuously deformable . This holds across factors , where the manifold is topologically symmetric with respect to different factors , i.e . its conditional submanifolds are homeomorphic . As an example , consider a disentangled generative model g ( z0 , z1 , z2 ) that traces a tri-axial ellipsoid x 2 z20 + y 2 z21 + z 2 z22 = 1 . If we condition the model on varying values of each factor , the resulting submanifolds are ellipses and have the same topology . Most complex manifolds have submanifolds that have non-homeomorphic factors of variation . For example , consider a generative model g ( z0 , z1 ) that traces a cylindrical shell with angle z0 , height z1 , and for simplicity , no thickness . The submanifolds conditioned on angle z0 form lines ( no holes ) , while the submanifolds conditioned on height z1 form circles ( a 1D hole ) . However , the topology remains the same for a given factor . A visualization of this principle on a cone is shown in Figure 3 . Taken together , this means that submanifolds within a factor ( intra-factor ) are homeomorphic , while submanifolds between factors ( extra-factor ) can be either homeomorphic or non-homeomorphic . Topological asymmetry . Because topologically asymmetric submanifolds are non-homeomorphic , using a single e that continuously deforms across submanifolds no longer holds under disentanglement . To address this , assume that for each factor j , there exists a continuous invertible encoder ej : X → Zj that exclusively encodes information on j from a generated sample . In the cylindrical shell example , this means continuously deforming across submanifolds conditioned on varying values of z0 using e0 ( deforming between lines ) and likewise for z1 using an e1 ( deforming between circles ) . Note that this formulation prevents continuous deformations between lines and circles . More generally , we can not continuously deform across submanifolds conditioned by arbitrary factors and expect the topology to be preserved . This procedure now amounts to performing latent traversals along an axis and observing the topology of the resulting submanifolds . In a disentangled model , the j-conditional Algorithm 1 : Procedure for producing W. RLTs on generated images : For generator g with latent prior P and N samples , Dz latent dimensions , and nd samples per latent dimension , returns a W. RLT for each dimension . RLT is the RLT procedure from Khrulkov and Oseledets ( 2018 ) . for i in 1 : N do Sample z ( i ) ∼ P { note : z ∈ RN×Dz } end for for latent dimension d in 1 : Dz do for k in 1 : nd do z′ ← copy ( z ) Set z′d ← k ∼ Pd Compute ez ← embedding ( g ( z′d ) ) { e.g . using VGG16 } Compute rlt [ d , k ] ← RLT ( ez , γ = 1/128 , L0 = 64 , n = 100 ) end for Compute WB [ d ] ←W.Barycenter ( rlt [ d , k ] ) end for return WB submanifolds exhibit the same topology by continuous composition of z = ej ( g ( z ) ) , using a linear mapping that only adapts factor j across the traversal , i.e . z′ = lj ( ej ( g ( z ) ) ) . In an entangled model by contrast , more than one factor—such as both the angle and height in the cylindrical shell example—exhibit variation along a dimension zj . Put another way , the topology on submanifolds conditioned on zj changes when multiple factors contribute to variation along this dimension . Concretely , following the cylindrical shell example , a dimension that encodes height and , after a certain height threshold , also begins to adapt the angle will result in a topology that changes to include a 2D hole . Consequently , submanifolds conditioned on the same latent dimension zj have the same topology in a disentangled model , yet different topology in an entangled one . Because we can not assume that the data manifold of a generative model is completely symmetric , we only consider submanifolds to be homeomorphic along the same factor in a disentangled model . By contrast , since these submanifolds are not homeomorphic in an entangled model , we can measure the similarity across submanifolds to evaluate a model ’ s disentanglement . Using this notion of intra-factor topological similarity , we may sufficiently measure disentanglement in most cases , but it does not shield us from the scenario where a generative model learns a single trivial factor along all dimensions , i.e . a factorization of one . If we assume that there exists assymmetries in the data manifold , then ensuring that the manifold exhibits topological dissimilarity between certain factors would disarm that case . We operationalize this by identifying homeomorphic clusters of factors , whereby each cluster has a distinct topology to ensure there is not a factorization of one . Within clusters , we measure topological similarity , but between clusters , we calculate topological dissimilarity . Consequently , topological similarity and dissimilarity form the basis of our evaluation metric . A more principled treatment of how manifold topology measures disentanglement is in Appendix A .

Summary: Introduces unsupervised disentangling metric that measures homeomorphic similarity between submanifolds conditioned on a given factor, and homeomorphic dissimilarity on submanifolds conditioned on different factors. The paper also includes a supervised variant which can directly assess topological similarity of submanifolds with label-spaces. The paper also introduces a novel variation of RLTs that  employs wasserstein distance instead of euclidean distance.