Evaluating the Disentanglement of Deep Generative Models through Manifold Topology

1 INTRODUCTION . Learning disentangled representations is important for a variety of tasks , including adversarial robustness , generalization to novel tasks , and interpretability ( Stutz et al. , 2019 ; Alemi et al. , 2017 ; Ridgeway , 2016 ; Bengio et al. , 2013 ) . Recently , deep generative models have shown marked improvement in disentanglement across an increasing number of datasets and a variety of training objectives ( Chen et al. , 2016 ; Lin et al. , 2020 ; Higgins et al. , 2017 ; Kim and Mnih , 2018 ; Chen et al. , 2018b ; Burgess et al. , 2018 ; Karras et al. , 2019 ) . Nevertheless , quantifying the extent of this disentanglement has remained challenging and inconsistent . As a result , evaluation has often resorted to qualitative inspection for comparisons between models . Existing evaluation metrics are rigid : while some rely on training additional ad-hoc models that depend on the generative model , such as a classifier , regressor , or an encoder ( Eastwood and Williams , 2018 ; Kim and Mnih , 2018 ; Higgins et al. , 2017 ; Chen et al. , 2018b ; Glorot et al. , 2011 ; Grathwohl and Wilson , 2016 ; Karaletsos et al. , 2015 ; Duan et al. , 2020 ) , others are tuned for a particular dataset ( Karras et al. , 2019 ) . These both pose problems to the evaluation metric ’ s reliability , its relevance to different models and tasks , and consequently , its applicable scope . Specifically , reliance on training and tuning external models presents a tendency to be sensitive to additional hyperparameters and introduces partiality for models with particular training objectives , e.g . variational methods ( Chen et al. , 2018b ; Kim and Mnih , 2018 ; Higgins et al. , 2017 ; Burgess et al. , 2018 ) or adversarial methods with an encoder head on the discriminator ( Chen et al. , 2016 ; Lin et al. , 2020 ) . In fact , this reliance may provide an explanation for the frequent fluctuation in model rankings when new evaluation metrics are introduced ( Kim and Mnih , 2018 ; Lin et al. , 2020 ; Chen et al. , 2016 ) . Meanwhile , dataset-specific preprocessing , such as automatically removing background portions from generated portrait images ( Karras et al. , 2019 ) , generally limits the scope of the evaluation metric ’ s applicability because it depends on the preprocessing procedure and may otherwise be unreliable . To address this , we introduce an unsupervised disentanglement evaluation metric that can be applied across different model architectures and datasets without training an ad-hoc model for evaluation or introducing a dataset-specific preprocessing step . We achieve this by using topology , the mathematical discipline which differentiates between shapes based on gross features such as holes , loops , etc. , alongside density analysis of samples . The combination of these two ideas are the basis for functional persistence , which is one of the areas of application of persistent homology ( Cayton , 2005 ; Narayanan and Mitter , 2010 ; Goodfellow et al. , 2016 ) . In discussing topology , we walk a fine line between perfect mathematical rigor on the one hand and concreteness for a more general audience on the other . We hope we have found the right level for the machine learning community . Our method investigates the topology of these low-density regions ( holes ) by estimating homology , a topological invariant that characterizes the distribution of holes on a manifold . We first condition the manifold on each latent dimension and subsequently measure the persistent homology of these conditional submanifolds . By comparing persistent homology , we examine the degree to which conditional submanifolds continuously deform into each other . This provides a notion of topological similarity that is higher across submanifolds conditioned on disentangled dimensions than those conditioned on entangled ones . From this , we construct our evaluation metric using the aggregate topological similarity across data submanifolds conditioned on every latent dimension in the generative model . In this paper , we make several key contributions : • We present an unsupervised metric for evaluating disentanglement that only requires the generative model ( decoder ) and is dataset-agnostic . In order to achieve this , we propose measuring the topology of the learned data manifold with respect to its latent dimensions . Our approach measures the topological dissimilarity measure across latent dimensions , and permits the clustering of submanifolds based on topological similarity . • We also introduce a supervised variant that compares the generated topology to a real reference . • For both variants , we develop a topological similarity criterion based on Wasserstein distance , which defines a metric on barcode space in persistent homology ( Carlsson , 2019 ) . • Empirically , we perform an extensive set of experiments to demonstrate the applicability of our method across 10 models and three datasets using both the supervised and unsupervised variants . We find that our results are consistent with several existing methods . 2 BACKGROUND . Our method draws inspiration from the Manifold Hypothesis ( Cayton , 2005 ; Narayanan and Mitter , 2010 ; Goodfellow et al. , 2016 ) , which posits that there exists a low-dimensional manifoldMdata on which real data lie and pdata ( x ) is supported , and that generative models g : Z → X learn an approximation of that manifold Mmodel . As a result , the true data manifold Mdata contains high-density regions , separated by large expanses of low-density regions.Mmodel approximates the topology ofMdata , and superlevel sets of density withinMdata , through the learning process . A k-manifold is a space X , for example a subset of Rn for some n , which locally looks like an open set in Rk ( formally , for every point x ∈ X , there is a subset can be reparametrized to an open disc in Rk ) . A coordinate chart for the manifold X is an open subset U of Rk together with a continuous parametrization g : U → X of a subset of X . An atlas for X is a collection of coordinate charts that cover X . For example , any open hemisphere in a sphere is a coordinate chart , and the collection of all open hemispheres form an atlas . We say two manifolds are homeomorphic if there is a continuous map from X to Y that has a continuous inverse . Intuitively , two manifolds are homeomorphic if one can be viewed as a continuous reparametrization of the other . If we have a continuous map f from a manifold X to Rn , and are given two nearby points ~x and ~y in Rn , it is often useful to compare the subsets f−1 ( ~x ) and f−1 ( ~y ) , which are manifolds ( where the Jacobian matrix of f is maximal rank ) . They are frequently homeomorphic , and we will be using topological invariants that can distinguish between two non-homeomorphic manifolds . Among the easiest topological invariants to numerically estimate is homology ( Hatcher , 2005 ) , which characterizes the number of k-dimensional holes in a topological space such as a manifold . Intuitively , these holes correspond to low-density regions on the manifold . The field of persistent homology offers several methods for estimating the homology of a topological space from data samples ( Carlsson , 2019 ) . Recent work on Relative Living Times ( RLTs ) ( Khrulkov and Oseledets , 2018 ) has applied persistent homology to generative model data manifolds , also enabling direct comparison of generated data manifolds to real ones . For low-dimensional data ( and images ) , points correspond to their vector representation ( flattened matrices of pixels ) ; for high-dimensional data , particularly images , points correspond to vectorized embeddings from a pretrained VGG16 ( Simonyan and Zisserman , 2015 ) . To obtain RLTs , we first construct a family of simplicial complexes–graph-like structures–from data samples , each starting with a set of vertices representing the data points and no edges ( Figure 2 ) . These are witness complexes that characterize the topology of a set of points , a common method for statistically estimating topological invariants in persistent homology ( Carlsson , 2019 ; Lim et al. , 2020 ) . These simplicial complexes approximate the persistent homology of the data manifold by identifying k-dimensional holes present in the simplices at varying levels of proximity . Proximity is defined with a dissimilarity metric ( Euclidean distance ) between points . It is used to build a simplicial complex in which a collection of points spans a simplex if all points have proximity measure less than some threshold , as shown in Figure 2d . Varying the threshold gives persistent homology . As proximity increases , simplices are added , creating varying numbers of k-dimensional holes , which gives rise to persistence barcodes ( Carlsson , 2019 ; Zomorodian and Carlsson , 2005 ; Ghrist , 2008 ) . RLTs are vectorizations of these persistence barcodes , specifically the discrete distributions over the duration of each k-dimensional hole as it appears and disappears , or their lifetime relative to other holes . This is merely one method to ( partially ) vectorize persistence barcodes efficiently , and we leave it to future work to explore alternate methods ( Adcock et al. , 2013 ; Bubenik , 2015 ) . To measure the topological similarity between data samples representing two generative model manifolds , Khrulkov and Oseledets ( 2018 ) then take the Euclidean mean of several RLTs to produce a discrete probability distribution , called a Mean Relative Living Time ; they propose employing the Euclidean distance between two Mean Relative Living Times as the measure of topological similarity between two sets of data samples , known as the Geometry Score . Additional background is in Appendix B . 3 MANIFOLD INTERPRETATION OF DISENTANGLEMENT . We use prevailing definitions of disentanglement where a disentangled model has a factorized latent space corresponding bijectively to factors of variation ( Shu et al. , 2019 ; Higgins et al. , 2018 ; Duan et al. , 2020 ) . We frame our approach primarily on the definition in Shu et al . ( 2019 ) , with a more formal connection between their approach and our method in Appendix A . From the manifold perspective , disentanglement is an extrinsic property that is dependent on the generative model ’ s atlas . Consider a disentangled generative model g with manifoldM that assumes topology τ . We can define another generative model g′ with the same underlying manifoldM and τ , but it is entangled and has a different atlas . In fact , we can define several alternate disentangled and entangled atlases , provided there are multiple valid factorizations of the space . As a result , we need a method that can detect whether an atlas is disentangled . In this paper , we sliceM into submanifolds Usi=v ⊂M that are conditioned on a factor si at value v. These conditional submanifolds may have different homology from their supermanifoldM . If we observe samples from one factor , e.g . Xs1=v ∼ Us1=v at varying values of v , we find that all samples Xs1=v appear identical , except with respect to that single factor of variation si set to a different value of v. For a generative model , the correspondence between latent dimensions zj and factors si is not known upfront . As a result , we perform this procedure by conditioning on each latent dimension zj . Conditional submanifold topology . For two submanifolds to have the same topology , there needs to be a continuous and invertible mapping between them . First , assume that there exists an invertible mapping , or encoder e : X → Z , and a generative model g : Z → X , where both functions are continuous . Then , for a given z and x = g ( z ) , we can recover z by the composition z = e ( g ( z ) ) . We can also construct a simple linear mapping l : z→ z ’ , which adapts a factor ’ s value , such that z ’ = l ( e ( g ( z ) ) ) remains continuously deformable . This holds across factors , where the manifold is topologically symmetric with respect to different factors , i.e . its conditional submanifolds are homeomorphic . As an example , consider a disentangled generative model g ( z0 , z1 , z2 ) that traces a tri-axial ellipsoid x 2 z20 + y 2 z21 + z 2 z22 = 1 . If we condition the model on varying values of each factor , the resulting submanifolds are ellipses and have the same topology . Most complex manifolds have submanifolds that have non-homeomorphic factors of variation . For example , consider a generative model g ( z0 , z1 ) that traces a cylindrical shell with angle z0 , height z1 , and for simplicity , no thickness . The submanifolds conditioned on angle z0 form lines ( no holes ) , while the submanifolds conditioned on height z1 form circles ( a 1D hole ) . However , the topology remains the same for a given factor . A visualization of this principle on a cone is shown in Figure 3 . Taken together , this means that submanifolds within a factor ( intra-factor ) are homeomorphic , while submanifolds between factors ( extra-factor ) can be either homeomorphic or non-homeomorphic . Topological asymmetry . Because topologically asymmetric submanifolds are non-homeomorphic , using a single e that continuously deforms across submanifolds no longer holds under disentanglement . To address this , assume that for each factor j , there exists a continuous invertible encoder ej : X → Zj that exclusively encodes information on j from a generated sample . In the cylindrical shell example , this means continuously deforming across submanifolds conditioned on varying values of z0 using e0 ( deforming between lines ) and likewise for z1 using an e1 ( deforming between circles ) . Note that this formulation prevents continuous deformations between lines and circles . More generally , we can not continuously deform across submanifolds conditioned by arbitrary factors and expect the topology to be preserved . This procedure now amounts to performing latent traversals along an axis and observing the topology of the resulting submanifolds . In a disentangled model , the j-conditional Algorithm 1 : Procedure for producing W. RLTs on generated images : For generator g with latent prior P and N samples , Dz latent dimensions , and nd samples per latent dimension , returns a W. RLT for each dimension . RLT is the RLT procedure from Khrulkov and Oseledets ( 2018 ) . for i in 1 : N do Sample z ( i ) ∼ P { note : z ∈ RN×Dz } end for for latent dimension d in 1 : Dz do for k in 1 : nd do z′ ← copy ( z ) Set z′d ← k ∼ Pd Compute ez ← embedding ( g ( z′d ) ) { e.g . using VGG16 } Compute rlt [ d , k ] ← RLT ( ez , γ = 1/128 , L0 = 64 , n = 100 ) end for Compute WB [ d ] ←W.Barycenter ( rlt [ d , k ] ) end for return WB submanifolds exhibit the same topology by continuous composition of z = ej ( g ( z ) ) , using a linear mapping that only adapts factor j across the traversal , i.e . z′ = lj ( ej ( g ( z ) ) ) . In an entangled model by contrast , more than one factor—such as both the angle and height in the cylindrical shell example—exhibit variation along a dimension zj . Put another way , the topology on submanifolds conditioned on zj changes when multiple factors contribute to variation along this dimension . Concretely , following the cylindrical shell example , a dimension that encodes height and , after a certain height threshold , also begins to adapt the angle will result in a topology that changes to include a 2D hole . Consequently , submanifolds conditioned on the same latent dimension zj have the same topology in a disentangled model , yet different topology in an entangled one . Because we can not assume that the data manifold of a generative model is completely symmetric , we only consider submanifolds to be homeomorphic along the same factor in a disentangled model . By contrast , since these submanifolds are not homeomorphic in an entangled model , we can measure the similarity across submanifolds to evaluate a model ’ s disentanglement . Using this notion of intra-factor topological similarity , we may sufficiently measure disentanglement in most cases , but it does not shield us from the scenario where a generative model learns a single trivial factor along all dimensions , i.e . a factorization of one . If we assume that there exists assymmetries in the data manifold , then ensuring that the manifold exhibits topological dissimilarity between certain factors would disarm that case . We operationalize this by identifying homeomorphic clusters of factors , whereby each cluster has a distinct topology to ensure there is not a factorization of one . Within clusters , we measure topological similarity , but between clusters , we calculate topological dissimilarity . Consequently , topological similarity and dissimilarity form the basis of our evaluation metric . A more principled treatment of how manifold topology measures disentanglement is in Appendix A .

Summary: The paper presents a disentanglement metric to measure the intrinsic properties of a generative model with respect to the factor of variation in the dataset. Toward this, the paper first assumes disentangled factors reside in different manifolds. These different manifolds are the sub-manifolds of some manifold M for a given disentangled generative model. The paper considers the fact that in an entangled model the sub-manifolds are not homeomorphic and thus similarity across submanifolds can be measured to evaluate a model’s disentanglement. As such, disentanglement is related to the topological similarity.  For measuring topological similarity, the paper then introduces Wasserstein Relative Living Times. The proposed metric is used to evaluate standard disentanglement methods and datasets demonstrating the importance.