Manifold Topology Divergence: a Framework for Comparing Data Manifolds. 

1 Introduction . Geometric perspective in working with data distributions has been pervasive in machine learning [ 5 , 8 , 12 , 10 , 23 , 20 ] . Reconstruction of the data from observing only a subset of its points has made a significant step forward since the invention of Generative Adversarial Networks ( GANs ) [ 13 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Despite the exceptional success that deep generative models achieved , there still exists a longstanding challenge of good assessment of the generated samples quality and diversity [ 7 ] . For images , the Fréchet Inception Distance ( FID ) [ 16 ] is the most popular GAN evaluation measure . However , FID is limited only to 2D images since it relies on pre-trained on ImageNet “ Inception ” network . FID unrealistically approximates point clouds by Gaussians in embedding space ; also , FID is biased [ 6 ] . Surprisingly , FID can ’ t be applied to compare adversarial and non-adversarial generative models since it is overly pessimistic to the latter ones [ 28 ] . The evaluation of generative models is about comparing two point clouds : the true data cloud Pdata and the model ( generated ) cloud Qmodel . In view of the commonly assumed Manifold Hypothesis [ 5 , 12 ] , we develop a topology-based measure for comparing two manifolds : the true data manifold Mdata and the model manifoldMmodel , by analysing samples Pdata ⊂Mdata andQmodel ⊂Mmodel . Contribution . In this work , we make the following contributions : 1 . We introduce a new tool : Cross-Barcode ( P , Q ) . For a pair of point clouds P and Q , the Cross-Barcode ( P , Q ) records the differences in multiscale topology between two manifolds approximated by the point clouds ; 2 . We propose a new measure for comparing two data manifolds approximated by point clouds : Manifold Topology Divergence ( MTop-Div ) ; 3 . We apply the MTop-Div to evaluate performance of GANs in various domains : 2D images , 3D shapes , time-series . We show that the MTop-Div correlates well with domain-specific measures and can be used for model selection . Also it provides insights about evolution of generated data manifold during training ; 4 . We have compared the MTop-Div against 7 established evaluation methods : FID , discriminative score , MMD , JSD , 1-coverage , IMD and Geometry score and found that MTop-Div is able to capture subtle differences in data geometry ; 5 . We have essentially overcame the known TDA scalability issues and in particular have carried out the MTop-Div calculations on most recent datasets such as FFHQ , with dimensions D up to 107 . The source code is available at https : //github.com/IlyaTrofimov/MTopDiv . Related work . GANs try to recover the true data distribution via a min-max game where two players , typically represented by deep neural networks , called Discriminator and Generator , compete by optimizing the common objective . Training curves are not informative since generator and discriminator counter each other , thus , the loss values are often meaningless . Several other measures were introduced to estimate the quality of GANs . However , there is no consensus which of them best captures strengths and limitations of the models and can be used for the fair model selection . The likelihood in Parzen window works poorly in high-dimensional spaces and does not correlate with the visual quality of generated samples [ 30 ] . The Inception Score ( IS ) [ 29 ] measures both the discriminability and diversity of generated samples . It relies on the pre-trained Inception network and is limited to 2D images domain . While IS correlates with visual image quality , it ignores the true data distribution , doesn ’ t detect mode dropping and is sensitive to image resolution . The Fréchet Inception Distance ( FID ) [ 16 ] is a distance between two multivariate Gaussians . These Gaussians approximate the features of generated and true data extracted from the last hidden layer of the pretrained Inception network . Similar variant is KID [ 6 ] which computes MMD distance between two distributions of features . The work [ 14 ] proposed to use the Duality Gap , a notion from the game theory , as a domain agnostic measure for GAN evaluation . Another variant of a measure inspired by game theory was proposed in [ 27 ] . Conventional precision and recall measures can be also used [ 22 , 21 ] . Accurate calculation of precision and recall is limited to simple datasets from low-dimensional manifolds . The Geometry Score ( GScore ) [ 19 ] is the L2-distance between mean Relative Living Times ( RLT ) of topological features calculated for the model distribution and the true data distribution . The GScore is domain agnostic , does not involve auxiliary pretrained networks and is not limited to 2D images . However , GScore is not sensitive even to some simple transformations - like constant shift , dilation , or reflection ( see our Fig . 2 , 4 ) . The barcodes in GScore are calculated approximately , based only on the approximate witness complexes on 64 landmark points sampled from each distribution . That ’ s why the procedure is stochastic and should be repeated several thousand times for averaging . Thus , the calculation of GScore can be prohibitively long for large datasets . We also refer reader to the comprehensive survey [ 7 ] . 2 Cross-Barcode and Manifold Topology Divergence . 2.1 Multiscale simplicial approximation of manifolds . According to the well-known Manifold Hypothesis [ 12 ] the support of the data distribution Pdata is often concentrated on a low-dimensional manifold Mdata . We construct a framework for comparing numerically such distribution Pdata with a similar distribution Qmodel concentrated on a manifold Mmodel . Such distribution Qmodel is produced , for example , by a generative deep neural network in one of applications ’ scenarios . The immediate difficulty here is that the manifold Mdata is unknown and is described only through discrete sets of samples from the distribution Pdata . One standard approach to resolve this difficulty is to approximate the manifold Mdata by simplices with vertices given by the sampled points . The simplices approximating the manifold are picked based on proximity information given by the pairwise distances between sampled points [ 5 , 25 ] . The standard approach is to fix a threshold r > 0 and to take the simplices whose edges do not exceed the threshold r. The choice of threshold is essential here since if it is too small , then only the initial points , i.e. , separated from each other 0-dimensional simplices , are allowed . And if the threshold is too large , then all possible simplices with sampled points as vertices are included and their union is simply the big blob representing the convex hull of the sampled points . Instead of trying to guess the right value of the threshold , the standard recent approach is to study all thresholds at once . This can be achieved thanks to the mathematical tool , called barcode [ 2 , 10 ] , that quantifies the evolution of topological features over multiple scales . For each value of r the barcode describes the topology , namely the numbers of holes or voids of different dimensions , of the union of all simplices included up to the threshold r . 2.2 Measuring the differences in simplicial approximation of two manifolds . However , to estimate numerically the degree of similarity between the manifolds Mmodel , Mdata ⊂ RD , it is important not just to know the numbers of topological features across different scales for simplicial complexes approximating Mmodel , Mdata , but to be able to verify that the similar topological features are located at similar places and appear at similar scales . Our method measures the differences in the simplicial approximation of the two manifolds , represented by samples P and Q , by constructing sets of simplices , describing discrepancies between the two manifolds . To construct these sets of simplices we take the edges connecting P−points with Q−points , and also P−points between them , ordered by their length , and start adding these edges one by one , beginning from the smallest edge and gradually increasing the threshold , see Figure 1 . We add also the triangles and k−simplices at the threshold when all their edges have been added . It is assumed that all edges between Q−points were already in the initial set . We track in this process the births and the deaths of topological features , where the topological features are allowed here to have boundaries on any simplices formed by Q−points . The longer the lifespan of the topological feature across the change of threshold the bigger the described by this feature discrepancy between the two manifolds . Homology is a tool that permits to single out topological features that are similar , and to decompose any topological feature into a sum of basic topological features . More specifically , in our case , a k−cycle is a collection of k−simplices formed by P− and Q− points , such that their boundaries cancel each other , perhaps except for the boundary simplices formed only by Q−points . For example , a cycle of dimension k = 1 corresponds to a path connecting a pair of Q−points and consisting of edges passing through a set of P−points . A cycle which is a boundary of a set of ( k + 1 ) −simplices is considered trivial . Two cycles are topologically equivalent if they differ by a boundary , and by collection of simplices formed only by Q−points . A union of cycles is again a cycle . Each cycle can be represented by a vector in the vector space where each simplex corresponds to a generator . In practice , the vector space over { 0 , 1 } is used most often . The union of cycles corresponds to the sum of vectors . The homology vector space Hk is defined as the factor of the vector space of all k−cycles modulo the vector space of boundaries and cycles consisting of simplices formed only by Q−points . A set of vectors forming a basis in this factor-space corresponds to a set of basic topological features , so that any other topological feature is equivalent to a sum of some basic features . The homology are also defined for manifolds and for arbitrary topological spaces . This definition is technical and we have to omit it due to limited space , and to refer to e.g . [ 15 , 24 ] for details . The relevant properties for us are the following . For each topological space X the vector spaces Hk ( X ) , k = 0 , 1 . . . , are defined . The dimension of the vector space Hk equals to the number of independent k−dimensional topological features ( holes , voids etc ) . An inclusion Y ⊂ X induces a natural map Hk ( Y ) → Hk ( X ) In terms of homology , we would like to verify that not just the dimensions of homology groups H∗ ( Mmodel ) and H∗ ( Mdata ) are the same but that more importantly the natural maps : ϕr : H∗ ( Mmodel ∩Mdata ) → H∗ ( Mmodel ) ( 1 ) ϕp : H∗ ( Mmodel ∩Mdata ) → H∗ ( Mdata ) ( 2 ) induced by the embeddings are as close as possible to isomorphisms . The homology of a pair is the tool that measures how far such maps are from isomorphisms . Given a pair of topological spaces Y ⊂ X , the homology of a pair H∗ ( X , Y ) counts the number of independent topological features in X that can not be deformed to a topological feauture in Y plus independent topological features in Y that , after the embedding to X , become deformable to a point . An equivalent description , the homology of a pair H∗ ( X , Y ) counts the number of independent topological features in the factor-space X/Y , where all points of Y are contracted to a single point . The important fact for us is that the map , induced by the embedding , H∗ ( Y ) → H∗ ( X ) is an isomorphism if and only if the homology of the pair H∗ ( X , Y ) are trivial . Moreover the embedding of simple simplicial complexes Y ⊂ X is an equivalence in homotopy category , if and only if the homology of the pair H∗ ( X , Y ) are trivial [ 33 ] . To define the counterpart of this construction for manifolds represented by point clouds , we employ the following strategy . Firstly , we replace the pair ( Mmodel∩Mdata ) ⊂Mmodel by the equivalent pair Mmodel ⊂ ( Mdata ∪Mmodel ) with the same factor-space . Then , we represent ( Mdata ∪Mmodel ) by the union of point clouds P∪Q , where the point clouds P , Q are sampled from the distributionsPdata , Qmodel . Our principal claim here is that taking topologically the quotient of ( Mdata ∪Mmodel ) by Mmodel is equivalent in the framework of multiscale analysis of topological features to the following operation on the matrix mP∪Q of pairwise distances of the cloud P ∪Q : we set to zero all pairwise distances within the subcloud Q ⊂ ( P ∪Q ) .

Summary: This work introduces a novel method based on relative persistent homology (because the term of persistent relative homology has been occupied before, referring to completely different concept) for measuring the similarity between manifolds, especially the sample point clouds of the manifolds. The distance uses the bar-code of persistent homology, and the homology groups are relative groups H_k(P,Q). The relative between the cross-bar codes and the Hausdorff distance between the point clouds   is proved in the appendix.  Based on the cross-barcode, the manifold topology divergence score (MTop-Divergence) is introduced and applied to assess generative models in various domains. The MTop-Div can detect mode collapsing , mode invention more effectively.  This is a first TDA-based practical method, and has many merits, scalability, domain agnostic, independence of pre-trained networks.