Manifold Topology Divergence: a Framework for Comparing Data Manifolds. 

1 Introduction . Geometric perspective in working with data distributions has been pervasive in machine learning [ 5 , 8 , 12 , 10 , 23 , 20 ] . Reconstruction of the data from observing only a subset of its points has made a significant step forward since the invention of Generative Adversarial Networks ( GANs ) [ 13 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Despite the exceptional success that deep generative models achieved , there still exists a longstanding challenge of good assessment of the generated samples quality and diversity [ 7 ] . For images , the Fréchet Inception Distance ( FID ) [ 16 ] is the most popular GAN evaluation measure . However , FID is limited only to 2D images since it relies on pre-trained on ImageNet “ Inception ” network . FID unrealistically approximates point clouds by Gaussians in embedding space ; also , FID is biased [ 6 ] . Surprisingly , FID can ’ t be applied to compare adversarial and non-adversarial generative models since it is overly pessimistic to the latter ones [ 28 ] . The evaluation of generative models is about comparing two point clouds : the true data cloud Pdata and the model ( generated ) cloud Qmodel . In view of the commonly assumed Manifold Hypothesis [ 5 , 12 ] , we develop a topology-based measure for comparing two manifolds : the true data manifold Mdata and the model manifoldMmodel , by analysing samples Pdata ⊂Mdata andQmodel ⊂Mmodel . Contribution . In this work , we make the following contributions : 1 . We introduce a new tool : Cross-Barcode ( P , Q ) . For a pair of point clouds P and Q , the Cross-Barcode ( P , Q ) records the differences in multiscale topology between two manifolds approximated by the point clouds ; 2 . We propose a new measure for comparing two data manifolds approximated by point clouds : Manifold Topology Divergence ( MTop-Div ) ; 3 . We apply the MTop-Div to evaluate performance of GANs in various domains : 2D images , 3D shapes , time-series . We show that the MTop-Div correlates well with domain-specific measures and can be used for model selection . Also it provides insights about evolution of generated data manifold during training ; 4 . We have compared the MTop-Div against 7 established evaluation methods : FID , discriminative score , MMD , JSD , 1-coverage , IMD and Geometry score and found that MTop-Div is able to capture subtle differences in data geometry ; 5 . We have essentially overcame the known TDA scalability issues and in particular have carried out the MTop-Div calculations on most recent datasets such as FFHQ , with dimensions D up to 107 . The source code is available at https : //github.com/IlyaTrofimov/MTopDiv . Related work . GANs try to recover the true data distribution via a min-max game where two players , typically represented by deep neural networks , called Discriminator and Generator , compete by optimizing the common objective . Training curves are not informative since generator and discriminator counter each other , thus , the loss values are often meaningless . Several other measures were introduced to estimate the quality of GANs . However , there is no consensus which of them best captures strengths and limitations of the models and can be used for the fair model selection . The likelihood in Parzen window works poorly in high-dimensional spaces and does not correlate with the visual quality of generated samples [ 30 ] . The Inception Score ( IS ) [ 29 ] measures both the discriminability and diversity of generated samples . It relies on the pre-trained Inception network and is limited to 2D images domain . While IS correlates with visual image quality , it ignores the true data distribution , doesn ’ t detect mode dropping and is sensitive to image resolution . The Fréchet Inception Distance ( FID ) [ 16 ] is a distance between two multivariate Gaussians . These Gaussians approximate the features of generated and true data extracted from the last hidden layer of the pretrained Inception network . Similar variant is KID [ 6 ] which computes MMD distance between two distributions of features . The work [ 14 ] proposed to use the Duality Gap , a notion from the game theory , as a domain agnostic measure for GAN evaluation . Another variant of a measure inspired by game theory was proposed in [ 27 ] . Conventional precision and recall measures can be also used [ 22 , 21 ] . Accurate calculation of precision and recall is limited to simple datasets from low-dimensional manifolds . The Geometry Score ( GScore ) [ 19 ] is the L2-distance between mean Relative Living Times ( RLT ) of topological features calculated for the model distribution and the true data distribution . The GScore is domain agnostic , does not involve auxiliary pretrained networks and is not limited to 2D images . However , GScore is not sensitive even to some simple transformations - like constant shift , dilation , or reflection ( see our Fig . 2 , 4 ) . The barcodes in GScore are calculated approximately , based only on the approximate witness complexes on 64 landmark points sampled from each distribution . That ’ s why the procedure is stochastic and should be repeated several thousand times for averaging . Thus , the calculation of GScore can be prohibitively long for large datasets . We also refer reader to the comprehensive survey [ 7 ] . 2 Cross-Barcode and Manifold Topology Divergence . 2.1 Multiscale simplicial approximation of manifolds . According to the well-known Manifold Hypothesis [ 12 ] the support of the data distribution Pdata is often concentrated on a low-dimensional manifold Mdata . We construct a framework for comparing numerically such distribution Pdata with a similar distribution Qmodel concentrated on a manifold Mmodel . Such distribution Qmodel is produced , for example , by a generative deep neural network in one of applications ’ scenarios . The immediate difficulty here is that the manifold Mdata is unknown and is described only through discrete sets of samples from the distribution Pdata . One standard approach to resolve this difficulty is to approximate the manifold Mdata by simplices with vertices given by the sampled points . The simplices approximating the manifold are picked based on proximity information given by the pairwise distances between sampled points [ 5 , 25 ] . The standard approach is to fix a threshold r > 0 and to take the simplices whose edges do not exceed the threshold r. The choice of threshold is essential here since if it is too small , then only the initial points , i.e. , separated from each other 0-dimensional simplices , are allowed . And if the threshold is too large , then all possible simplices with sampled points as vertices are included and their union is simply the big blob representing the convex hull of the sampled points . Instead of trying to guess the right value of the threshold , the standard recent approach is to study all thresholds at once . This can be achieved thanks to the mathematical tool , called barcode [ 2 , 10 ] , that quantifies the evolution of topological features over multiple scales . For each value of r the barcode describes the topology , namely the numbers of holes or voids of different dimensions , of the union of all simplices included up to the threshold r . 2.2 Measuring the differences in simplicial approximation of two manifolds . However , to estimate numerically the degree of similarity between the manifolds Mmodel , Mdata ⊂ RD , it is important not just to know the numbers of topological features across different scales for simplicial complexes approximating Mmodel , Mdata , but to be able to verify that the similar topological features are located at similar places and appear at similar scales . Our method measures the differences in the simplicial approximation of the two manifolds , represented by samples P and Q , by constructing sets of simplices , describing discrepancies between the two manifolds . To construct these sets of simplices we take the edges connecting P−points with Q−points , and also P−points between them , ordered by their length , and start adding these edges one by one , beginning from the smallest edge and gradually increasing the threshold , see Figure 1 . We add also the triangles and k−simplices at the threshold when all their edges have been added . It is assumed that all edges between Q−points were already in the initial set . We track in this process the births and the deaths of topological features , where the topological features are allowed here to have boundaries on any simplices formed by Q−points . The longer the lifespan of the topological feature across the change of threshold the bigger the described by this feature discrepancy between the two manifolds . Homology is a tool that permits to single out topological features that are similar , and to decompose any topological feature into a sum of basic topological features . More specifically , in our case , a k−cycle is a collection of k−simplices formed by P− and Q− points , such that their boundaries cancel each other , perhaps except for the boundary simplices formed only by Q−points . For example , a cycle of dimension k = 1 corresponds to a path connecting a pair of Q−points and consisting of edges passing through a set of P−points . A cycle which is a boundary of a set of ( k + 1 ) −simplices is considered trivial . Two cycles are topologically equivalent if they differ by a boundary , and by collection of simplices formed only by Q−points . A union of cycles is again a cycle . Each cycle can be represented by a vector in the vector space where each simplex corresponds to a generator . In practice , the vector space over { 0 , 1 } is used most often . The union of cycles corresponds to the sum of vectors . The homology vector space Hk is defined as the factor of the vector space of all k−cycles modulo the vector space of boundaries and cycles consisting of simplices formed only by Q−points . A set of vectors forming a basis in this factor-space corresponds to a set of basic topological features , so that any other topological feature is equivalent to a sum of some basic features . The homology are also defined for manifolds and for arbitrary topological spaces . This definition is technical and we have to omit it due to limited space , and to refer to e.g . [ 15 , 24 ] for details . The relevant properties for us are the following . For each topological space X the vector spaces Hk ( X ) , k = 0 , 1 . . . , are defined . The dimension of the vector space Hk equals to the number of independent k−dimensional topological features ( holes , voids etc ) . An inclusion Y ⊂ X induces a natural map Hk ( Y ) → Hk ( X ) In terms of homology , we would like to verify that not just the dimensions of homology groups H∗ ( Mmodel ) and H∗ ( Mdata ) are the same but that more importantly the natural maps : ϕr : H∗ ( Mmodel ∩Mdata ) → H∗ ( Mmodel ) ( 1 ) ϕp : H∗ ( Mmodel ∩Mdata ) → H∗ ( Mdata ) ( 2 ) induced by the embeddings are as close as possible to isomorphisms . The homology of a pair is the tool that measures how far such maps are from isomorphisms . Given a pair of topological spaces Y ⊂ X , the homology of a pair H∗ ( X , Y ) counts the number of independent topological features in X that can not be deformed to a topological feauture in Y plus independent topological features in Y that , after the embedding to X , become deformable to a point . An equivalent description , the homology of a pair H∗ ( X , Y ) counts the number of independent topological features in the factor-space X/Y , where all points of Y are contracted to a single point . The important fact for us is that the map , induced by the embedding , H∗ ( Y ) → H∗ ( X ) is an isomorphism if and only if the homology of the pair H∗ ( X , Y ) are trivial . Moreover the embedding of simple simplicial complexes Y ⊂ X is an equivalence in homotopy category , if and only if the homology of the pair H∗ ( X , Y ) are trivial [ 33 ] . To define the counterpart of this construction for manifolds represented by point clouds , we employ the following strategy . Firstly , we replace the pair ( Mmodel∩Mdata ) ⊂Mmodel by the equivalent pair Mmodel ⊂ ( Mdata ∪Mmodel ) with the same factor-space . Then , we represent ( Mdata ∪Mmodel ) by the union of point clouds P∪Q , where the point clouds P , Q are sampled from the distributionsPdata , Qmodel . Our principal claim here is that taking topologically the quotient of ( Mdata ∪Mmodel ) by Mmodel is equivalent in the framework of multiscale analysis of topological features to the following operation on the matrix mP∪Q of pairwise distances of the cloud P ∪Q : we set to zero all pairwise distances within the subcloud Q ⊂ ( P ∪Q ) .

Summary: The manuscript introduces the cross-barcode between two point clouds, and uses it to construct the manifold topology divergence (MTD). It is proposed that this can be used for performance evaluation of GANs. A selection of experiments are run to show that the MTD is robust where other evaluation metrics fail.