Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The manuscript introduces the cross-barcode between two point clouds, and uses it to construct the manifold topology divergence (MTD). It is proposed that this can be used for performance evaluation of GANs. A selection of experiments are run to show that the MTD is robust where other evaluation metrics fail. 
main_review: It is a shame that most of the cross-barcode is thrown away in favour of just the 1st order one in the construction of the MTD, I assume that this is because computation of the higher order ones are infeasible for large data sets, but it would be nice with some comparisons on smaller sets or at least some more commentary on it.

As a domain agnostic tool I think it has applications, and the construction of the cross-barcode is interesting and could be the object of further research.
limitations_and_societal_impact: It is unclear to me how exactly the cross-barcode is constructed. In section 2.3 it says that the simplicial complex is constructed from the metric space consisting of P and Q where the pairwise distances in Q are set to zero. As this space is not Hausdorff, or a metric space, this looks slightly odd at first, and could probably be explained better.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This work introduces a novel method based on relative persistent homology (because the term of persistent relative homology has been occupied before, referring to completely different concept) for measuring the similarity between manifolds, especially the sample point clouds of the manifolds. The distance uses the bar-code of persistent homology, and the homology groups are relative groups H_k(P,Q). The relative between the cross-bar codes and the Hausdorff distance between the point clouds   is proved in the appendix.  Based on the cross-barcode, the manifold topology divergence score (MTop-Divergence) is introduced and applied to assess generative models in various domains. The MTop-Div can detect mode collapsing , mode invention more effectively.  This is a first TDA-based practical method, and has many merits, scalability, domain agnostic, independence of pre-trained networks.
main_review: This work uses topological data analysis (TDA) method for deep learning, especially for measuring the distance between point clouds, therefore it is suitable for studying the similarities between distributions on manifolds. From theoretical point of view, the work is novel and inspiring, it shows the rigor and elegance of algebraic topology for data analysis. From practical point of view, the work has been tested on various generative models on different domains, the experiments show the effectiveness for discovering mode dropping, collapsing and mode invention. The way of combing powerful applications with solid theories should be promoted in deep learning.

The manuscript is well written. The introduction is highly motivated, and explains the key ideas clearly. The previous work part focuses on the literature in deep learning, and completely omits the large literature for persistent homology. It will be helpful to give some important references in TDA there. 

The theoretic section 2 represents the key theoretic results and the idea of the methodology. It is challenging to explain the whole picture to general audience without the knowledge of persistent homology. The authors  give many intuitive examples to explain the idea, and postpone the formulation to the later part, and give formal definitions and prove the claims in the appendix.  

The "spider web" metaphor is very intuitive and interesting. The proof for the position 1,  the upper bound of cross-barcode is given by the Hausdorff distance, is convincing. There are some theoretical considerations. 
1. All the distances among the points are using the Euclidean metric of the embedding space. Since the distributions are defined on the data manifolds, it is natural to consider the Riemannian metric of the manifolds. It is possible that the geodesic distance is very big but the Euclidean distance is very small. How to address this in the current framework ?

2. It is well known in geometric field that Hausdorff distance is the most preliminary distance to measure the shape similarities.  There are many cases that a sequence of shapes converge to a target shape in Hausdorff distance, but the curvature, area diverge. A more refined method may need to considered.

3. In order to measure the distance between distributions, the most common metric is the Wasserstein distance under Lp norm. The proposed method is sensitive to the topological changes,  therefore easily capture the multiple modes; but optimal transport map emphasizes more on distances. Some quantitative comparisons will be valuable.

The experiments are thorough and convincing, it might be helpful to compare with the methods based on optimal transportation. 

In summary, the work introduces rigorous persistent homology method for measuring the distance between distributions defined on manifolds embedded in Euclidean spaces. The work combines mathematical rigor with practical usage. The manuscript is well written, and work is inspiring. I advocate the acceptance. 

limitations_and_societal_impact: The  persistent homology is constructed using the Euclidean distance, for manifold data analysis, it is natural to consider geodesic distance.

The method focuses on the topology of the simplicial complex, most of the times, the regularity of the manifold also plays an important role. The Hausdorff distance is not sufficient to ensure the convergence of manifolds with Riemannian metrics.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a method to compare two data manifolds based on topology. A persistent diagram is constructed using both sets of samples (cross-barcode), and the proposed metric is computed as the average length of segments in the cross-barcode. The efficiency of the method is demonstrated in several experimental settings.
main_review: Originality:
First of all, I am not an expert in topological data analysis. However, it seems that the cross-barcode is new idea, which is shown to be potentially useful. Also, I think that the proposed idea is a sufficient contribution compared to the most related method (Geometry Score). As regards the related work, I believe that the paper "The Shape of Data: Intrinsic Distance for Data Distributions", A. Tsitsulin, et. al., ICLR 2020, should be cited and probably included in the experiments.


Quality:
As far as I can tell, the technical part of the paper seems to be correct. In addition, the claims are supported by the experiments. Maybe some toy data with nonlinear manifolds could be used in order to show the behavior in complicated scenarios. The authors do not mention any weakness of the methodology, and I could not think of such a case (see Clarity).


Clarity:
In my opinion, the clarity of the paper should be improved. In particular, it is quite difficult for a non-expert reader (like me) to understand the intuition behind the proposed approach. Especially, in Sec 2.2 - 2.6 there is a lot of text that is not easily accessible. I think that some images could help the reader to understand better the proposed idea. Also, I am aware that the used terminology is in accordance to the topological data analysis field, but instead, I would like the authors to provide a manuscript easier to understand from a non-expert reader. Anyways, the proposed method aims researches in machine learning that potentially need a tool to compare generative models, and not only researchers familiar with topology.


Significance:
I think that the proposed idea is potentially quite useful, since it provides a tool to assess the performance of generative modelling. This is, in general, rather useful for machine learning applications. From the experiments, it seems that the method is efficient and better from the related models. However, my main concern is that as a non-expert reader, I cannot understand in depth the proposed idea, so probably I cannot judge it objectively and find the potential weaknesses. In my opinion this limits the significance of the method.
limitations_and_societal_impact: The authors do not discuss potential negative societal impact, neither the limitations of their work.
review_title: null

Meta-review: Congratulations, the paper is accepted to NeurIPS 2021!
Please make an honest effort making this paper more accessible to general ML audience (non-experts in TDA). Clarify the barcode construction. Include persistent homology literature background. Elaborate limitations. Please include other clarifications, edits and additions as discussed in rebuttal and reviews. 
Decision: Accept (Poster)