Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the author(s) identify special types of problems where persistent homology (PH)  performs well or even better than other state-of-the-art methods in data analysis. They found that PH can work better in three fundamental shape analysis tasks: the detection of the number of holes, curvature and convexity. They have carried out  different experiments to demonstrate the advantage of PH than existing models. 
strengths_and_weaknesses: The paper has discussed about the advantage of PH in several different areas. It is interesting to see a summerization of these advantages of PH.

PH-based machine learning models have already been widely used in different areas. In fact, the advantage, strength and limitations of PH models have been extensively discussed in various excellent application papers and reviewer papers. The paper lacks a detailed discussion of these models and the proof-of-concept type of experimental tests are not convincing at all! 
questions: To demonstrate the "effectiveness" of PH, realistic problems should be considered and a through comparison with the state-of-the-art models should be carried out. In fact, there are many excellent papers, in which these comparison is carried out together with a detailed descriptions of the advantage and limitations of PH models.  For instance, 

Z. X. Cang and Guo-Wei Wei, TopologyNet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions, PLOS Computational Biology, 13(7), e1005690 (2017).

Z. X. Cang, Lin Mu and Guo-Wei Wei, Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening, PLOS Computational Biology,  14(1), e100592 (2018).

Duc Duy Nguyen, Zixuan Cang, Kedi Wu, Menglun Wang, Yin Cao  and Guo-Wei Wei, Mathematical deep learning for pose and binding affinity prediction and ranking in D3R Grand Challenges, Journal of Computer Aided Molecular Design, 33, 71-82   (2019).

Duc Nguyen, Zixuan Cang, and Guo-Wei Wei, A review of mathematical representations of biomolecular data, Physical Chemistry Chemical Physics, 22, 4343-4367 (2020).

Menglun Wang, Z.  X. Cang, and Guo-Wei Wei,   A topology-based network tree for the prediction of protein-protein binding free energy changes following mutation, Nature Machine Intelligence, 2, 116-123 (2020).

Zhenyu Meng and Kelin Xia, "Persistent spectral–based machine learning (PerSpect ML) for protein-ligand binding affinity prediction." Science Advances, 7 (19), eabc5329 (2021)

Zhao, Qi, Ze Ye, Chao Chen, and Yusu Wang. "Persistence enhanced graph neural network." In International Conference on Artificial Intelligence and Statistics, pp. 2896-2906. PMLR, 2020.

Wong, Chi-Chong, and Chi-Man Vong. "Persistent homology based graph convolution network for fine-grained 3D shape segmentation." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7098-7107. 2021.
limitations: As stated above, to demonstrate the "effectiveness" of PH, realistic problems should be considered and a through comparison with the state-of-the-art models should be carried out. In general, it should be easy to show that PH models can characterize certain curvature and convexity related information. However, there are gigantic amount of models that can have much better accuracy in the characterization of curvature and convexity. Only when the PH models are fully compared with state-of-the-art models, the authors can argue about the "Effectiveness of Persistent Homology". Otherwise, it is just the potential application!
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this article, the authors propose to investigate the data science tasks where persistent homology performs actually better than standard methods. They focus on three different problems: holes, curvature and convexity detection, and they show that, for each of them, persistent homology leads to better accuracies than neural nets.
strengths_and_weaknesses: I think the question of identifying the tasks where computational topology improves on existing methods is definitely important, making the paper well motivated. Moreover,  the writing of the paper is very good and clear.

However, the significance of the paper is limited by two major weaknesses.

First, there is no theoretical statements or results about the topological methods used to solve the tasks. This makes the claims of the paper harder to believe, since there is no way of estimating the generalization capability of topological methods on data sets that are different from those used in the paper.

Second, the experiments of the paper are only done on synthetic data sets in two dimensions. When there is no theory, it is necessary to benchmark many different data sets, from both synthetic and real data, in order for the claims to be convincing, but the proposed experiments are much too synthetic and simple.
questions: ---What are the variances of the accuracies in the figures? Since some experiments involve adding random noise, and since training and test sets are drawn randomly, accuracies should always be computed on several runs, and their means and variances across the runs should be reported.

---Why is PH suited for curvature detection? One can easily get the intuition why PH is good for hole and convexity, but it is less clear concerning curvature. In particular, why is 0-dimensional PH efficient, but not 1-dimensional PH? Why do the short bars matter more than the long ones? It would be nice if the authors provided some more intuition for this task.

---The paper should be more explicit about the parameters: in the hole detection experiment, what are the DTM parameters? In the convexity detection experiment, what is the resolution of the image computed from the point cloud? 

---Instead of computing an image from the point cloud in the convexity experiment, could one use scalar field analysis instead (https://link.springer.com/article/10.1007/s00454-011-9360-x)?

---What is the influence of the point cloud dimension? Is there any curse of dimensionality? In other words, is there any reason to believe that the proposed topological methods would still be efficient?
limitations: I think the authors properly identified the limitations of their experiments.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the effectiveness of PH for detection of number of holes, curvature and convexity using point clouds samples from shapes. The analyses are performed on various simple datasets in comparison with baselines providing promising initial results.
strengths_and_weaknesses: PH is an important method used for topological data analysis. The paper introduces a nice analysis of the employment of PH for several tasks on point clouds. The paper is well written, and analyses are clearly presented. In several benchmark, PH outperforms baseline with promising results for the future work.

The main weakness of the paper is the datasets. The paper can have more impact if the analyses are extended to point clouds of more complicated shapes, especially shapes of real-world objects. Although the current results are interesting, their generalization to more complicated datasets is not clear.  As mentioned in the limitation, a more detailed comparison with state-of-the-art would improve the paper.
questions: Training curves show that NNs overfit data easily. Could you please repeat these experiments using networks endowed with normalization and regularization methods such as ResNets?
limitations: Limitations were briefly addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper mainly discusses what can be detected by persistent homology (PH) features by introducing three shape analysis tasks. Experiments on these tasks show that PH can effectively detect these shapes even with limited computational resources and limited training data.
strengths_and_weaknesses: In general, the paper does add some interesting and new discussions, but it seems that some discussions have been covered in previous works.

Strengths:

The paper is generally well-written, and not hard to follow, except for some minor mistakes. E.g., “0- and 1-dim PH” in Figure 1 should be replaced by “0- and 1-dim PI”; In Line 282, it seems that the longest interval should be topological noise, while the second-longest one is useful.

The discussion in the paper is interesting. Recently, there are many works integrating PH into various downstream deep learning tasks and achieving state-of-the-art results. However, many works lack insight on why PH can boom performance on these tasks. Therefore, researching what can be detected by PH quite interests me. The discussion on long/short persistence is also interesting.

The discussion on using PH to detect convexity is new.

Weakness:

First, detecting the number of holes with PH is not new. Actually, we can directly compute the betti number (the number of holes) of a given point cloud data. (E.g., in [56], researchers compute the betti number with several preprocessing steps.) Then what is the superiority of the proposed method: (to infer/learn the number of holes with PH features)? 

Second, detecting curvature is not new either. Actually, a similar experiment that uses PH to detect curvature is done in [17]. In fact, they also provide theoretical guarantees and experiments on unsupervised learning.

Also, there are some comparably minor problems. (1) building simplicial complexes and computing PH is quite computational costly, therefore better add comparison on times in the experiment part; (2) adding persistent homology on downstream tasks that require topological information such as curvature/convexity/betti number can better support the experiment part of the paper. But considering the limited rebuttal time, this can be future work.

--**Post rebuttal**--

Generally speaking, the topic of the paper is worth researching, and some of the discussions are interesting. Although most of the experiments are based on synthetic datasets, it should be admitted that very few real-world graphs totally rely on the number of holes/convexity/curvature to predict certain attributes, e.g., their label.  Therefore I would like to raise my score, but certain improvements should be considered:

(1) Add PH to deep learning models and evaluate whether it can boost performance on real-world tasks. For example, over-squashing is a curvature-related attribute [1], therefore it can be evaluated whether adding PH in GNN can solve the problem.

(2) Some works combining PH with deep learning models are missing, e.g., [2], [3]. 

[1] Topping J, Di Giovanni F, Chamberlain B P, et al. Understanding over-squashing and bottlenecks on graphs via curvature. ICLR 2021.

[2] Hu X, Li F, Samaras D, et al. Topology-preserving deep image segmentation. Neurips 2019.

[3] Yan Z, Ma T, Gao L, et al. Link prediction with persistent homology: An interactive view. ICML 2021.

questions: See "Weakness".
limitations: The limitations are adequately addressed.
review_title: null

Meta-review: metareview: This paper focuses on evaluating the learning power of topological features (i.e., persistent homology (PH)) on specific tasks: predicting topological and geometrical properties. On synthetic point cloud data, topological feature outperforms various baseline learning methods when predicting: topology (Betti numbers), curvature and convexity. 

This paper received both positive and negative reviews. Both sides have valid points. The major concern is that the experiments are mostly restricted to synthetic settings rather than exploiting the real world problems where PH has shown its learning power -- graph, point cloud data and molecular data. Meanwhile, reviewers do consider the paper valuable as it provides more insights into what can PH effectively predicts and how long/short persistence barcodes should be valued as signal. 

After carefully reading the paper, reviews and rebuttals/discussions, We recommend the paper to be accepted. The paper does make a valid contribution by focusing on synthetic data and basic topological and geometric properties. This provides clear message of what PH can predict. It can help the community by providing good insights/guidelines when one tries to incorporate PH into deep learning models. 

We do agree with reviewers that the title and the main claim need to be adjusted to be more specific/concrete to reflect the actual contribution. Also the authors should incorporate reviewers' suggestions and strengthen the experiments and references. 

Some further suggestions: 
In the main paper, there should be a list of all baseline methods, even though their details can be provided in the supplemental. Also please comment on whether the baseline methods (e.g., PointNet) have been trained with effective data augmentation (rotation, scaling, etc). 
confidence: Certain
Decision: Accept