Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Through the analysis of heat flow, this paper proposes a more general graph neural diffusion scheme, where the authors induce a new class of PDE-based graph neural networks. 
The authors further explore the graph neural PDEs’ robustness properties from a theoretical point of view and show it is Lyapunov stable. 
The numerical experiments show certain robustness of the induced graph neural PDEs (including mean curvature flow-based and Beltrami flow-based methods) against topology perturbations.  

strengths_and_weaknesses: Strengths:
1. This paper first reviews a couple of math concepts and notions, including differential geometry, diffusion equations, gradient/divergence operators, etc. Those preliminaries are introduced systematically, and close to the following sections. Therefore, it helps the reader quickly get the main idea of each section instead of being stuck in the learning curve. 
2. The paper explores graph neural PDEs' robustness property and mathematically shows they are inherently Lyapunov stable and robust against topology perturbations.
3. The author proposed a model framework and several insights to design a new class of GNNs based on heat flows. The numerical experiments also validate the above analysis to some degree. Lastly, the related codes are also open-sourced, benefiting the community. 

Weaknesses:
1. One of my concerns comes from the title and some paragraphs (esp. Line 24-27 and Line 251-258). The paper seems to target the "robustness" topic, which covers a large area (including topology modification, node attribute perturbation, different attack settings, and so on) as the above sentence/paragraphs mentioned in the paper. However, the proposed theoretical analysis (i.e., propositions 1 and 2) and numerical experiments only focus on topology perturbation, which is just a portion of the big robustness topic. 
In other words, I am unclear the roles of Line 24-27 and 251-258. For example, Line 251-258 discuss four different attack settings, but I didn't see how the two methods (SPEIT and TDGIA) used in the experiments tie back to those four settings.
2. The experiment results are not well analyzed. They should connect to the theory in the previous section closer. For example, the reason for good or bad performance is not well illustrated as we can see the proposed method performance in Table 1 and Table S2 is not always outperforming others. Moreover, the propositions require a small perturb assumption. It will also be interesting to see more discussions about the impact of different perturb degrees on model performance.
3. Lastly, there are some other minor issues:
(1) It is better if the author can report the training and test speed, or do some algorithm/memory complexity analysis, especially for the section “PDE Solvers”. In this way, the paper can demonstrate better in its practical value.
(2) It would be better if some table can make consistency. For example, Table 4 and Table S2 don’t report the results under the TDGIA setting. And I find the Table 4 results are not match with any other previous tables, so what is the PDE solver used in Table 1?
(3) Since the author mentioned the over-smoothing problem and stacking multiple layers, it would be better if the author could provide more information about it.

questions: The following questions mainly related to the above weaknesses:
1. Are the proposed methods or theoretical analysis suitable for the node attribute to perturb scenario?
2. How do the experiments reflect the setting mentioned in Line 251-258?
3. How long does it need to run the proposed model?

It would be appreciated if the authors could also answer other questions or problems listed in the weakness.

limitations: Related to the above weakness and Questions, one of the limitations I can see is that the discussion in the paper may not be suitable for all the attack scenarios, such as node attribute perturbation.  
Another limitation is that the theory guard may not be valid once the perturbation is significant or bigger than a certain “threshold”. Those two potential limitations may need lots of effort and explorations in the future. 

To the best of my knowledge, there is no potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors address the important property of robustness to adversarial attacks in the context of graph neural PDEs. This is the first work to do so. The authors demonstrate that graph neural PDEs are more robust than other GNNs. The authors propose a theoretical reason for this robustness property. 
strengths_and_weaknesses: The paper is well written and well organised. 

The authors show that heat diffusion on a Riemannian manifold is robust to small perturbations of the manifold metric. The authors use this as a justification for why graph neural PDEs are robust to changes in the underlying graph. I think the authors should justify why small perturbations to the underlying graph give small perturbations to the graph metric. This seems to be assumed in lines 186-189. In the case of the shortest path distance, this can change quite radically with just a single edge removal/addition. As an example, consider a cycle graph. By removing an edge, the distance between nodes close to the removed edge goes from very small to very large. By increasing the size of the graph, you can make this change arbitrarily drastic. 

The experiments are extensive and convincing. However, the black box attacks trains a surrogate model which is closer to many of the benchmark models (message passing GNNs). This may give an unfair advantage to the method presented in the paper. 

“Table 2 shows that BeltramiGuard makes attacks’ efforts totally in vain and exceeds GNNGuard in Citeseer and PubMed datasets.”  The citeseer results are very similar (specially given the error on table 1 for GNNGuard). Can the authors check table 2 is correct for Citeseer, as the values are identical.

The datasets used are all very small, it may be more relevant to try a larger dataset although given the large combination of models/dataset this could be forgiven on computational grounds. Is it feasible to run experiments on a larger dataset in the review period?

Minor comments:
- Table 1 should be centred.
- The paper would be more self-contained and easier to read if the authors defined symmetric measurable coefficients used in Theorem 1.
- The codebase is quite complex with just a small README.md It would be useful to have instructions to reproduce the main figures/tables in the paper.
questions: How do lines 92-94 follow from [31, theorem 7.1]? Perhaps it should be theorem 7.13?

Would Figure 1 make sense/be easier to compare if you used t-SNE on one instance and then used the same position to plot the other graphs? (similarly, figure 3).

The codebase is huge. Has it been forked from an existing repo?
limitations: I address the main limitations in the strengths/weaknesses section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The authors provide theoretical and empirical insights into the robustness properties of graph neural PDE networks.
- They show that the heat kernel is stable under small perturbations of the Laplace operator and link this to the graph models by viewing a graph as a discretization of a Riemannian manifold
- They introduce a general graph PDE framework and instantiate three different diffusion processes to analyze their robustness (which are already introduced in related work)
strengths_and_weaknesses: Strengths:
- The theoretical findings on the robustness of the heat kernel are interesting and novel
- It is the first analysis of neural graph PDE model robustness
- As neural graph diffusion gains importance recently a robustness analysis is important to guide the development of these models
    
Weaknesses:
- The major weakness is that the analysis itself could be more in-depth, i.e. (i) I recommend to analyze failure modes of that robustness? (ii) Are there specific attacks that these models don’t defend well. (iii) I would suggest evaluation more threat models. Even though the chosen threat model represents attacks in the practice of social networks, for analysis it would be still beneficial to see exposure to other threat models and attack types. In particular, it seems unfair that no attack is introduced specifically targeting the new model.
- It is a bit unclear what is proposed by the authors and what is a new contribution on a first glance. It could be written clearer in that regard.
- Some unclear/open aspects regarding the experiments (see also questions below)
questions: - What of the robustness is attributed to the Lipschitzness of the network? As a Lipschitz constraint can improve robustness itself, an ablation study would be important.
- The Heat kernel e.g. does not perform better than the APPNP or GAT (for SPEIT attack) while the main robustness result is attributed to the stability of the heat kernel. Can you please explain this result.
- Why do you use so many layers? In general, on graphs, not many layers are used so the motivation for the Lipschitz constraint is unclear to me.
- Why do you need more layers for BLEND / GRAND than in their paper?
- For the inductive setting: are the edges to the unseen data for classification removed before training? Otherwise, due to the increased number of layers, there might be information flow in the training phase

limitations: -
review_title: null

Meta-review: metareview: This paper studies the robustness properties of graph neural partial differential equations and empirically demonstrates that graph neural PDEs are intrinsically more robust against topology perturbation compared to other graph neural networks. The reviewers found the experiments extensive and convincing.  The authors provided an extensive and detailed discussion in the rebuttal phase and answered the questions raised by the reviewers. Overall, the reviewers believe that the paper discusses an interesting and important topic, but also provided some comments for improvements, such as comparing the PGD attacks with other GNNs / defense methods. 
confidence: Certain
Decision: Accept