Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Motivated by the connections between PDEs and conventional convolutional networks, this paper proposes the view of graph convolutional networks (GCNs) as discretizations of PDEs on manifolds. Using this view, the authors propose a a family of graph convolutional architectures that utilize discrete graph operators as their basis and argue that this approach can address existing issues with graph convolutional networks, such as over-smoothing. Moreover, the authors also argue that this approach can have a more even performance accorss a diverse range of tasks, instead of being specialized to only a specific type of problem.
main_review: The work is presented clearly and the paper is easy to follow and understand.

The authors present an analysis of the two types of dynamics (diffusion and hyperbolic) they propose as a basis for their graph convolutional architecture. They connect this analysis to the issue of over-smoothing and the different performance of architectures on different type of tasks. It is this analysis that forms the basis for the proposed architecture

The empirical evaluations presented in the paper are comprehensive, including diverse relevant baselines. The proposed method does perform well across a variety of tasks, either matching or surpassing existing methods.

Nevertheless, even though the choice is argued as being motivated by the characteristics of the task at hand, the authors do have the advantage point of proposing two methods and having the possibility of choosing between them for the convenience of the results. Since a mixture model was proposed and evaluated on a few tasks, it would be interesting to know if that single "unified" model is able to maintain the performance of the separate models in the cases where one significantly outperforms the other.

As a final remark, it would be interesting if the authors could expand on the comment on line 229 ("We argue that the success of our network stems from its capability of a priori extracting features from graphs, due to its utilization of discretized differential operators and PDE guided construction"). What specifically in the formulation allows it to better extract information a priori? A better discussion, even if not possible to completely ground it in theory of empirical results, would be important when making such a claim.

Overall, despite some of the comments above, the paper does present an interesting analysis and a good empirical evaluation with good results on important datasets when compared to standard existing methods. 

-------------------

Update post authors' response:

The reported experiments with the "unified" model strengthen the experimental evaluation, by showing that the results are not a consequence of arbitrary model-task combinations. 

Overall, I originally had only small issues with this paper, and these were addressed by the authors. My previous assessment already reflected the evaluation that "the paper does present an interesting analysis and a good empirical evaluation with good results on important datasets". The authors' response confirms my positive evaluation and I will thus maintain my evaluation and score.


limitations_and_societal_impact: The authors have adequately addressed the potential negative societal impact of their work.

As the authors state, they do address some limitations, in the form of the conditions for their theorems to apply. Further discussion of limitations (e.g. practical aspects) might have been of interest, but is not a significant detriment to the paper. The empirical demonstration in Section 4.2 that this argument is reflected in practice (when a mixture model is given a choice of which type of dynamics to use for different types of problems) is an interesting connection to the theoretical analysis.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed using discretized PDE operator on the graph to generalize current graph convolutions. With diffusion and hyperbolic equations, the proposedd method has the flexibility to fit the unseen graph signals.
main_review: This paper proposed discretized PDE to generalize graph convolution. 

### Motivation 
- The used PDE include non-linear diffusion and hyperbolic. However, the justification for using such two equations is not clear. (Why using these two rather than the other first/second-order PDE?)
- several related works are missing, like _Combining differentiable PDE solvers and graph neural networks for fluid flow prediction_, ICML 20 _Continuous Graph Neural Networks_, ICML 20

### Technical analysis
- Some related work generalized discretized graph convolution into continuous operators. This work generalized graph convolution as discretized operators. In theory, continuous operators should be more flexible than being discretized. So this methodology may need to be discussed. 
- If the motivation of using diffusion and hyperbolic is to adapt unseen data, especially conserving and/or smoothing the signals. Then many existing works regarding solving over-smoothing have similar logic, such as _DeepGCN_, ICCV 19.

### Experiment
- The node classification didn't show significant and consistent superiority in performance (many baselines take the lead) 
- The protein-protein interaction task showed that the proposed method is among the best, but it is still not the best.
- In Table 7, the proposed method seems also has an over-smoothing issue: performance decrease as layers go deeper
limitations_and_societal_impact: - Justify the motivation of the proposed method over the related works, including continuous PDE, ResNet for GCN, etc.
- Many graph data are suitable for diffusion since the signals are in low-frequency. It's better to use proper datasets to show the advantage of the proposed method, such as a mix of high and low-frequency.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new class of graph neural networks motivated by numerical methods for solving PDEs on manifolds. The idea is that there is a correspondence between specific GNNs (e.g., GNNs that learn the weight of edges such as GATs) and discretized PDEs on manifolds. Here, the discretization is spatially. For time (i.e., the depth of the GNN) one can use numerical solvers for PDEs similar to what was proposed by the original Neural ODE paper. The main motivation is that there are types of PDEs that do not suffer from the oversmoothing problem and, therefore, that their discretized versions might be a better fit for some problem domains where oversmoothing is a problem. 
main_review: First of all I want to say that the idea of drawing connections between PDEs/ODEs and GNNs as spatially discretized PDEs is really interesting. I also think that the experiments are comprehensive and while the improvements are marginal at best, they still validate the idea of relating PDEs on manifolds and GNNs as one that can lead to interesting new architectures. 

Unfortunately, I have some concerns when it comes to positioning the work in the context of existing methods. There are several papers that should be discussed here:  

[1] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Multipole graph neural operator for parametric partial differential equations. In NeurIPS, 2020b.

[2] Belbute-Peres, F. d. A., Economon, T., and Kolter, Z. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In ICML. PMLR, 2020

I would ask the authors to discuss the differences in their rebuttal. Especially the latter seems highly related but perhaps more focused on a particular dynamical system? Moreover, there is an ICML 2021 paper that showed up on arxiv about a month ago:

[3] https://arxiv.org/abs/2106.10934

While [3] appeared on arxiv *after* the NeurIPS submission deadline and therefore, is  not a reason to reject the current submission!, it would still be beneficial to write a few sentences outlining the differences and similarities. One difference I see is the way the edge weight matrix is modelled (in the submission denoted by W and modelled as the geometric mean of the degree of adjacent nodes, in [3] denoted by A and modelled as a GAT). There are also differences in the numerical solvers used. 
[3] also compares to several other *ODE* based graph neural networks which, at least, should be discussed in the related work section of the submission.

Again, I don’t think there is a problem with several papers coming out at roughly the same time with roughly the same ideas, as it often points to an important topic ripe for dissemination. But please make sure you end of discussing prior work (even if only motivated by ODEs) on combining differential equations in some form with GNNs. 

Besides the related work section, the one area where I see a lot of room for improvement is the presentation. For instance, the order in which you introduce parts of the paper seems odd and counterintuitive. For instance, you state two theorems on the third page. At this point, however, they seem out of context and, in their statement, use terms such as PDE-GCN that have not been introduced before. Either you state these theorems as standalone background theory about PDEs on manifolds, or you first introduce the concepts needed to understand their premises.

limitations_and_societal_impact: The authors claim that the limitations of the paper are sufficiently discussed since "Our theorems state that the activation function must be sign-preserving, bounded and monotone."

What I am wondering about: what about efficiency of the method? What about the dependence on defining W the way you did?
review_title: null

Meta-review: All ratings were (weak) "accept". 

Generally the work is novel and timely. A related paper appeared on arxiv in late June (GRAND), which one reviewer pointed to, but obviously this paper was submitted before that appeared, and doesn't factor into this paper's evaluation (as the reviewer appropriately recognized). To me this supports the relevance and importance of this work. My sense was that the reviewers' ratings were low given the feedback they provided, and the authors addressed the reviewers thoroughly and appropriately, causing one to raise their score.

If accepted, my suggestion to the authors is that they do a careful revision of the text to emphasize the broader scope and applicability of this work, in case some of the reviewers' 6 ratings were due to not clearly seeing connections to other areas.
Decision: Accept (Poster)