Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an approach of viewing the (residual) GNNs / graph ODEs through the point of an energy and corresponding gradient flow. In other words, the residual GNNs are viewed as derived from taking the gradient of a parameterised energy function. One of the ‘take-home’ points emphasised throughout the paper is the aspect that one should parameterise the energy and not the evolution equation of the GNNs.

strength_and_weaknesses: Strength: The topic is definitely timely and interesting, and the general approach relevant. The paper is generally well-written. There are multiple concepts of interest, as using Dirchlect energy to quantify HFD/LFD nature of node features, emphasising the importance of residual connections, the attractive-repulsive multi-particle perspective. 

Weaknesses: The motivation, particularly with respect to the core idea of energy is unclear. Out of oversquashing, over-smoothing, and heterophily, the authors tell us that their approach is suited to arrive at principled GNNs, guided by the perspective of energy functional/gradient flow. However, the relevance of the energy function/ gradient flow to this is not entirely clear - in fact most of their analysis, proofs, and lemmas, continue to be based on the evolution function, and on the Dirichlect energy, that is used to analyse high-frequency and low-frequency asymptotic behaviour of the latent feature evolution.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and of good quality. Aspects of the paper are novel and contribute to the field of building physics-inspired GNNs. The code is not available for anonymity reasons but I believe that it will be provided if the paper is accepted. There are some implementation details that are not clear such as the choice of the ODEsolver and the hyperparameters but I believe that these issues can be easily fixed. 
summary_of_the_review: Below are a few comments that could hopefully improve the quality of the paper. They are mainly related to the intuition, motivation, and validation of the approach.

Motivation of the energy view:

- The energy functional itself is arrived in a sort of reverse-engineered manner by starting from the evolution equation (5), without the non-linearity $\sigma$. Equation (5) itself seems to be presented without explaining why such a specific form alone would be necessary, but that does not affect the analysis seriously. Once the energy functional is defined, we are shown how the mixing matrix and its spectra regularizes the frequency spectra asymptotically — that such an energy helps mediate both smoothness (through positive eigenvalues of the mixing matrix W) and non-smoothness or high-frequency behaviour (through negative spectra of W), which is often connected to heterophily. This is an interesting concept, but not developed fully.  From Section 3 onwards, we see that the rest of the analysis takes the form of a regular graph/neural ODE such as GRAND, and does not directly make use of the energy except in Section 5.

- In the end of Section 3, the authors argue ‘We argue that energies rather than evolution equations should be the object to parameterise for deriving more principled GNNs that are easier to interpret and analyse. ‘ This argument continues throughout without really being made clear of its necessity or advantages. 

- I like the perspective of the parameterised inspired from ‘finding the right notion of smoothness for our task’ as mentioned on page 4. However, beyond this minor intuition, I am unable to see the importance or the unique bias the energy view of GNNs brings over the evolution equations.

- Lot of the later arguments regarding LFD/HFD is done using Dirichlect energy and seems disconnected from the energy functional we are introduced to before. This characterisation is nonetheless interesting and of value. However, the connection between HFD and heterophily is not obvious, particularly given that we are not shown how these properties pop up in the actual benchmark datasets, specially in low-homophily ones.

Experimental validation 

- The authors present to us the interesting case of a synthesized Cora dataset at different levels of homophily. We are shown comparison with cases where the mixing matrix W is strictly negative in spectrum (neg-prod), positive in spectrum (‘prod’), and a general W. We are shown that in low homophile cases, the ‘neg-prod’ seems to have better performance over ‘prod’, and the other way round for high-homophily. However, in all cases, the individual performances never exceeds having a general W which means even in extremely low homophily situations, the energy functional learns to retain a component which promotes smoothness across features of LFD nature. This is interesting, and not entirely clear why - given the argument that heterophily essentially would be better off to have ‘repulsion’ amongst the particles.

- The authors should better explain why the performance still remains lower than an MLP. Further, it would be advisable to show the reader the relative spectral components (signs of eigenvalues) of the learned W for different homophily levels — this would help validate the hypothesis that indeed the attraction repulsion dynamic is in play (as discussed in section 3.2). Also given that Graph isomorphism convolution layer (GINConv) [R0] is known to be one of the most expressive graph convolution layers available, I suggest to make comparisons with that also as a baseline in the experiments.

- Similarly, it would be interesting to see the spectral aspects of the mixing matrices learnt for all of the real-world  heterophily datasets reported in Table 1 - do the learnt W and their eigenvalues mostly agree with the sign that is expected from the proposed analysis through Section 3.2?


Other minor comments:

- Theorem 5.1, I believe should be seen as Lipschitz continuity and not monotonicity as the authors claim on page 7.

- The performance of GRAFF and GRAFF_{NL} is often fairly close — does this mean that the non-linearity  $\sigma$ has relatively no role to play? Alternatively, how does the spectrum of the learnt W matrices differ for both the cases for a given dataset?

- Before Section 5, the authors statement ‘Convolutional GNN models can deal with heterophily if the channel mixing matrix has negative eigenvalues. ‘ — does not seem clear or substantiated. The authors should make this clear, or explain this through examples.

- In the experiments on benchmark data, we observe that in many cases, GGCN or even GCNII (that are known to be special cases of equation (5)) outperform GRAFF even at low homophily levels. Can this be explained in some way? Perhaps through the associated learnt energy functionals — currently I don’t see how the energy functional view has helped us in understanding these observations.


- Finally, since the entire framework is based on dynamical systems, it would be nice to see if the model is able to capture actual dynamic systems, as considered in [R1] for example.



[R0] How Powerful are Graph Neural Networks? K. Xu, W. Hu, J. Leskovec, S. Jegelka. ICLR 2019

[R1] Chengxi Zang and Fei Wang. 2020. Neural Dynamics on Complex Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20). Association for Computing Machinery, New York, NY, USA, 892–902. https://doi.org/10.1145/3394486.3403132
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Inspired by gradient flows as (solution curves to) differential equations that minimize an energy functional, the authors view GNNs as a gradient flow equation of a parametric energy. The authors show that in graph convolutional models (GCN), the positive/negative eigenvalues of the channel mixing matrix correspond to attractive/repulsive forces between adjacent features. They also demonstrate how the channel-mixing can learn to steer the dynamics towards low or high frequencies, which allows to deal with heterophilic graphs.
The authors provide a use case to show experimentally how  the gradient flow framework is more efficient than GCN, and achieves competitive performance on graph datasets of varying homophily often outperforming recent baselines specifically designed to target heterophily.
strength_and_weaknesses: Strengths
   + Provides an interesting (albeit not completely new) perspective to view GNNs from the lens of gradient flows (of a energy functional).

    + Detailed proofs of the mathematical claims made in the paper

Weaknesses:
   - There are implicitly assumptions that are not clearly stated and some of the terminologies seem to be loosely used (see the next section for more details); 
   -  There also appear to be gaps between the "theoretical framework" and how it is applied in practice (with discrete datasets). 
clarity,_quality,_novelty_and_reproducibility: Mathematically, gradient flow (in a linear space) -- or steepest descent curve -- is a smooth curve x : R --> X (a linear space) such that $x'(t) = - \nabla E(x(t))$ for a linear function $E: X-->R$. 

I assume that in the paragraph "What is a gradient flow" in Section 2, F(t) is the gradient flow, and ODE(F(t) indicates $\dot{F}(t)$?

Can you provide some justification for eq.(5) (and equivalently eq.(11)), and why it represents the gradient flow for the "energy functional" defined in eq.(6)?    In other words, given eq.(6), why does eq.(11) represents its gradient flow? 

The same question applies to eq.(10).

Furthermore, how does the "graph" (or adjacency matrix A) come to the picture? What is the underlying space/domain X that the energy function is defined, and how it evolves across the space/domain over time? 

There are a lot of "hand-wavy" statements. What are "features"  $f_i$ and $f_j$?  

In practice, what do you actually try to learn, the "energy functional" or "gradient flows"? I assume the former.
In terms of experiments, your eqs. (17) and (18) basically provide an "evolution" equation. How can you ensure that it gives the right "gradient flow" for the energy functional that you try to learn?  With the specific forms you used, basically the second part in the right hand side of eq.(17) (or eq.(18) gives us $\nabla E$. But since it involves the parameters that you need to learn, how do you ensure that it provides the "correct" gradient of the energy function?  In the end, it seems that you are just blindly using GNNs to learn some function based on some loss functional. We end up in square one --- we still do not necessary learn what the GNN actually learns.

A more basic question: can you clearly define what you mean by the gradient operator $\nabla$,  e.g., $\nabla F$ (or $\nabla E_{\theta}$) that you used in the paper? Note that in general, $\nabla$ operator is defined using a (local) Euclidean coordinate system.  Here you are talking about a graph. What is the underlying domain in which either the energy function $E_{\theta}$ or the gradient flow $F$ is defined? If the nodes in the "graph" represent "particles" in an N-particle system, these particles still "reside" within, say, a 3-dim Euclidean space.
summary_of_the_review: The paper presents an  interesting (albeit not completely new) perspective to view GNNs from the lens of gradient flows (of a energy functional).  While I appreciate this new lens to view GNNs, there are a lot of implicit assumptions, confusing notions, vague definitions and various formulas that are not well explained or justified. As such, it makes decipher the mathematical statements and proofs hard to follow and ascertain. There are a number of ad hoc constructions. It is unclear the proposed framework really provides any deeper insight into how GNNs work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work suggests an energy based perspective on GNNs, and by an analysis of the application of GNN layers and the channel-mixing operator sheds light on the behavior of GNNs.
Several experiments are conducted showing improvement over the considered methods in some cases.
strength_and_weaknesses: Strengths:

+This work takes inspiration from recent works like GRAND, PDE-GCN and GraphCON and extends the analysis of GNNs and their interpretation as dynamical systems that can be explored through their energies.

+The authors sufficiently cite and discuss relevant background.

Weaknesses:

-The authors discuss the possibility of 'negative edge weight' in page 7. However, it is not clear what is the relation between allowing negative edge weights and recent works that propose designated architectures for this kind of data, for instance "Learning signed network embedding via graph attention" (AAAI 2020), "Signed graph attention networks" (ICANN 2019).

-The authors show in Theorem 5.1 that the considered energy decreases from layer to layer which as discussed in the paper it is not anticipated to work well for heterophilic datasets. Then, I do not understand the obtained accuracy on such datasets. What causes the method to behave well? did the authors use only a small number of layers?

-The phenomenon of oversmoothing is widely spread in GNNs and many works (that are properly cited in the paper) propose various ways to tackle this issue. The authors also discuss this issue but no study was done in this important aspect. 

-In page 8, paragraph "The model and the parameterisation" the authors suggest two models for the learned matrix W, but it is not clear what is the real influence of each of them. According to the appendix, I can only know what model was used, but as a reader I cannot infer what is the impact of those models on the different datasets. I think that the authors should add an experiment that studies this aspect.

-It is not reported how many layers are used for the different experiments. I think that in this aspect it is important also given the theorems that regard to the energy change between layers.

-Regarding the results in table 1, I believe that the authors are missing comparison with recent works with better performance. For instance "Is Heterophily A Real Nightmare For Graph Neural Networks To Do Node Classification?", "Simple Truncated SVD based Model for Node Classification on Heterophilic Graphs" and "Improving Graph Neural Networks with Simple Architecture Design" obtain significantly better accuracy on some of the datasets (e.g., Chameleon and Squirrel)

-I appreciate the number of experiments but I think that to allow a broader comparison with many existing methods the authors should also report the accuracy on the semi-supervised Cora, Citeseer and Pubmed datasets.

-It is claimed that the code and hyperparameters are provided but I cannot find them.

*A question to the authors: can this method be used for graph classification? 

clarity,_quality,_novelty_and_reproducibility: *The paper is not the easiest to follow.

*The paper offers a quality analysis of GNNs through their energy functions.

*The paper continues the recent efforts in GNNs (as properly cited by the authors) and adopts insights and ideas from the Image Processing world (e.g., Scale-space and edge detection using anisotropic diffusion, as properly cited by the authors)

*The authors claim to include the code and hyperparameters, but I could not find it anywhere. I therefore cannot say that it is reproducible.
summary_of_the_review: The paper continues the interesting direction of interpreting GNNs through their energy function. I think that it has a merit but the experiments can be extended and compared with more methods, and many of the details are missing.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This mostly theoretical work presents a framework of GNNs in the context of Dirichlet energy minimization/maximization. GNNs are framed as estimators for gradients of energy functionals that point to either of those two directions. It shows that for certain classes of graphs and GNNs, attractive or repulsive forces between node features overweigh the other, leading to dominance of low or high frequencies, respectively, after convergence.  Based on this framework, the authors are able to design a GNN architecture that works well for low-homophily and high-homophily, depending on parameterization.
strength_and_weaknesses: Strengths:
- The presented theoretical framework is novel and interesting to read. It provides a new perspective on GNNs.
- The framework unifies many existing GNN architectures and is able to derive conclusions about their behaviour with respect to favoring high or low frequencies. These conclusions are partially supported by proof of concept experiments.
- The paper exposes a guideline of how to design GNNs that favor high frequency encodings, which might be useful to certain practical tasks, as most existing GNNs favor low frequency.
- The paper is very well written. I am not an expert in ODEs and only roughly familiar with the most recent GNN theory and I was able to follow the paper mostly.

Weaknesses:
- The experiments are on the weak side of the paper. While theoretically shown effects are supported by proof of concept experiments, real applications of the presented phenomena are lacking.
- In general, it would be great if the authors make stronger connections to practice, e.g. by zooming in on a specific application/graph type that would benefit from the presented method in a special way. That would also give a good intuition about how the knowledge can be applied to other tasks as well.
clarity,_quality,_novelty_and_reproducibility: - Paper clarity and quality is great
- The work is original and novel to my knowledge
- The presented GNN architecture seems easy to reproduce with additional content given in suplementals
summary_of_the_review: In summary, I think this paper should be accepted, as it provides a new perspective of GNNs as gradient estimators for energy functionals. The paper, while reading, gave me a few ideas and a mental framework to work with, when thinking about GNN solutions for specific problems.

Questions:
- This work focusses on architectures favoring low or high frequency content. I was wondering if it wouldn't be best to converge to a range of different frequency content, e.g. in different feature groups, to extract most from the given graph. What do the authors think here?
- As far as I understand, the architecture only allows for switching between low and high frequency convergence with sign flipping. Could you think of a way of providing more explicit control over the frequency in the given framework?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This  paper gives a new characterization of GNNs as a gradient flow to minimize a energy functional that is a combination of training loss, Laplacian smoothness penalty and deviation from the initialization. Based on this viewpoint, the authors proposed two methods called GRAFF and GRAFF_NL and these methods are compared with other methods in numerical experiments. 

Strength: The paper is well written. The gradient flow characterization is quite intuitive and instructive to understand what the GNNs are doing in the internal layers.  
Weakness: On the other hand, there are some weakness in the paper.  
First, the gradient flow viewpoint itself is not new. The ODE characterization has been suggested by several existing work as cited in this paper. The main novelty of this paper is to consider a bit different energy functional with attractive and repulsion power (and deviation from the initialization). However, given recent methods utilizing negative weight edge or tailored to heterophilic data, this kind of proposal does not seem entirely new. A more convincing argument to highlights the novelty would be required. (Actually, the numerical experiments do not show a clear performance improvement over existing methods, though this would not be a main issue.)  
Another concern is that this paper mainly discusses the linear ODEs, and there is very limited discussion on the nonlinear activation. This is a challenging issue in the literature. However, Theorem 5.1 for nonlinear activation only asserts that the energy does not increase so much and it does not give any convergence of a properly defined energy.

justification_for_why_not_higher_score: The main drawback of this paper is its novelty. The gradient flow viewpoint itself is not so much novel. The construction of energy functional would be novel, but it do not give a quite new algorithm. Indeed, the resultant algorithm is rather a combination of existing approaches. Hence, this paper is not sufficiently strong for ICLR publication.
Decision: Reject