Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Extending recent works on Hamiltonian Neural Networks (and Riemannian Manifold GNNs), the paper considers the graph as a discretization of an underlying manifold, and construct a Hamiltonian neural network architecture (HamGNN) for graph learning problems. Building on work  in [Chen et al, 2021],  The Hamiltonian neural network architecture learns the symplectic form defining  the Hamilton flow associated with the Hamiltonian scalar function on its cotangent bundle. The authors investigate several symplectic forms and empirically demonstrate that the proposed approach achieves better node classification accuracy than popular state-of-the-art GNNs.
strength_and_weaknesses: Strengths

   + While based on the "Neural symplectic form" first developed in  [Chen et al, 2021], the paper appears to generalize to graph learning problems.
   + The paper provides right amount of background and mathematical preliminaries to make the paper accessible.
   + The terms and notations are clearly defined.


Weaknesses:
   - The model architecture in section 4.3 could be better explained. Reading the paper and supplementary material only do not provide the reviewer a good idea how the proposed model works. One has to go back to  [Chen et al, 2021] to learn how simpletic forms are actually learned.
   -  While the evaluation results show the proposed HmmGNN outperforms some of the state-of-the-art, I would have liked to better explain why Hamilton flows make sense for the node classifications using Citeseer and Cora datasets. For example, what does  Poincaré 2-form intuitively capture here?
clarity,_quality,_novelty_and_reproducibility:  + In terms of clarity, overall, the paper is well written; with the right background knowledge (about Hamilton flows and differential geometry), the paper is not too hard to follow.

 - In terms of novelty, most of the novelty comes for the paper  [Chen et al, 2021] which first proposes directly learning the sympletic 2-forms.  The paper  [Chen et al, 2021]  focuses primarily learning unknown Hamiltonian equations from sampled datasets.  This paper applies to "general" graph learning problems, and investigates several symplectic two-forms.

 +/-  In terms of quality, the paper could have done a better job in justifying why the Hamiltonian flow framework is the right one for many graph learning problems. Given that the Hamilton flows come from physics with certain "conservation laws", what do "Hamiltonian flows" and "sympletic" 2-forms for non-physics settings, e.g., "social network analysis" problems using citesser and Cora datasets, capture? Otherwise, it still seems to be a "black" magic.  Although the authors claim that the proposed HmmGNN helps advoid oversmoothing, there are no theoretical justifications for it.

Incidentally, while the authors in  [Chen et al, 2021]  claim that their method is a "coordinate-free" framework from learning symplectic forms, it is in fact not --- it merely does not assume the specific Darboux coordinate system, and instead assumes the 2-form is expressed in a general local coordinate system. Any time one writes any form in terms of partial differentials, one needs to use a local chart  -- thus one can computations (locally). For the examples in the paper, it seems that the authors assume that the Hamiltonian equations can be written in a single local chart (i.e., everything operates in a Eucliean space), The paper does not address the general manifold where one needs multiple (local) charts, thus chart transitions are needed to ensure consistency of the learned 2-forms.

  +/- The authors provide code and other material as "supplementary material." While the authors show results that illustrate better results, I am not sure these are necessarily fair comparisons. I would have liked to see addition information regarding the number of parameters used, training time, etc.

 
summary_of_the_review: Building on work  in [Chen et al, 2021], the paper considers the graph as a discretization of an underlying manifold, and construct a Hamiltonian neural network architecture (HamGNN) for graph learning problems.  The authors investigate several symplectic forms and empirically demonstrate that the proposed approach achieves better node classification accuracy than popular state-of-the-art GNNs. The authors could have done a better job in justifying why the proposed framework (that originates from solving physics problems) is a general fit for other non-physics graph learning problems.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a novel application of differential geometry and Hamiltonian flows to graph neural networks and demonstrate its applicability to the common GNN problem of node classification. The theory gives rise to a GNN model which is then shown to perform relatively well on standard datasets and somewhat to resist over-smoothing as the depth of the network increases.

strength_and_weaknesses: The authors propose an interesting application of Hamiltonian flows and differential geometry to graph neural networks. However, most of the models they compare with in the experiments section are not the newest and despite the paper’s claims, no longer represent the state of the art. I recommend the authors compare to more modern graph neural networks including GCNII (2020), LGCN (which is cited in the related works section but not compared to), GRAND (Chamberlain et al.), PDE-GCN (Eliasof et al. 2021), GraphCon (Rusch et al. 2021), EGNN (Zhou et al 2021) or newer. According to the results listed, the proposed method does not achieve SOTA performance compared to the works cited above.

In the absence of SOTA performance, the paper does not otherwise substantiate the use of the mathematical theory. The contribution of the proposed methods is not discussed in terms of any other metric, such as reduced running times or the number of parameters, etc. Furthermore, the authors explore different network depths, but the point of obtaining deeper networks is lost when accuracy is not comparable to other models of similar depth.

The paper is difficult to follow, and the method of Hamiltonian flows is not clearly motivated for the task of node classification. See the next section for details.

clarity,_quality,_novelty_and_reproducibility: The authors demonstrate a deep understanding of the subject matter and present an interesting framework for graph neural networks. The proposed method seems to be a novel application of Hamiltonian flows to graph neural networks and the novelty of the work is positioned well by the related works section.

That being said, I find the paper difficult to follow, and several typos can be found in the text. Furthermore, the mathematical theory used is not sufficiently motivated and the advantage of the method is not sufficiently clarified — especially given that the performance of this method is not better than the current state-of-the-art methods (see the previous section).

The paper spends a lot of space presenting definitions and theorems from differential geometry and Hamiltonian mechanics using notation that may be greatly simplified and clarified. For example, the definition of $q$ on page 3 is confusing, and the definition immediately following it is unclear. Also, the fact that definitions have no number or name associated makes it difficult to refer to them.

Very little space is dedicated to discussing the actual method being proposed, and generally, I find that it is difficult to locate the salient information in the text. The notational conventions used may not be clear to members of the community, e.g., the coproduct or $U \rightarrow q(U)$. It is also recommended to include a citation of the respective papers in comparison tables since model names tend to be very similar to each other and can be confusing.

The authors have provided code to facilitate reproducibility. The proposed architectures, experimental setups, and training parameters are not listed fully in the main paper and appendices. Perhaps the supplementary materials are sufficient to reproduce this research. However, I did not attempt to run the code. 

summary_of_the_review: I find the paper very interesting, however the authors do not present compelling motivation or outstanding experimental results for this reviewer to consider using the methods. They do not discuss the methods and experimental setups in sufficient detail, and do not provide comparisons to state-of-the-art methods.

The paper is well presented, though could benefit from more editing work to fix typos, clarify notations, and add necessary details to motivate its inclusion into scientific archives.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a Hamiltonian based feature learning in GNNs to overcome the oversmoothing and oversquashing phenomena that are evident in many existing GNNs. 
A rigorous theoretical discussion is provided and several experiments are conducted. 
strength_and_weaknesses: Strengths:

- The authors provide a rigorous theoretical discussion.
- The authors clearly explain why Hamiltonian based GNNs can be useful.

Weaknesses:

- I am not convinced that the proposed method is the first to be based on Hamiltonian (hyperbolic) flows in GNNs. For example:
"PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations"

"Graph-Coupled Oscillator Networks"

also utilize such flows but are not discussed in the paper.

- The experimental section lacks comparison with recent and state-of-the-art methods, which are stronger by a significant margin. For example:
GRAND: Graph Neural Diffusion

Simple and Deep Graph Convolutional Networks

Dirichlet Energy Constrained Learning for Deep Graph Neural Networks

- The experimental section's scope is quite narrow. I think that more experiments on additional datasets (e.g., heterophilic datasets like Cornell and Wisconsin, larger datasets like ogbn-arxiv) are required to measure the benefit of such an approach. 

- The authors focus only on transductive datasets (this is stated in text), but it is not clear why. Is it a limitation of the method?

-The paper is hard to follow and in my opinion can be better organized.

- The authors discuss the oversmoothing problem but in the experimental section only report the results with up to 20 layers. I can not draw conclusions based on 20 layers and would recommend the authors to include results with a larger number of layers(e.g, 64 layers).






clarity,_quality,_novelty_and_reproducibility: The methodological part of the paper is not clear and not easy to follow. 

The authors do rigorously analyze the Hamiltonian flows model, but the idea itself is not novel (please see papers references in my review above).

I am not sure that given the details in the paper I could reproduce the model proposed by the authors.
summary_of_the_review: The paper sheds more light and analyzes the Hamiltonian flows approach in GNNs. Several experiments are conducted, but they are far from current model accuracy and performance, and proper discussion of existing and relevant model is missing, including a quantitative comparison with recent GNNs.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose to address the oversmoothing and oversquashing problem in graph neural networks by introducing the Hamiltionian flow. 

The main contributions are: 
1. Introduction of Hamlitonian flow from differential geometry perspective
2. Propose to use Hamiltonian Neural Network in graph node embedding and classification problem
3. By the claim of the paper, the proposed method achieves state-of-the-art performance in node classification task 

strength_and_weaknesses: The strength of this paper lies in its novel idea of using the Hamiltonian dynamic system in replace of the traditional bipartite graph-based neural network architecture in each layer. By leveraging the energy conservation property of the Hamiltonian, it stabilizes the node embedding process to overcome the over-smoothing issue due to deep architecture of the neural network.   

The major weakness of this paper are as below:
1. The paper structure is highly imbalanced. The main contribution of this paper is a novel graph neural architecture in node classification, which is introduced within one paragraph. On the other hands, 3 pages of paper are dedicated to “preliminaries” that neither sufficient nor necessary. It is not sufficient since it requires the readers to finish at least one year worth of study in smooth manifold theory to comprehend all the concepts introduced in the section. It is unnecessary since when it reaches to the implementation of neural network, it comes back to the classical Hamiltonian mechanics which can be introduced without using differential geometry. 

Moreover, what is the benefit to view the problem from differential geometrical perspective is not clear to the reader. As stated above, the node embedding itself is certainly not invariant under change of coordination or diffeomorphism, and the goal for this system to be introduced is to learn the stable/optimal node embedding for classification. It is also unclear, after the introduction of concepts on symplectic vs Riemanian, how these two types of manifolds generate different graphs after discretization. The reader is left wondering why we need to introduce a new type of manifold when Riemanian manifold is naturally available. 

2. The imbalance of the paper is also reflected in the introduction section. The abstract discussed that the challenge of graph neural network is the oversmoothing and oversquashing. These challenges are not discussed in the introduction section. Instead, the authors discuss differential geometric perspective in node embedding models. It feels that the author is motivated by the geometry, not the problem. 

3. The details of the experiment sections are not enough in the main page and many important results are left in the appendix which itself is not clear. For instance, the implementation of Hamiltonian flow part is left with no explanation. The reader is expected to guess what type of ODE solver is involved. It is also not mentioning that normal ODE solver is not guaranteed to have the energy preserving property thus it is not suitable for this experiment. Thus lack of details in implementation makes it impossible to reproduce the result unless to copy the code of the author (which is very uninformative and unclear)

clarity,_quality,_novelty_and_reproducibility: This paper extends the Hamiltonian network to graph node embedding and classification. The paper writing is not clear. It is unclear in multiple aspects:
1. Unnecessarily abstract: Differential geometry is introduced to reveal that the an quantity is invariant under change of parameterization and thus more fundamental than its representation. This problem, however, is about how to learn optimal representation for the given task. In fact, reading the paper, it is unclear that how the theorems in section 3.2 has helped to develop the Hamiltonian neural network model introduced below. In fact, the entire Hamiltonian neural network can be introduced without mentioning of covector and the vector/covector fields. From the purpose of the paper, i.e. to propose a solution to optimize the node classification task, it is not clear why we add these level of abstraction. The differential geometrical perspective has shown fundamental distinct perspective compared to classical mechanics. 

2. Transition from section 2,3  to the proposed framework in section 4 is too sudden. The concept of momentum is a new variable for the network. What is the objective function of the neural network ? it is not clear and the author assume the reader to know this but it is not possible by reading the paper. Also it states “We consider a graph as a discretization of a manifold M” It is unclear why and how. No reference in this part is provided. Is there any tradeoff for this approximation ? it is unclear.

3. The important experiment i.e. the details for resilience to over-smoothing is left in the appendix and only a paper is shown in the main paper. It is not clear as well.  

The paper provides a set of codes for the experiment reproduction. The code structure is not very clear and need more comments and documentation for people to use it. From the experiment section of this paper, it is hopeless to reproduce due to lack of too much details in the implementation. 

summary_of_the_review: In summary, i would not recommend this paper for ICLR. There are several issues that the author need to rework:

1. Rephrase the problem and stress the challenge in the introduction section, instead of focussing on the geometry. Introduction is supposed to motivate the reader to follow the paper not to confuse them.

2. Emphasize the intuition over the abstraction. Differential geometry focus on invariance under diffeomorphism and it is important to stress what is invariant under this new formulation.

3. Please push more geometrical concepts into the appendices and swap them with the experiment details. It is the experiments that the readers are more interested in. 

4. Providing more details on your implementation which also helps to make abstract concepts concrete. This entire paper is floating in the air now since no concrete formulation is provided. 

5. Consider publication in the applied math domains. It is likely that the geometry instead of the graph node classification is the main interest of this paper. It is thus recommended for applied math related domains.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a GNN architecture based on Hamiltonian equations. It falls under the broad recent category of physics-inspired/DE-based GNN models. 

The reviewers complained about insufficient clarity and poor motivation of the use of a differential-geometric mathematical model (not very familiar to the broad ML audience). This is exacerbated by lack of SOTA results (other methods from a same class often perform better). The rebuttal addressed some questions but overall we believe the paper is below the ICLR bar. 

justification_for_why_not_higher_score: lack of SOTA results and clear advantage over related DE-based methods
Decision: Reject