Port-Hamiltonian Architectural Bias for Long-Range Propagation in Deep Graph Networks

Authors: Simon Heilig, Alessio Gravina, Alessandro Trenta, Claudio Gallicchio, Davide Bacciu
Primary area: learning on graphs and other geometries & topologies
Keywords: graph representation learning, long-range propagation, ordinary differential equations

Abstract:
The dynamics of information diffusion within graphs is a critical open issue that heavily influences graph representation learning, especially when considering long-range propagation. This calls for principled approaches that control and regulate the degree of propagation and dissipation of information throughout the neural flow. Motivated by this, we introduce port-Hamiltonian Deep Graph Networks, a novel framework that models neural information flow in graphs by building on the laws of conservation of Hamiltonian dynamical systems. We reconcile under a single theoretical and practical framework both non-dissipative long-range propagation and non-conservative behaviors, introducing tools from mechanical systems to gauge the equilibrium between the two components. Our approach can be applied to general message-passing architectures, and it provides theoretical guarantees on information conservation in time. Empirical results prove the effectiveness of our port-Hamiltonian scheme in pushing simple graph convolutional architectures to state-of-the-art performance in long-range benchmarks.

Decision: accept (poster)
Reviewer scores: 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1729822898690

thread_type:
official_review

<thread_object>
creator: reviewer_7pxw

<content>
summary:
This work provides a novel methodology for the incorporation of port-Hamiltonian dynamics in graph representation learning. The central model, called a port-Hamiltonian Deep Graph Network (PH-DGN), is introduced first in a purely conservative setting using only the Hamiltonian and no non-conservative terms. Several theorems are developed to show that this conservative case leads to long-range data propagation between graph nodes, where the graph dynamics exhibit no energy loss and gradients do not vanish as the backward sensitivity matrix is bounded below. Dissipitive forces are then added to the full port-Hamiltonian model, which are two additional networks that may be trained to capture non-conservative forces. Several experiments follow, including a showcase of energy conservation and sensitivity to empirically verify theoretical work, and a graph transfer problem and graph property prediction problem to compare performance on benchmark tasks against other graph models in settings which require long-range information propagation.

strengths:
- There is very clear motivation to this work, and it builds nicely upon other references.
- The proposed port-Hamiltonian approach is an original and clever way to allow for non-conservative dynamics in a graph network while still maintaining long-range message passing.
- The theoretical results for the conservative case are strong, and the motivation and interpretation for these are presented nicely.
- The experimental setup is superb; the care taken to ensure ease of replication is applauded. Model details are presented very clearly and choices are explained well for each step of the setup.
- A strong suite of models are compared against, with many different competitors and different approaches. The consistently favorable results provide a great deal of strength to claims of the proposed method's performance.
- The appendices are comprehensive for both proofs and experimental setup, and made clear many of the questions I had on an initial read.

weaknesses:
- The majority of the theoretical results are developed for the conservative case. This makes sense in the context, as conservative long-range message passing is stated as a goal, but I would also be quite interested to see what could be proven for the fully general port-Hamiltonian case.
- In the explanation of Theorem 2.3, the statement that "the final representation of each node retains its complete past" seems somewhat strong. While I understand that the BSM result shows the influence of the entire past history on the current state, this statement as written seems to imply something stronger, and perhaps could be made more clear.
- The dissipitive force terms are added in some experiments to great success, but the explanations of their performance are more intuitive and are not supported by hard data in the paper. There may be a great opportunity here for visualization to support the intuitive claims.

There are two very minor typos:
- In the first paragraph of Section 2, "node states' in graph" has an unnecessary apostrophe.
- In Appendix D.3, "on a grid of n. layers" has an unnecessary period.

questions:
- How would classical GCN aggregation interact with Theorem 2.4? Can the bound be easily extended for that case?
- In Section 3.1, you mention that the growing behavior can be controlled by regularizing the weight matrices or using normalized aggregation functions. Did you try this? How are the empirical results?
- Have you examined the interpretability of a trained PH-DGN? In particular, do the learned non-conservative forces make sense for the associated problem?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645938384

thread_type:
official_review

<thread_object>
creator: reviewer_xxsd

<content>
summary:
This paper proposes PH-DGN, a new GNN based on the port-Hamiltonian system to develop GNN that can solve graph learning tasks that require long-range dependencies. Two variants of PH-DGN are proposed: a conservative PH-DGN based on the Hamiltonian system and a PH-DGN based on the port-Hamiltonian system by introducing learnable dissipative terms. 
The theoretical analyses show that the conservative PH-DGN is stable and energy-preserving as a dynamical system, and derive a lower bound for the sensitivity, implying the possibility of long-range interaction.
Numerical experiments show that the conservative PH-DGN is energy-preserving without gradient vanishing using synthesis datasets (Section 3.1) and that the long-range interaction can be achieved in graph learning tasks that require long-range interactions (Sections 3.2, 3.3). Also, the usefulness of two variants of PH-DGN is evaluated on long-range graph benchmarks of real datasets.

strengths:
- The background knowledge of the port-Hamiltonian system is explained carefully, increasing accessibility for readers unfamiliar with this topic.
- For conservative PH-DGN, the ability of long-range interaction is theoretically shown by giving the lower bounds of sensitivity.
- The validity of the proposed methods for synthesis datasets is properly demonstrated.

weaknesses:
- As the title indicates, the theme of this paper is the relationship between GNN and long-range propagation based on the port-Hamiltonian system. However, no theoretical guarantees on long-range propagation are given for general PH-DGNs with dissipative components.
- The tables of experimental results could be clearer (in particular Tables 2 and 5).
- For $\mathrm{PH-GDN}_{\mathrm{C}}$, which has theoretical guarantees, the prediction performance on the real dataset (long-range graph benchmark) does not outperform existing methods, and hence its practical usefulness is limited.

questions:
**Questions**

- l.161: What is the definition of anti-derivative of an activation function $\sigma$?
- l.223, l.228: Which norm is used in $\| \partial \mathbf{b}_u(T) / \partial \mathbf{b}_u(T-t) \|$? 
- l.259: *Therefore, [...]. , it holds the capability of PH-GDN to perform long-range propagation effectively*: the authors compare the upper bounds of sensitivity for the existing model (Theorem 2.3) and the proposed model (Theorem 2.4), then argue that since the latter is larger than the former, the proposed model is more effective in performing long-range propagation. However, it is difficult to claim so because there is the possibility that Theorem 2.4 only shows a looser bound than Theorem 2.3, which does not exclude the possibility that the upper bound does not properly reflect the sensitivity of the PH-GDN. It should be shown theoretically or experimentally that this upper bound adequately reflects the sensitivity of the PH-GDN.
- l.1129: I want to clarify the setup of the Graph Transfer Task: If I understand correctly, the feature vectors of all nodes are 1-dimensional and randomly sampled from $\mathrm{Unif}([0, 0.5))$. The target value is 0 for the source node and 1 for the target node. Assuming that this problem setup is correct, I need help understanding why this task is solvable because the model cannot distinguish source or target nodes from other nodes from feature vectors.


**Minor Comments**

- l.196: *non-dissipative (i.e., long-range)*: I think this is a slightly misleading expression. My understanding is that this paper uses non-dissipative in the sense of energy-perserving. Although non-dissipative implies long-range propagation (Theorem 2.3), they are not equivalent. In fact, this paper argues that non-dissipative PH-DGN performs better than conservative PH-DGN in predicting the LRGB dataset in some numerical experiments.
- l.436: The use of position encoding is only mentioned in the caption of Table 2 and should be explicitly stated in the text, specifically in the setup of Section 3.4.
- l.436: The correspondence between Table 2 and Table 5 needs to be clarified. For example, the method titled *MPNNs* in Table 2 is titled *re-evaluated* in Table 5. However, GCNII is not listed as *re-evaluated*, but as *MPNNs*. Furthermore, it is difficult to tell from the captions of Table 5 whether the authors test each listed method by themselves or is a citation of existing research. The reference should be indicated for each method in Table 5, for example, by adding a column such as *reference* to Table 5.
- l.443: *Overall, our port-Hamiltonian framework [...] shows great benefit [...] without requiring additional strategies such as global position encoding, global attention mechanism, or rewiring techniques [...].*: I suggest clarifying how the usefulness of each strategy is shown by comparing it with existing methods. More specifically:
  - The superiority of the proposed method over global position encoding is justified by the comparison with MPNN-based models using position encoding.    　　
  - The superiority over the global attention mechanism is by comparison with the Transformer-based method.
  - The superiority of rewiring methods by comparison with Drew.
- l.1153: The reference to Adam (Kingma & Ba, 2015) should be cited.
- l.1343: n. layers -> the number of layers
- l.1353: Table 6 claims that PH-DGN achieves both Hamiltonian conservation and Learnable driving forces. However, I think this is misleading because to achieve Hamiltonian conservation, the dissipative component must be removed, resulting in $\mathrm{PH-DGN}_{\mathrm{C}}$. It is not possible to achieve both characteristics at the same time in one architecture. Instead, two variants that achieve only one of the two are proposed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706862818

thread_type:
official_review

<thread_object>
creator: reviewer_x6rz

<content>
summary:
This paper introduces port-Hamiltonian Deep Graph Networks (PH-DGN), a new framework for graph neural networks that addresses the challenge of long-range information propagation. The approach embeds message passing within a port-Hamiltonian dynamical system framework, where node states are split into position (q) and momentum (p) components coupled through skew-symmetric matrices. The Hamiltonian function incorporates graph structure through neighborhood aggregation functions, and the port-Hamiltonian extension introduces learnable dissipative components (internal dampening and external forces) that allow the network to balance conservative information preservation with task-dependent information modification. The authors provide theoretical analysis of the framework's properties, including bounds on sensitivity and gradient behavior, and demonstrate empirically that their method outperforms existing approaches on several tasks requiring long-range information propagation, including synthetic graph property prediction tasks and real-world molecular property prediction. The framework can incorporate different message passing schemes and provides improved efficiency compared to previous Hamiltonian-based graph neural networks, with experimental results showing that while the purely conservative version performs well, including the dissipative components often leads to better task performance.

strengths:
Overall, the idea of using Hamilton's dynamics for GNN is attractive, though not entirely new. 
Nevertheless, the paper is solid and is more general than existing approaches. 
The improved efficiency over previous Hamiltonian GNN approaches and the practical benefits for long-range propagation make it a useful contribution to the field.
The experimental results are also interesting. 

1. Technical soundness:
- Clear theoretical analysis of conservation properties
- Explicit connection between Hamiltonian dynamics and message passing
- Thorough experimental validation

2. Practical value:
- More efficient than previous Hamiltonian GNN approaches
- Good performance on long-range tasks
- Can incorporate different message passing schemes

3. Clarity:
- Well-structured presentation
- Good balance of theory and empirics
- Clear comparisons to prior work

weaknesses:
1. Novelty is incremental:
- Builds on existing ideas (Hamiltonian GNNs, message passing)
- Main contribution is combining these effectively rather than fundamentally new concepts

2. Technical questions:
- The derivation of the discretization scheme could use more justification
- Some assumptions about the structure of $W$ and $V$ matrices for explicit updates feel restrictive
- Could better explain why port-Hamiltonian framework is more appropriate than simpler alternatives

3. Empirical:
- Some ablation studies could be stronger (e.g., analyzing impact of different dissipative terms)
- Could better justify hyperparameter choices

questions:
1. While the paper shows good empirical performance, it's unclear what types of problems would theoretically benefit most from a port-Hamiltonian approach versus standard message passing. Could the authors provide analysis or insights about which properties of the underlying data generation process would suggest using their method?

2. The authors demonstrate that adding dissipative components often improves performance, but how does the balance between conservative (Hamiltonian) and dissipative parts relate to the nature of the task? It would be valuable to see an analysis of when pure conservation might be preferable to including dissipation, and how practitioners should choose this balance for new problems.

3. Given that the method is inspired by physical Hamiltonian systems, it's surprising that there are no experiments on problems with actual Hamiltonian dynamics (e.g., molecular dynamics simulations). Such experiments could help validate whether the method's conservation properties provide advantages for physically meaningful conservation laws, beyond just improving general information flow.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732285259487

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
We sincerely thank the reviewers for their thoughtful and detailed feedback, as well as for recognizing the key strengths of our work. We are happy to read that the reviewers found that our paper provide an ***attractive*** and ***novel*** methodology (Revs. X6rz, 7PXw) for the incorporation of port-Hamiltonian dynamics in graph representation learning. We are also grateful for the acknowledgment of the ***clarity*** (Revs. X6rz, XxSd, 7PXw) and ***technical soundness*** (Rev. X6rz) of our work while reporting ***strong theoretical results*** (Revs. X6rz, XxSd, 7PXw) and ***thorough*** (Rev. X6rz) and ***superb*** (Rev. 7PXw) experimental validation to show the practical benefits (Revs. X6rz, XxSd) for long-range propagation. 

We are also thankful for the constructive feedback, which has further improved the quality of our paper. Specifically:

**Additional Experiments:**

- Following the Reviewer **X6rz**’s and Reviewer **XxSd** ’s suggestions, we have added a novel experiment on a recent benchmark to: (i) appreciate the impact of the different dissipative components in our approach, and (ii) highlight the utility of the purely conservative version of PH-DGN (i.e., PH-DGN$_C$). To further address point (ii), we have also included an additional ablation study on the graph transfer task, demonstrating the improved utility of PH-DGN$_C$ in tasks that require preserving all information.

**Revisions to the Paper:**

- Following the Reviewer **XxSd**’s and Reviewer **7PXw**’s suggestions, we incorporated additional theoretical guarantees on long-range propagation that also accounts for the presence of non-conservative forces.
- In Appendix A.3, we clarified that the assumption on the structure of W and V matrices is not restricting the final implementation of $p$ and $q$ components. 
- We improved the discussion on the choice of the hyperparameters.
- We improved the clarity of Tables 2 and 5 as well as provided a deeper clarification on the usefulness of the proposed model with respect to existing techniques in Section 3.4. 
- We clarified the goal of the graph transfer task.
- We revised Table 6 to clarify that driving forces do not allow for purely Hamiltonian conservation.

----

As the author-reviewer discussion comes close to an end, we would like to thank the reviewers again for their invaluable feedback and the positive assessment of our paper. We did our best efforts to provide a detailed response to reviewers’ comments, and we hope these revisions address all concerns while further emphasizing the significance and robustness of our contributions.

In particular, we would greatly appreciate hearing from Reviewer **XxSd** whether they were satisfied with our responses. We hope that this is the case, and, if so, we would like to kindly ask the reviewer to consider revising their score.

Thank you all again for your constructive feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**(a) Scientific Claims and Findings:**
The paper introduces a novel framework called port-Hamiltonian Deep Graph Networks (pH-DGNs). This framework models neural information flow in graphs by leveraging principles from Hamiltonian dynamical systems, aiming to address challenges in long-range information propagation within graph representation learning. By incorporating both non-dissipative and non-conservative behaviors, the approach seeks to balance information conservation and dissipation, providing theoretical guarantees on information preservation over time. Empirical results demonstrate that pH-DGNs enhance the performance of simple graph convolutional architectures, achieving state-of-the-art results on benchmarks requiring long-range propagation.

**(b) Strengths:**
* Innovative Framework: The introduction of port-Hamiltonian systems into graph neural networks offers a fresh perspective on managing information flow, potentially addressing limitations in existing architectures concerning long-range dependencies.
* Theoretical Foundations: The framework is grounded in well-established principles from Hamiltonian dynamics, providing a solid theoretical basis for the proposed approach.
* Empirical Performance: The proposed method demonstrates superior performance on benchmarks that require long-range information propagation, indicating its practical effectiveness.
* Applicability: The approach can be integrated into general message-passing architectures, suggesting broad applicability across various graph-based learning tasks. 

**(c) Weaknesses:**
* Complexity: The incorporation of port-Hamiltonian systems may introduce additional complexity into the model, potentially impacting computational efficiency and implementation.
* Scope of Evaluation: While the empirical results are promising, the evaluation is primarily focused on benchmarks requiring long-range propagation. Assessing the framework's performance across a wider range of tasks and datasets would provide a more comprehensive understanding of its capabilities. The empirical improvements are limited in some cases.
* Practical Implementation Details: The paper could benefit from a more detailed discussion on the practical aspects of implementing the proposed framework, including computational requirements and potential challenges in real-world applications.

**(d) Reasons for Acceptance:**
After a thorough evaluation of the paper, I recommend acceptance based on the following considerations:
1. Novel Contribution: The paper presents a unique integration of port-Hamiltonian systems into graph neural networks, offering a new approach to addressing challenges in long-range information propagation.
2. Theoretical Rigor: The proposed framework is underpinned by solid theoretical foundations from Hamiltonian dynamics, enhancing the credibility and potential impact of the work.
3. Empirical Validation: The method demonstrates state-of-the-art performance on relevant benchmarks, providing empirical evidence of its effectivenes. Comparisons for the Minesweeper task from Platonov et al. (2023) were added during the rebuttal.
4. Broader Impact: The approach's applicability to general message-passing architectures suggests it could influence a wide range of graph-based learning tasks, contributing to advancements in the field.
While there are areas for improvement, such as expanding the scope of evaluation and providing more practical implementation details, the paper's strengths and contributions to the field warrant its acceptance.