Title: Reversible and irreversible bracket-based dynamics for deep graph neural networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

structure preserving machine learning

neural ordinary differential equations

hamiltonian dynamics

metriplectic dynamics
Abstract: Recent works have shown that physics-inspired architectures allow the training of deep graph neural networks (GNNs) without oversmoothing. The role of these physics is unclear, however, with successful examples of both reversible (e.g., Hamiltonian) and irreversible (e.g., diffusion) phenomena producing comparable results despite diametrically opposed mechanisms, and further complications arising due to empirical departures from mathematical theory. This work presents a series of novel GNN architectures based upon structure-preserving bracket-based dynamical systems, which are provably guaranteed to either conserve energy or generate positive dissipation with increasing depth.  It is shown that the theoretically principled framework employed here allows for inherently explainable constructions, which contextualize departures from theory in current architectures and better elucidate the roles of reversibility and irreversibility in network performance. Code is available at the Github repository \url{https://github.com/natrask/BracketGraphs}.

Reversible and irreversible bracket-based dynamics for deep graph neural networks

Abstract

Recent works have shown that physics-inspired architectures allow the training of deep graph neural networks (GNNs) without oversmoothing. The role of these physics is unclear, however, with successful examples of both reversible (e.g., Hamiltonian) and irreversible (e.g., diffusion) phenomena producing comparable results despite diametrically opposed mechanisms, and further complications arising due to empirical departures from mathematical theory. This work presents a series of novel GNN architectures based upon structurepreserving bracket-based dynamical systems, which are provably guaranteed to either conserve energy or generate positive dissipation with increasing depth. It is shown that the theoretically principled framework employed here allows for inherently explainable constructions, which contextualize departures from theory in current architectures and better elucidate the roles of reversibility and irreversibility in network performance. Code is available at the Github repository https://github.com/natrask/BracketGraphs .

1 Introduction

Graph neural networks (GNNs) have emerged as a powerful learning paradigm able to treat unstructured data and extract 'object-relation'/causal relationships while imparting inductive biases which preserve invariances through the underlying graph topology [1, 2, 3, 4]. This framework has proven effective for a wide range of both graph analytics and data-driven physics modeling problems. Despite successes, GNNs have generally struggle to achieve the improved performance with increasing depth typical of other architectures. Well-known pathologies, such as oversmoothing, oversquashing, bottlenecks, and exploding/vanishing gradients yield deep GNNs which are either unstable or lose performance as the number of layers increase [5, 6, 7, 8].

To combat this, a number of works build architectures which mimic physical processes to impart desirable numerical properties. For example, some works claim that posing message passing as either a diffusion process or reversible flow may promote stability or help retain information, respectively. These present opposite ends of a spectrum between irreversible and reversible processes, which either dissipate or retain information. It is unclear, however, what role (ir)reversibility plays [9]. One could argue that dissipation entropically destroys information and could promote oversmoothing, so should be avoided. Alternatively, in dynamical systems theory, dissipation is crucial to realize a low-dimensional attractor, and thus dissipation may play an important role in realizing dimensionality reduction. Moreover, recent work has shown that dissipative phenomena can actually sharpen information as well as smooth it [10], although this is not often noticed in practice since typical empirical tricks (batch norm, etc.) lead to a departure from the governing mathematical theory.

In physics, Poisson brackets and their metriplectic/port-Hamiltonian generalization to dissipative systems provide an abstract framework for studying conservation and entropy production in dynamical systems. In this work, we construct four novel architectures which span the (ir)reversibility spectrum, using geometric brackets as a means of parameterizing dynamics abstractly without empirically assuming a physical model. This relies on an application of the data-driven exterior calculus (DDEC) [11], which allows a reinterpretation of the message-passing and aggregation of graph attention networks [12] as the fluxes and conservation balances of physics simulators [13], providing a simple but powerful framework for mathematical analysis. In this context, we recast graph attention as an inner-product on graph features, inducing graph derivative 'building-blocks' which may be used to build geometric brackets. In the process, we generalize classical graph attention [12] to higher-order clique cochains (e.g., labels on edges and loops). The four architectures proposed here scale with identical complexity to classical graph attention networks, and possess desirable properties that have proven elusive in current architectures. On the reversible and irreversible end of the spectrum we have Hamiltonian and Gradient networks. In the middle of the spectrum, Double Bracket and Metriplectic architectures combine both reversibility and irreversibility, dissipating energy to either the environment or an entropic variable, respectively, in a manner consistent with the second law of thermodynamics. We summarize these brackets in Table 1, providing a diagram of their architecture in Figure 1.

Primary contributions:

Theoretical analysis of GAT in terms of exterior calculus. Using DDEC we establish a unified framework for construction and analysis of message-passing graph attention networks, and provide an extensive introductory primer to the theory in the appendices. In this setting, we show that with our modified attention mechanism, GATs amount to a diffusion process for a special choice of activation and weights.

Generalized attention mechanism. Within this framework, we obtain a natural and flexible extension of graph attention from nodal features to higher order cliques (e.g. edge features). We show attention must have a symmetric numerator to be formally structure-preserving, and introduce a novel and flexible graph attention mechanism parameterized in terms of learnable inner products on nodes and edges.

Novel structure-preserving extensions. We develop four GNN architectures based upon bracketbased dynamical systems. In the metriplectic case, we obtain the first architecture with linear complexity in the size of the graph while previous works are O ( N 3 ) .

Unified evaluation of dissipation. We use these architectures to systematically evaluate the role of (ir)reversibility in the performance of deep GNNs. We observe best performance for partially

dissipative systems, indicating that a combination of both reversibility and irreversibility are important. Pure diffusion is the least performant across all benchmarks. For physics-based problems including optimal control, there is a distinct improvement. All models provide near state-of-the-art performance and marked improvements over black-box GAT/NODE networks.

2 Previous works

Neural ODEs: Many works use neural networks to fit dynamics of the form ˙ x = f ( x, θ ) to time series data. Model calibration (e.g., UDE [14]), dictionary-based learning (e.g., SINDy [15]), and neural ordinary differential equations (e.g., NODE [16]) pose a spectrum of inductive biases requiring progressively less domain expertise. Structure-preservation provides a means of obtaining stable training without requiring domain knowledge, ideally achieving the flexibility of NODE with the robustness of UDE/SINDy. The current work learns dynamics on a graph while using a modern NODE library to exploit the improved accuracy of high-order integrators [17, 18, 19].

Structure-preserving dense networks: For dense networks, it is relatively straightforward to parameterize reversible dynamics, see for example: Hamiltonian neural networks [20, 21, 22, 23], Hamiltonian generative networks [24], Hamiltonian with Control (SymODEN) [25], Deep Lagrangian networks [26] and Lagrangian neural networks [27]. Structure-preserving extensions to dissipative systems are more challenging, particularly for metriplectic dynamics [28] which require a delicate degeneracy condition to preserve discrete notions of the first and second laws of thermodynamics. For dense networks such constructions are intensive, suffering from O ( N 3 ) complexity in the number of features [29, 30, 31]. In the graph setting we avoid this and achieve linear complexity by exploiting exact sequence structure. Alternative dissipative frameworks include Dissipative SymODEN [32] and port-Hamiltonian [33]. We choose to focus on metriplectic parameterizations due to their broad potential impact in data-driven physics modeling, and ability to naturally treat fluctuations in multiscale systems [34].

Physics-informed vs structure-preserving: "Physics-informed" learning imposes physics by penalty, adding a regularizer corresponding to a physics residual. The technique is simple to implement and has been successfully applied to solve a range of PDEs [35], discover data-driven models to complement first-principles simulators [36, 37, 38], learn metriplectic dynamics [39], and perform uncertainty quantification [40, 41]. Penalization poses a multiobjective optimization problem, however, with parameters weighting competing objectives inducing pathologies during training, often resulting in physics being imposed to a coarse tolerance and qualitatively poor predictions [42, 43]. In contrast, structure-preserving architectures exactly impose physics by construction via carefully designed networks. Several works have shown that penalty-based approaches suffer in comparison, with structure-preservation providing improved long term stability, extrapolation and physical realizability.

Structure-preserving graph networks: Several works use discretizations of specific PDEs to combat oversmoothing or exploding/vanishing gradients, e.g. telegraph equations [44] or various reaction-diffusion systems [45]. Several works develop Hamiltonian flows on graphs [46, 47]. For metriplectic dynamics, [48] poses a penalty based formulation on graphs. We particularly focus on GRAND, which poses graph learning as a diffusive process [49], using a similar exterior calculus framework and interpreting attention as a diffusion coefficient. We show in Appendix A.5 that their

Attention

Encoder

Energy/Entropy

Gradients

Graph derivatives

Metric adjoints

Bracket matrices

Decoder

Input/Output

Trainable

Calculated

Integrator

analysis fails to account for the asymmetry in the attention mechanism, leading to a departure from the governing theory. To account for this, we introduce a modified attention mechanism which retains interpretation as a part of diffusion PDE. In this purely irreversible case, it is of interest whether adherence to the theory provides improved results, or GRAND's success is driven by something other than structure-preservation.

3 Theory and fundamentals

Here we introduce the two essential ingredients to our approach: bracket-based dynamical systems for neural differential equations, and the data-driven exterior calculus which enables their construction. A thorough introduction to this material is provided in Appendices A.1, A.3, and A.2.

Bracket-based dynamics: Originally introduced as an extension of Hamiltonian/Lagrangian dynamics to include dissipation [50], bracket formulations are used to inform a dynamical system with certain structural properties, e.g., time-reversibility, invariant differential forms, or property preservation. Even without dissipation, bracket formulations may compactly describe dynamics while preserving core mathematical properties, making them ideal for designing neural architectures.

Bracket formulations are usually specified via some combination of reversible brackets { F, G } = ⟨∇ F, L ∇ G ⟩ and irreversible brackets [ F, G ] = ⟨∇ F, M ∇ G ⟩ , {{ F, G }} = 〈 ∇ F, L 2 ∇ G 〉 for potentially state-dependent operators L ∗ = -L and M ∗ = M . The particular brackets which are used in the present network architectures are summarized in Table 1. Note that complete systems are the dynamical extensions of isolated thermodynamical systems: they conserve energy and produce entropy, with nothing lost to the ambient environment. Conversely, incomplete systems do not account for any lost energy: they only require that it vanish in a prescribed way. The choice of completeness is an application-dependent modeling assumption.

Exterior calculus: In the combinatorial Hodge theory [51], an oriented graph G = {V , E} carries sets of k -cliques, denoted G k , which are collections of ordered subgraphs generated by ( k +1) nodes. This induces natural exterior derivative operators d k : Ω k → Ω k +1 , acting on the spaces of functions on G k , which are the signed incidence matrices between k -cliques and ( k +1) -cliques. An explicit representation of these derivatives is given in Appendix A.1, from which it is easy to check the exact sequence property d k +1 ◦ d k = 0 for any k . This yields a discrete de Rham complex on the graph G (Figure 2). Moreover, given a choice of inner product (say, ℓ 2 ) on Ω k , there is an obvious dual de Rham complex which comes directly from adjointness. In particular, one can define dual derivatives d ∗ k : Ω k +1 → Ω k via the equality

from which nontrivial results such as the Hodge decomposition, Poincaré inequality, and coercivity/invertibility of the Hodge Laplacian ∆ k = d ∗ k d k + d k -1 d ∗ k -1 follow (see e.g. [11]). Using the derivatives d k , d ∗ k , it is possible to build compatible discretizations of PDEs on G which are guaranteed to preserve exactness properties such as, e.g., d 1 ◦ d 0 = curl ◦ grad = 0 .

The choice of inner product ⟨· , ·⟩ k thus induces a definition of the dual derivatives d ∗ k . In the graph setting [52], one typically selects the ℓ 2 inner product, obtaining the adjoints of the signed incidence matrices as d ∗ k = d ⊺ k . By instead working with the modified inner product ( v , w ) = v ⊺ A k w for a machine-learnable A k , we obtain d ∗ k = A -1 k d ⊺ k A k +1 (see Appendix A.3). This parameterization

inherits the exact sequence property from the graph topology encoded in d k while allowing for incorporation of geometric information from data. This leads directly to the following result, which holds for any (potentially feature-dependent) symmetric positive definite matrix A k .

Theorem 3.1. The dual derivatives d ∗ k : Ω k +1 → Ω k adjoint to d k : Ω k → Ω k +1 with respect to the learnable inner products A k : Ω k → Ω k satisfy an exact sequence property.

As will be shown in Section 4, by encoding graph attention into the A k , we may exploit the exact sequence property to obtain symmetric positive definite diffusion operators, as well as conduct the cancellations necessary to enforce degeneracy conditions necessary for metriplectic dynamics.

For a thorough review of DDEC, we direct readers to Appendix A.1 and [11]. For exterior calculus in topological data analysis see [52], and an overview in the context of PDEs see [53, 54].

4 Structure-preserving bracket parameterizations

We next summarize properties of the bracket dynamics introduced in Section 3 and displayed in Table 1, postponing details and rigorous discussion to Appendices A.3 and A.6. Letting x = ( q , p ) denote node-edge feature pairs, the following operators will be used to generate our brackets.

As mentioned before, the inner products A 0 , A 1 , A 2 on Ω k which induce the dual derivatives d ∗ 0 , d ∗ 1 , are chosen in such a way that their combination generalizes a graph attention mechanism. The precise details of this construction are given below, and its relationship to the standard GAT network from [12] is shown in Appendix A.5. Notice that L ∗ = -L , while G ∗ = G , M ∗ = M are positive semidefinite with respect to the block-diagonal inner product ( · , · ) defined by A = diag ( A 0 , A 1 ) (details are provided in Appendix A.6). Therefore, L generates purely reversible (Hamiltonian) dynamics and G , M generate irreversible (dissipative) ones. Additionally, note that state-dependence in L , M , G enters only through the adjoint differential operators, meaning that any structural properties induced by the topology of the graph G (such as the exact sequence property mentioned in Theorem 3.1) are automatically preserved.

Remark 4.1. Strictly speaking, L is guaranteed to be a truly Hamiltonian system only when d ∗ 0 is state-independent, since it may otherwise fail to satisfy Jacobi's identity. On the other hand, energy conservation is always guaranteed due to the fact that L is skew-adjoint.

In addition to the bracket matrices L , M , G , it is necessary to have access to energy and entropy functions E,S and their associated functional derivatives with respect to the inner product on Ω 0 ⊕ Ω 1 defined by A . For the Hamiltonian, gradient, and double brackets, E is chosen simply as the 'total kinetic energy'

whose A -gradient (computed in Appendix A.6) is just ∇ E ( q , p ) = ( A -1 0 q A -1 1 p ) ⊺ . Since the metriplectic bracket uses parameterizations of E,S which are more involved, discussion of this case is deferred to later in this Section.

Attention as learnable inner product: Before describing the dynamics, it remains to discuss how the matrices A i , 0 ≤ i ≤ 2 , are computed in practice, and how they relate to the idea of graph attention. Recall that if n V > 0 denotes the nodal feature dimension, a graph attention mechanism takes the form a ( q i , q j ) = f (˜ a ij ) / ∑ j f (˜ a ij ) for some differentiable pre-attention function ˜ a : n V × n V → R (e.g., for scaled dot product [55]) one typically represents a ( q i , q j ) as a softmax, so that f = exp( q ) ). This suggests a decomposition a ( q i , q j ) = A -1 0 A 1 where A 0 = ( a 0 ,ii ) is diagonal on nodes and A 1 = ( a 1 ,ij ) is diagonal on edges,

Treating the numerator and denominator of the standard attention mechanism separately in A 0 , A 1 allows for a flexible and theoretically sound incorporation of graph attention directly into the adjoint differential operators on G . In particular, if A 1 is symmetric with respect to edge-orientation and p is an edge feature which is antisymmetric, it follows that

which is just graph attention combined with edge aggregation. This makes it possible to give the following informal statement regarding graph attention networks which is explained and proven in Appendix A.5.

Remark 4.2. The GAT layer from [12] is almost the forward Euler discretization of a metric heat equation.

The 'almost' appearing here has to do with the fact that (1) the attentional numerator f (˜ a ( q i , q j )) is generally asymmetric in i, j , and is therefore symmetrized by the divergence operator d ⊺ 0 , (2) the activation function between layers is not included, and (3) learnable weight matrices W k in GAT are set to the identity.

Remark 4.3. The interpretation of graph attention as a combination of learnable inner products admits a direct generalization to higher-order cliques, which is discussed in Appendix A.4.

Hamiltonian case: A purely conservative system is generated by solving ˙ x = L ( x ) ∇ E ( x ) , or

This is a noncanonical Hamiltonian system which generates a purely reversible flow. In particular, it can be shown that

so that energy is conserved due to the skew-adjointness of L .

Gradient case: On the opposite end of the spectrum are generalized gradient flows, which are totally dissipative. Consider solving ˙ x = -G ( x ) ∇ E ( x ) , or

This system is a metric diffusion process on nodes and edges separately. Moreover, it corresponds to a generalized gradient flow, since

due to the self-adjoint and positive semi-definite nature of G .

Remark 4.4. The architecture in GRAND [49] is almost a gradient flow, however the pre-attention mechanism lacks the requisite symmetry to formally induce a valid inner product.

Double bracket case: Another useful formulation for incomplete systems is the so-called doublebracket formalism. Consider solving ˙ x = L ∇ E + L 2 ∇ E , or

This provides a dissipative relationship which preserves the Casimirs of the Poisson bracket generated by L , since L ∇ C = 0 implies L 2 ∇ C = 0 . In particular, it follows that

since L is skew-adjoint and therefore L 2 is self-adjoint.

Remark 4.5. It is interesting to note that the matrix L is essentially a Dirac operator (square root of the Hodge Laplacian ∆ = ( d + d ∗ ) 2 ) restricted to cliques of degree at most 1. However, here L 2 = -∆ , so that L is in some sense 'pure imaginary'.

Metriplectic case: Metriplectic systems are expressible as ˙ x = L ∇ E + M ∇ S where E,S are energy resp. entropy functions which satisfy the degeneracy conditions L ∇ S = M ∇ E = 0 . One way of setting this up in the present case is to define the energy and entropy functions

where s is sum aggregation over nodes resp. edges, f E : R n V → R acts on node features, and g E , g S : R n E → R act on edge features. Denoting the 'all ones' vector (of variable length) by 1 , it is shown in Appendix A.6 that the A -gradients of energy and entropy can be computed as

Similarly, it is shown in Appendix A.6 that the degeneracy conditions L ∇ S = M ∇ E = 0 are satisfied by construction. Therefore, the governing dynamical system becomes

With this, it follows that the system obeys a version of the first and second laws of thermodynamics,

Remark 4.6. As seen in the increased complexity of this formulation, enforcing the degeneracy conditions necessary for metriplectic structure is nontrivial. This is accomplished presently via an application of the exact sequence property in Theorem 3.1, which we derive in Appendix A.6.

Remark 4.7. It is worth mentioning that, similar to the other architectures presented in this Section, the metriplectic network proposed here exhibits linear O ( N ) scaling in the graph size. This is in notable contrast to [29, 31] which scale as O ( N 3 ) .

5 Experiments

This section reports results on experiments designed to probe the influence of bracket structure on trajectory prediction and nodal feature classification. Additional experimental details can be found in Appendix B. In each Table, orange indicates the best result by our models, and blue indicates the best of those compared. We consider both physical systems, where the role of structure preservation is explicit, as well as graph-analytic problems.

5.1 Damped double pendulum

As a first experiment, consider applying one of these architectures to reproduce the trajectory of a double pendulum with a damping force proportional to the angular momenta of the pendulum masses (see Appendix B.1 for details).

Since this system is metriplectic when expressed in position-momentum-entropy coordinates (c.f. [56]), it is useful to see if any of the brackets from Section 4 can adequately capture these dynamics without an entropic variable. The results of applying the architectures of Section 4 to reproduce a trajectory of five periods are displayed in Table 2, alongside comparisons with a black-box NODE network and a latent NODE with feature encoder/decoder. While each network is capable of producing a small mean absolute error, it is clear that the metriplectic and Hamiltonian networks produce the most accurate trajectories. It is remarkable both that the Hamiltonian bracket does so well here and that the gradient bracket does so poorly, being that the damped double pendulum system is quite dissipative. On the other hand, it is unlikely to be only the feature encoder/decoder leading to good performance here, as both the NODE and NODE+AE architectures perform worse on this task by about one order of magnitude.

5.2 MuJoCo Dynamics

Next we test the proposed models on more complex physical systems that are generated by the Multi-Joint dynamics with Contact (MuJoCo) physics simulator [57]. We consider the modified versions of Open AI Gym environments [23]: HalfCheetah, Hopper, and Swimmer.

We represent an object in an environment as a fully-connected graph, where a node corresponds to a body part of the object and, thus, the nodal feature q i corresponds to a position of a body part or an angle of a joint. 3 As the edge features, a pair of nodal velocities p α = ( v src ( α ) , v dst ( α ) ) are provided, where v src ( α ) and v dst ( α ) denote velocities of the source and destination nodes connected to the edge.

Since the MuJoCo environments contain an actor applying controls, additional control input is accounted for with an additive forcing term which is parameterized by a multi-layer perceptron and introduced into the bracket-based dynamics models. See Appendix B.2 for additional experimental details. The problem therefore consists of finding an optimal control MLP, and we evaluate the improvement which comes from representing the physics surrogate with bracket dynamics over NODE.

All models are trained via minimizing the MSE between the predicted positions ˜ q and the ground truth positions q and are tested on an unseen test set. Table 3 reports the errors of network predictions on the test set measured in the relative ℓ 2 norm, ∥ q -˜ q ∥ 2 / ∥ q ∥ 2 ∥ ˜ q ∥ 2 . Similar to the double pendulum experiments, all models are able to produce accurate predictions with around or less than 10% errors. While the gradient bracket makes little to no improvements over NODEs, the Hamiltonian, double, and metriplectic brackets produce more accurate predictions. Interestingly, the Hamiltonian bracket performs the best in this case as well, meaning that any dissipation present is effectively compensated for by the autoencoder which transforms the features.

5.3 Node classification

Moving beyond physics-based examples, it remains to see how bracket-based architectures perform on 'black-box' node classification problems. Table 4 and Table 5 present results on common benchmark problems including the citation networks Cora [58], Citeseer [59], and Pubmed [60], as well as the coauthor graph, CoauthorCS [61], and the Amazon co-purchasing graphs, Computer and Photo [62]. For comparison, we report results on a standard GAT [12], a neural graph differential equation

architecture (GDE) [63], and the nonlinear GRAND architecture (GRAND-nl) from [49] which is closest to ours. Since our experimental setting is similar to that of [49], the numbers reported for GAT, GDE, and GRAND-nl are taken directly from this paper. Note that, despite the similar O ( N ) scaling in the metriplectic architecture, the high dimension of the node and edge features on the latter three datasets led to trainable E,S functions which exhausted the memory on our available machines, and therefore results are not reported for these cases. A full description of experimental details is provided in Appendix B.3.

Remark 5.1. To highlight the effect of bracket structure on network performance, only minimal modifications are employed during network training. In particular, we do not include any additional regularization, positional encoding, graph rewiring, extraction of connected components, extra terms on the right-hand side, or early stopping. While it is likely that better classification performance could be achieved with some of these modifications included, it becomes very difficult to isolate the effect of structure-preservation. A complete list of tunable hyperparameters is given in Appendix B.3.

The results show different behavior produced by each bracket architecture. It is empirically clear that there is some value in full or partial reversibility, since the Hamiltonian and double bracket architectures both perform better than the corresponding gradient architecture on datasets such as Computer and Photo. Moreover, it appears that the partially reversible double bracket performs the best of the bracket architectures in every case, which is consistent with the idea that both reversible and irreversible dynamics are critical for capturing the behavior of general dynamical systems. Interestingly, the metriplectic bracket performs worse on these tasks by a large margin. We conjecture this architecture may be harder to train for larger problems despite its O ( N ) complexity in the graph size, suggesting that more sophisticated training strategies may be required for large problems.

6 Conclusion

This work presents a unified theoretical framework for analysis and construction of graph attention networks. The exact sequence property of graph derivatives and coercivity of Hodge Laplacians which follow from the theory allow the construction of four structure-preserving brackets, which we use to evaluate the role of irreversibility in both data-driven physics simulators and graph analytics problems. In all contexts, the pure diffusion bracket performed most poorly, with mixed results between purely reversible and partially dissipative brackets.

The linear scaling achieved by the metriplectic brackets has a potential major impact for data-driven physics modeling. Metriplectic systems emerge naturally when coarse-graining multiscale systems. With increasing interest in using ML to construct digital twins, fast data-driven surrogates for complex multi-physics acting over multiple scales will become crucial. In this setting the stability encoded by metriplectic dynamics translates to robust surrogates, with linear complexity suggesting the possibility of scaling up to millions of degrees of freedom.

Limitations: All analysis holds under the assumption of modified attention mechanisms which allow interpretation of GAT networks as diffusion processes; readers should take care that the analysis is for a non-standard attention. Secondly, for all brackets we did not introduce empirical modifications (e.g. regularization, forcing, etc) to optimize performance so that we could study the role of (ir)reversibility in isolation. With this in mind, one may be able to add 'tricks' to e.g. obtain a diffusion architecture which outperforms those presented here. Finally, note that the use of a feature autoencoder in the bracket architectures means that structure is enforced in the transformed space. This allows for applicability to more general systems, and can be easily removed when appropriate features are known.

Broader impacts: The work performed here is strictly foundational mathematics and is intended to improve the performance of GNNs in the context of graph analysis and data-driven physics modeling. Subsequent application of the theory may have societal impact, but the current work anticipated to improve the performance of machine learning in graph settings only at a foundational level.

References

Marco Gori, Gabriele Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. , volume 2, pages 729-734. IEEE, 2005.

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open , 1:57-81, 2020.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv:1709.05584 , 2017.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 3438-3445, 2020.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. Advances in neural information processing systems , 33:4917-4928, 2020.

Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318 , 2020.

Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Dissecting the diffusion process in linear graph convolutional networks. Advances in Neural Information Processing Systems , 34:5758-5769, 2021.

Francesco Di Giovanni, James Rowbottom, Benjamin P. Chamberlain, Thomas Markovich, and Michael M. Bronstein. Understanding convolution on graphs via energies, 2023.

Nathaniel Trask, Andy Huang, and Xiaozhe Hu. Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs. arXiv preprint arXiv:2012.11799 , 2020.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Khemraj Shukla, Mengjia Xu, Nathaniel Trask, and George E Karniadakis. Scalable algorithms for physics-informed neural and graph networks. Data-Centric Engineering , 3:e24, 2022.

Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385 , 2020.

Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences , 113(15):3932-3937, 2016.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages 6572-6583, 2018.

Michael Poli, Stefano Massaroli, Atsushi Yamashita, Hajime Asama, Jinkyoo Park, and Stefano Ermon. Torchdyn: Implicit models and neural numerical methods in pytorch.

Louis-Pascal Xhonneux, Meng Qu, and Jian Tang. Continuous graph neural networks. In International Conference on Machine Learning , pages 10432-10441. PMLR, 2020.

Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent El Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems , 33:11984-11995, 2020.

Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Marc Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying hamiltonian and lagrangian neural networks via explicit constraints. Advances in neural information processing systems , 33:13880-13889, 2020.

Renyi Chen and Molei Tao. Data-driven prediction of general hamiltonian dynamics via learning exactlysymplectic maps. In International Conference on Machine Learning , pages 1717-1727. PMLR, 2021.

Nate Gruver, Marc Anton Finzi, Samuel Don Stanton, and Andrew Gordon Wilson. Deconstructing the inductive biases of hamiltonian neural networks. In International Conference on Learning Representations .

Peter Toth, Danilo J Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Representations , 2019.

Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-Net: Learning Hamiltonian dynamics with control. In International Conference on Learning Representations .

Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations , 2018.

Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations , 2020.

Partha Guha. Metriplectic structure, leibniz dynamics and dissipative systems. Journal of Mathematical Analysis and Applications , 326(1):121-136, 2007.

Kookjin Lee, Nathaniel Trask, and Panos Stinis. Machine learning structure preserving brackets for forecasting irreversible processes. Advances in Neural Information Processing Systems , 34:5696-5707, 2021.

Kookjin Lee, Nathaniel Trask, and Panos Stinis. Structure-preserving sparse identification of nonlinear dynamics for data-driven modeling. In Mathematical and Scientific Machine Learning , pages 65-80. PMLR, 2022.

Zhen Zhang, Yeonjong Shin, and George Em Karniadakis. Gfinns: Generic formalism informed neural networks for deterministic and stochastic dynamical systems. Philosophical Transactions of the Royal Society A , 380(2229):20210207, 2022.

Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Dissipative symoden: Encoding hamiltonian dynamics with dissipation and control into deep learning. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations .

Shaan A Desai, Marios Mattheakis, David Sondak, Pavlos Protopapas, and Stephen J Roberts. Porthamiltonian neural networks for learning explicit time-dependent dynamical systems. Physical Review E , 104(3):034312, 2021.

Miroslav Grmela. Generic guide to the multiscale dynamics and thermodynamics. Journal of Physics Communications , 2(3):032001, 2018.

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics , 357:125-141, 2018.

Filippo Masi and Ioannis Stefanou. Multiscale modeling of inelastic materials with thermodynamics-based artificial neural networks (tann). Computer Methods in Applied Mechanics and Engineering , 398:115190, 2022.

Ravi G Patel, Indu Manickam, Nathaniel A Trask, Mitchell A Wood, Myoungkyu Lee, Ignacio Tomas, and Eric C Cyr. Thermodynamically consistent physics-informed neural networks for hyperbolic systems. Journal of Computational Physics , 449:110754, 2022.

Quercus Hernández, Alberto Badías, David González, Francisco Chinesta, and Elías Cueto. Structurepreserving neural networks. Journal of Computational Physics , 426:109950, 2021.

Yibo Yang and Paris Perdikaris. Adversarial uncertainty quantification in physics-informed neural networks. Journal of Computational Physics , 394:136-152, 2019.

Dongkun Zhang, Lu Lu, Ling Guo, and George Em Karniadakis. Quantifying total uncertainty in physicsinformed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics , 397:108850, 2019.

Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing , 43(5):A3055-A3081, 2021.

Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics , 449:110768, 2022.

T Konstantin Rusch, Ben Chamberlain, James Rowbottom, Siddhartha Mishra, and Michael Bronstein. Graph-coupled oscillator networks. In International Conference on Machine Learning , pages 18888-18909. PMLR, 2022.

Jeongwhan Choi, Seoyoung Hong, Noseong Park, and Sung-Bae Cho. Gread: Graph neural reactiondiffusion equations. arXiv preprint arXiv:2211.14208 , 2022.

Alvaro Sanchez-Gonzalez, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint arXiv:1909.12790 , 2019.

Suresh Bishnoi, Ravinder Bhattoo, Jayadeva Jayadeva, Sayan Ranu, and NM Anoop Krishnan. Enhancing the inductive biases of graph neural ode for modeling physical systems. In The Eleventh International Conference on Learning Representations .

Quercus Hernandez, Alberto Badias, Francisco Chinesta, and Elias Cueto. Thermodynamics-informed graph neural networks. IEEE Transactions on Artificial Intelligence , 1(01):1-1, 2022.

Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learning , pages 1407-1418. PMLR, 2021.

PJ Morrison. Thoughts on brackets and dissipation: old and new. In Journal of Physics: Conference Series , volume 169, page 012006. IOP Publishing, 2009.

Oliver Knill. The dirac operator of a graph, 2013.

Xiaoye Jiang, Lek-Heng Lim, Yuan Yao, and Yinyu Ye. Statistical ranking and combinatorial hodge theory. Mathematical Programming , 127(1):203-244, 2011.

Pavel B Bochev and James M Hyman. Principles of mimetic discretizations of differential operators. In Compatible spatial discretizations , pages 89-119. Springer, 2006.

Douglas N Arnold. Finite element exterior calculus . SIAM, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Ignacio Romero. Thermodynamically consistent time-stepping algorithms for non-linear thermomechanical systems. International Journal for Numerical Methods in Engineering , 79(6):706-732, 2023/05/14 2009.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval , 3:127-163, 2000.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Galileo Namata, Ben London, Lise Getoor, Bert Huang, and U Edu. Query-driven active surveying for collective classification. In 10th international workshop on mining and learning with graphs , volume 8, page 1, 2012.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. ArXiv , abs/1811.05868, 2018.

Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval , pages 43-52, 2015.

Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07532 , 2019.

Xiaoye Jiang, Lek-Heng Lim, Yuan Yao, and Yinyu Ye. Statistical ranking and combinatorial hodge theory. Mathematical Programming , 127(1):203-244, nov 2010.

Anthony Bloch, P. S. Krishnaprasad, Jerrold E. Marsden, and Tudor S. Ratiu. The euler-poincaré equations and double bracket dissipation. Communications in Mathematical Physics , 175(1):1-42, 1996.

Darryl D Holm, Jerrold E Marsden, and Tudor S Ratiu. The euler-poincaré equations and semidirect products with applications to continuum theories. Advances in Mathematics , 137(1):1-81, 1998.

Hans Christian Oettinger. Irreversible dynamics, onsager-casimir symmetry, and an application to turbulence. Physical Review E , 90(4):042121, 2014.

Anthony Gruber, Max Gunzburger, Lili Ju, and Zhu Wang. Energetically consistent model reduction for metriplectic systems. Computer Methods in Applied Mechanics and Engineering , 404:115709, 2023.

Robert J. Renka. A simple explanation of the sobolev gradient method. 2006.

Chris Yu, Henrik Schumacher, and Keenan Crane. Repulsive curves. ACM Trans. Graph. , 40(2), may 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Joe Chen. Chaos from simplicity: an introduction to the double pendulum. Technical report, University of Canterbury, 2008.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Romeo Ortega, Arjan Van Der Schaft, Bernhard Maschke, and Gerardo Escobar. Interconnection and damping assignment passivity-based control of port-controlled hamiltonian systems. Automatica , 38(4):585596, 2002.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540 , 2016.

Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.

Glossary of Notation and Symbols

The next list describes several symbols that will be later used within the body of the document

{{· , ·}} (Irreversible) double bracket on functions with generator L 2

[ · , · ] Degenerate (irreversible) metric bracket on functions with generator M ⊺ = M

{· , ·} Poisson (reversible) bracket on functions with generator L ⊺ = -L

N ( i ) , N ( i ) Neighbors of node i ∈ V , neighbors of node i ∈ V including i

[ S ] Indicator function of the statement S

δf, ∇ f Adjoint of df with respect to ⟨· , ·⟩ , adjoint of df with respect to ( · , · )

∆ k Hodge Laplacian d k d ∗ k + d ∗ k d k

δ ij Kronecker delta

˙ f Derivative of f with respect to time

( · , · ) k Learnable metric inner product on k -cliques with matrix representation A k

⟨· , ·⟩ k Euclidean ℓ 2 inner product on k -cliques

G , V , E Oriented graph, set of nodes, set of edges

G k , Ω k Set of k -cliques, vector space of real-valued functions on k -cliques

d, d k Exterior derivative operator on functions, exterior derivative operator on k -cliques

d ⊺ k , d ∗ k Adjoint of d k with respect to ⟨· , ·⟩ k , adjoint of d k with respect to ( · , · ) k

A Mathematical foundations

This Appendix provides the following: (1) an introduction to the ideas of graph exterior calculus, A.1, and bracket-based dynamical systems, A.2, necessary for understanding the results in the body, (2) additional explanation regarding adjoints with respect to generic inner products and associated computations, A.3, (3) a mechanism for higher-order attention expressed in terms of learnable inner products, A.4, (4) a discussion of GATs in the context of exterior calculus, A.5, and (5) proofs which are deferred from Section 4, A.6.

A.1 Graph exterior calculus

Here some basic notions from the graph exterior calculus are recalled. More details can be found in, e.g., [11, 51, 64].

1

2

3

4

5

6

1

2

3

4

5

6

1

As mentioned in Section 3, an oriented graph G = {V , E} carries sets of k -cliques , denoted G k , which are collections of ordered subgraphs generated by ( k +1) nodes. For example, the graph in Figure 3 contains six 0-cliques (nodes), six 1-cliques (edges), and one 2-clique. A notion of combinatorial derivative is then given by the signed incidence matrices d k : Ω k → Ω k +1 , operating on the space Ω k of differentiable functions on k -cliques, whose entries ( d k ) ij are 1 or -1 if the j th k -clique is

incident on the i th ( k +1) -clique, and zero otherwise. For the example in Figure 3, these are:

Remark A.1. While the one-hop neighborhood of node i in G , denoted N ( i ) , does not include node i itself, many machine learning algorithms employ the extended neighborhood N ( i ) = N ( i ) ∪ { i } . Since this is equivalent to considering the one-hop neighborhood of node i in the self-looped graph G , this modification does not change the analysis of functions on graphs.

It can be shown that the action of these matrices can be conveniently expressed in terms of totally antisymmetric functions f ∈ Ω k , via the expression

where ( i 0 , ..., i k +1 ) denotes a ( k +1) -clique of vertices v ∈ V . As convenient shorthand, we often write subscripts, e.g., ( d k f ) i 0 i 1 ...i k +1 , instead of explicit function arguments. Using [ S ] to denote the indicator function of the statement S , it is straightforward to check that d ◦ d = 0 ,

since ( -1) j + l -1 = ( -1) -1 ( -1) j + l = ( -1)( -1) j + l and the final sum follows from swapping the labels j, l . This shows that the k -cliques on G form a de Rham complex [53]: a collection of function spaces Ω k equipped with mappings d k satisfying Im d k -1 ⊂ Ker d k as shown in Figure 4. When

Figure 4: Illustration of the de Rham complex on G induced by the combinatorial derivatives, where K > 0 is the maximal clique degree.

K = 3 , this is precisely the graph calculus analogue of the de Rham complex on R 3 formed by the Sobolev spaces H 1 , H (curl) , H (div) , L 2 which satisfies div ◦ curl = curl ◦ grad = 0 .

While the construction of the graph derivatives and their associated de Rham complex is purely topological, building elliptic differential operators such as the Laplacian relies on a dual de Rham complex, which is specified by an inner product on Ω k . In the case of ℓ 2 , this leads to dual derivatives which are the matrix transposes of the d k having the following explicit expression.

Proposition A.1. The dual derivatives d ⊺ k : Ω k +1 → Ω k adjoint to d k through the ℓ 2 inner product are given by

where [ i j , ..., i k +1 ] = i k +1 , i j , ..., i k indicates a cyclic permutation forward by one index.

Proof. This is a direct calculation using the representation of d k in terms of antisymmetric functions. More precisely, let an empty sum Σ denote summation over all unspecified indices. Then, for any g ∈ Ω k ,

which establishes the result.

Proposition A.1 is perhaps best illustrated with a concrete example. Consider the graph gradient, defined for edge α = ( i, j ) as ( d 0 f ) α = ( d 0 f ) ij = f j -f i . Notice that this object is antisymmetric with respect to edge orientation, and measures the outflow of information from source to target nodes. From this, it is easy to compute the ℓ 2 -adjoint of d 0 , known as the graph divergence, via

where we have re-indexed under the double sum, used that i ∈ N ( j ) if and only if j ∈ N ( i ) , and used that there are no self-edges in E . Therefore, it follows that the graph divergence at node i is given by

which reduces to the common form ( d ⊺ 0 g ) i = -∑ j g ij if and only if the edge feature g ij is antisymmetric.

Remark A.2. When the inner product on edges E is not L 2 , but defined in terms of a nonnegative, orientation-invariant, and (edge-wise) diagonal weight matrix W = ( w ij ) , a similar computation shows that the divergence becomes

The more general case of arbitrary inner products on V , E is discussed in section A.3.

The differential operators d ⊺ k induce a dual de Rham complex since d ⊺ k -1 d ⊺ k = ( d k d k -1 ) ⊺ = 0 , which enables both the construction of Laplace operators on k -cliques, ∆ k = d ⊺ k d k + d k -1 d ⊺ k -1 , as well as the celebrated Hodge decomposition theorem, stated below. For a proof, see, e.g., [11, Theorem 3.3].

Theorem A.3. (Hodge Decomposition Theorem) The de Rham complexes formed by d k , d ⊺ k induce the following direct sum decomposition of the function space Ω k ,

In the case where the dual derivatives d ∗ k are adjoint with respect to a learnable inner product which does not depend on graph features, the conclusion of Theorem A.3 continues to hold, leading to an interesting well-posedness result proved in [11] involving nonlinear perturbations of a Hodge-Laplace problem in mixed form.

Theorem A.4. ([11, Theorem 3.6]) Suppose f k ∈ Ω k , and g ( x ; ξ ) is a neural network with parameters ξ which is Lipschitz continuous and satisfies g ( 0 ) = 0 . Then, the problem

has a unique solution on Ω k / Ker∆ k .

This result shows that initial-value problems involving the Hodge-Laplacian are stable under nonlinear perturbations. Moreover, when ∆ 0 is the Hodge Laplacian on nodes, there is a useful connection between ∆ 0 and the degree and adjacency matrices of the graph G . Recall that the degree matrix D = ( d ij ) is diagonal with entries d ii = ∑ j ∈N ( i ) 1 , while the adjacency matrix A = ( a ij ) satisfies a ij = 1 when j ∈ N ( i ) and a ij = 0 otherwise.

Proposition A.2. The combinatorial Laplacian on V , denoted ∆ 0 = d ⊺ 0 d 0 , satisfies ∆ 0 = D -A .

Proof. Notice that

̸

where we used that D is diagonal, A is diagonal-free, and ( d 0 ) αi ( d 0 ) αj = -1 whenever α = ( i, j ) is an edge in E , since one of ( d 0 ) αi , ( d 0 ) αj is 1 and the other is -1.

A.2 Bracket-based dynamical systems

Here we mention some additional facts regarding bracket-based dynamical systems. More information can be found in, e.g., [50, 65, 66, 67].

As mentioned before, the goal of bracket formalisms is to extend the Hamiltonian formalism to systems with dissipation. To understand where this originates, consider an action functional A ( q ) = ∫ b a L ( q, ˙ q ) dt on the space of curves q ( t ) , defined in terms of a Lagrangian L on the tangent bundle to some Riemannian manifold. Using L q , L ˙ q to denote partial derivatives with respect to the subscripted variable, it is straightforward to show that, for any compactly supported variation δq of q , we have

where the final equality follows from integration-by-parts and the fact that variational and temporal derivatives commute in this setting. It follows that A is stationary (i.e., d A = 0 ) for all variations only when ∂ t L ˙ q = L q . These are the classical Euler-Lagrange equations which are (under some regularity conditions) transformed to Hamiltonian form via a Legendre transformation,

which defines the Hamiltonian functional H on phase space, and yields the conjugate momentum vector p = L ˙ q . Substituting L = ⟨ p, ˙ q ⟩ -H into the previously derived Euler-Lagrange equations leads immediately to Hamilton's equations for the state x = ( q p ) ⊺ ,

which are an equivalent description of the system in question in terms of the anti-involution J and the functional gradient ∇H .

An advantage of the Hamiltonian description is its compact bracket-based formulation, ˙ x = J ∇H = { x , H} , which requires only the specification of an antisymmetric Poisson bracket {· , ·} and a Hamiltonian functional H . Besides admitting a direct generalization to more complex systems such as Korteweg-de Vries or incompressible Euler, where the involved bracket is state-dependent, this formulation makes the energy conservation property of the system obvious. In particular, it follows immediately from the antisymmetry of {· , ·} that

̸

while it is more difficult to see immediately that the Euler-Lagrange system obeys this same property. The utility and ease-of-use of bracket formulations is what inspired their extension to other systems of interest which do not conserve energy. On the opposite end of this spectrum are the generalized gradient flows, which can be written in terms of a bracket which is purely dissipative. An example of this is heat flow ˙ q = ∆ q := -[ q, D ] , which is the L 2 -gradient flow of Dirichlet energy D ( q ) = (1 / 2) ∫ b a | q ′ | 2 dt (c.f. Appendix A.3). In this case, the functional gradient ∇D = -∂ tt is the negative of the usual Laplace operator, so that the positive-definite bracket [ · , · ] is generated by the identity operator M = id . It is interesting to note that the same system could be expressed using the usual kinetic energy E ( q ) = (1 / 2) ∫ b a | q | 2 dt instead, provided that the corresponding bracket is generated by M = -∆ . This is a good illustration of the flexibility afforded by bracket-based dynamical systems.

Since physical systems are not always purely reversible or irreversible, other useful bracket formalisms have been introduced to capture dynamics which are a mix of these two. The double bracket ˙ x = { x , E } + {{ x , E }} = L ∇ E + L 2 ∇ E is a nice extension of the Hamiltonian bracket particularly because it is Casimir preserving, i.e., those quantities which annihilate the Poisson bracket {· , ·} also annihilate the double bracket. This allows for the incorporation of dissipative phenomena into idealized Hamiltonian systems without affecting desirable properties such as mass conservation, and has been used to model, e.g., the Landau-Lifschitz dissipative mechanism, as well as a mechanism for fluids where energy decays but entrophy is preserved (see [65] for additional discussion). A complementary but alternative point of view is taken by the metriplectic bracket formalism, which requires that any dissipation generated by the system is accounted for within the system itself through the generation of entropy. In the metriplectic formalism, the equations of motion are ˙ x = { x , E } +[ x , S ] = L ∇ E + M ∇ S , along with important and nontrivial compatibility conditions L ∇ S = M ∇ E = 0 , also called degeneracy conditions, which ensure that the reversible and irreversible mechanisms do not cross-contaminate. As shown in the body of the paper, this guarantees that metriplectic systems obey a form of the first and second thermodynamical laws. Practically, the degeneracy conditions enforce a good deal of structure on the operators L , M which has been exploited to generate surrogate models [29, 68, 31]. In particular, it can be shown that the reversible and irreversible brackets can be parameterized in terms of a totally antisymmetric order-3 tensor ξ = ( ξ ijk ) and a partially symmetric order-4 tensor ζ = ( ζ ik,jl ) through the relations (Einstein summation assumed)

Moreover, using the symmetries of ζ , it follows (see [67]) that this tensor decomposes into the product ζ ik,jl = Λ m ik D mn Λ n jl of a symmetric matrix D and an order-3 tensor Λ which is skew-symmetric in its lower indices. Thus, by applying symmetry relationships, it is easy to check that {· , S } = [ · , E ] = 0 .

Remark A.5. In [29], trainable 4- and 3- tensors ξ ijk and ζ ik,jl are constructed to achieve the degeneracy conditions, mandating a costly O ( N 3 ) computational complexity. In the current work we overcome this by instead achieving degeneracy through the exact sequence property.

A.3 Adjoints and gradients

Beyond the basic calculus operations discussed in section A.1 which depend only on graph topology, the network architectures discussed in the body also make extensive use of learnable metric information coming from the nodal features. To understand this, it is useful to recall some information about general inner products and the derivative operators that they induce. First, recall that the usual ℓ 2 inner product on node features a , b ∈ R |V| , ⟨ a , b ⟩ = a ⊺ b , is (in this context) a discretization of the standard L 2 inner product ∫ V ab dµ which aggregates information from across the vertex set V . While this construction is clearly dependent only on the graph structure (i.e., topology), any symmetric positive definite (SPD) matrix A 0 : Ω 0 → Ω 0 also defines an inner product on functions a ∈ Ω 0 through the equality

which gives a different way of measuring the distance between a and b . The advantage of this construction is that A 0 can be chosen in a way that incorporates geometric information which implicitly regularizes systems obeying a variational principle. This follows from the following

intuitive fact: the Taylor series of a function does not change, regardless of the inner product on its domain. For any differentiable function(al) E : Ω 0 → R , using d to denote the exterior derivative, this means that the following equality holds

where δE denotes the ℓ 2 -gradient of E and ∇ E denotes its A 0 -gradient, i.e., its gradient with respect to the derivative operator induced by the inner product involving A 0 . From this, it is clear that δE = A 0 ∇ E , so that the A 0 -gradient is just an anisotropic rescaling of the ℓ 2 version. The advantage of working with ∇ over δ in the present case of graph networks is that A 0 can be learned based on the features of the graph. This means that learnable feature information (i.e., graph attention) can be directly incorporated into the differential operators governing our bracket-based dynamical systems by construction.

The prototypical example of where this technique is useful is seen in the gradient flow of Dirichlet energy. Recall that the Dirichlet energy of a differentiable function u : R n → R is given by D ( u ) = (1 / 2) ∫ |∇ u | 2 dµ , where ∇ now denotes the usual ℓ 2 -gradient of the function u on R n . Using integration-by-parts, it is easy to see that d D ( u ) v = -∫ v ∆ u for any test function v with compact support, implying that the L 2 -gradient of D is -∆ and ˙ u = ∆ u is the L 2 -gradient flow of Dirichlet energy: the motion which decreases the quantity D ( u ) the fastest as measured by the L 2 norm . It can be shown that high-frequency modes decay quickly under this flow, while low-frequency information takes much longer to dissipate. On the other hand, we could alternatively run the H 1 -gradient flow of D , which is motion of fastest decrease with respect to the H 1 inner product ( u, v ) = ∫ ⟨∇ u, ∇ v ⟩ dµ . This motion is prescribed in terms of the H 1 -gradient of D , which by the discussion above with A 0 = -∆ is easily seen to be the identity. This means that the H 1 -gradient flow is given by ˙ u = -u , which retains the minimizers of the L 2 -flow but with quite different intermediate character, since it functions by simultaneously flattening all spatial frequencies. The process of preconditioning a gradient flow by matching derivatives is known as a Sobolev gradient method (c.f. [69]), and these methods often exhibit faster convergence and better numerical behavior than their L 2 counterparts [70].

Returning to the graph setting, our learnable matrices A k on k -cliques will lead to inner products ( · , · ) k on functions in Ω k , and this will induce dual derivatives as described in Appendix A.1. However, in this case we will not have d ∗ 0 = d ⊺ 0 , but instead the expression given by the following result:

Proposition A.3. The A k -adjoints d ∗ k to the graph derivative operators d k are given by d ∗ k = A -1 k d ⊺ k A k +1 . Similarly, for any linear operator B : Ω k → Ω k , the A k -adjoint B ∗ = A -1 k B ⊺ A .

Therefore, we see that d ⊺ k A k +1 = A k d ∗ k and hence d ∗ k = A -1 k d ⊺ k A k +1 . Similarly, if q , q ′ denote vectors of k -clique features, it follows from the ℓ 2 -self-adjointness of A k that

Remark A.6. It is common in graph theory to encounter the case where a i > 0 are nodal weights and w ij > 0 are edge weights. These are nothing more than the (diagonal) inner products A 0 , A 1 in disguise, and so Proposition A.3 immediately yields the familiar formula for the induced divergence

Note that all of these notions extend to the case of block inner products in the obvious way. For example, if q , p are node resp. edge features, it follows that A = diag ( A 0 , A 1 ) is an inner product on node-edge feature pairs, and the adjoints of node-edge operators with respect to A are computed as according to Proposition A.3.

Remark A.7. For convenience, this work restricts to diagonal matrices A 0 , A 1 . However, note that a matrix which is diagonal in 'edge space' G 2 is generally full in a nodal representation. This is because an (undirected) edge is uniquely specified by the two nodes which it connects, meaning that a purely local quantity on edges is necessarily nonlocal on nodes.

A.4 Higher order attention

As mentioned in the body, when f = exp and ˜ a ( q i , q j ) = (1 /d ) ⟨ W K q i , W Q q j ⟩ , defining the learnable inner products A 0 = ( a 0 ,ii ) , A 1 = ( a 1 ,ij ) as

recovers scaled dot product attention as A -1 0 A 1 .

Remark A.8. Technically, A 1 is an inner product only with respect to a predefined ordering of the edges α = ( i, j ) , since we do not require A 1 be orientation-invariant. On the other hand, it is both unnecessary and distracting to enforce symmetry on A 1 in this context, since any necessary symmetrization will be handled automatically by the differential operator d ∗ 0 .

Similarly, other common attention mechanisms are produced by modifying the pre-attention function ˜ a . While A -1 0 A 1 never appears in the brackets of Section 4, letting α = ( i, j ) denote a global edge with endpoints i, j , it is straightforward to calculate the divergence of an antisymmetric edge feature p at node i ,

This shows that b ( q i , q j ) = ( a 1 ,ij + a 1 ,ji ) /a 0 ,ii appears under the divergence in d ∗ 0 = A -1 0 d ⊺ 0 A 1 , which is the usual graph attention up to a symmetrization in A 1 .

Remark A.9. While A 1 is diagonal on global edges α = ( i, j ) , it appears sparse nondiagonal in its nodal representation. Similarly, any diagonal extension A 2 to 2-cliques will appear as a sparse 3-tensor A 2 = ( a 2 ,ijk ) when specified by its nodes.

This inspires a straightforward extension of graph attention to higher-order cliques. In particular, denote by K > 0 the highest degree of clique under consideration, and define A K -1 = ( a K -1 ,i 1 i 2 ...i K ) by

where W ∈ R ⊗ K n V is a learnable K -tensor. Then, for any 0 ≤ k ≤ K -2 define A k = ( a k,i 1 i 2 ...i k +1 ) by

This recovers the matrices A 0 , A 1 from before when K = 2 , and otherwise extends the same core idea to higher-order cliques. It's attractive that the attention mechanism captured by d ∗ k remains asymmetric, meaning that the attention of any one node to the others in a k -clique need not equal the attention of the others to that particular node.

Remark A.10. A more obvious but less expressive option for higher-order attention is to let

for any 0 ≤ k ≤ K -2 . However, application of the combinatorial codifferential d ⊺ k -1 appearing in d ∗ k -1 will necessarily symmetrize this quantity, so that the asymmetry behind the attention mechanism is lost in this formulation.

To illustrate how this works more concretely, consider the extension K = 3 to 2-cliques, and let N ( i, j ) = N ( i ) ∩N ( j ) . We have the tensors A 2 = ( a 2 ,ijk ) , A 1 = ( a 1 ,ij ) , and A 0 = ( a 0 ,i ) defined by

This provides a way for (features on) 3-node subgraphs of G to attend to each other, and can be similarly built-in to the differential operator d ∗ 1 = A -1 1 d ⊺ 0 A 2 .

A.5 Exterior calculus interpretation of GATs

Let N ( i ) denote the one-hop neighborhood of node i , and let N ( i ) = N ( i ) ∪{ i } . Recall the standard (single-headed) graph attention network (GAT) described in [12], described layer-wise as

where σ is an element-wise nonlinearity, W k is a layer-dependent embedding matrix, and a ( q i , q j ) denotes the attention node i pays to node j . Traditionally, the attention mechanism is computed through

where the pre-attention coefficients ˜ a ( q i , q j ) and nodal weights σ i are defined as

However, the exponentials in the outer Softmax are often replaced with other nonlinear functions, e.g. Squareplus , and the pre-attention coefficients ˜ a appear as variable (but learnable) functions of the nodal features. First, notice that (1) the attention coefficients a ( q i , q j ) depend on the node features q and not simply the topology of the graph, and (2) the attention coefficients are not symmetric, reflecting the fact that the attention paid by node i to node j need not equal the attention paid by node j to node i . A direct consequence of this is that GATs are not purely diffusive under any circumstances, since it was shown in Appendix A.1 that the combinatorial divergence d ⊺ 0 will antisymmetrize the edge features it acts on. In particular, it is clear that the product a ( q i , q j ) ( q i -q j ) is asymmetric in i, j under the standard attention mechanism, since even the pre-attention coefficients ˜ a ( q i , q j ) are not symmetric, meaning that there will be two distinct terms after application of the divergence. More precisely, there is the following subtle result.

Proposition A.4. Let q ∈ R |V|× n V denote an array of nodal features. The expression

where a = A -1 0 A 1 is not the action of a Laplace operator whenever A 1 is not symmetric.

Proof. From Appendix A.3, we know that any Laplace operator on nodes is expressible as d ∗ 0 d 0 = A -1 0 d ⊺ 0 A 1 d 0 for some positive definite A 0 , A 1 . So, we compute the action of the Laplacian at node i ,

which shows that a ( q i , q j ) = (1 / 2) ( a 1 ,ji + a 1 ,ij ) /a 0 ,ii must have symmetric numerator.

While this result shows that GATs (and their derivatives, e.g., GRAND) are not purely diffusive, it also shows that it is possible to get close to GAT (at least syntactically) with a learnable diffusion mechanism. In fact, setting σ = W k = I in (1) yields precisely a single-step diffusion equation provided that a ( q k i , q k j ) is right-stochastic (i.e., ∑ j a ( q i , q j ) 1 j = 1 i ) and built as dictated by Proposition A.4.

Theorem A.11. The GAT layer (1) is a single-step diffusion equation provided that σ = W k = I , and the attention mechanism a ( q i , q j ) = (1 / 2) ( a 1 ,ji + a 1 ,ij ) /a 0 ,ii is right-stochastic.

Proof. First, notice that the Laplacian with respect to an edge set which contains self-loops is computable via

Therefore, taking a single step of heat flow ˙ q = -∆ 0 q with forward Euler discretization and time step τ = 1 is equivalent to

which is just a modified and non-activated GAT layer with W k = I and attention mechanism a .

Remark A.12. Since Softmax and its variants are right-stochastic, Theorem A.11 is what establishes equivalence between the non-divergence equation

and the standard GAT layer seen in, e.g., [49], when a ( q i , q j ) is the usual attention mechanism.

Remark A.13. In the literature, there is an important (and often overlooked) distinction between the positive graph/Hodge Laplacian ∆ 0 and the negative 'geometer's Laplacian' ∆ which is worth noting here. Particularly, we have from integration-by-parts that the gradient ∇ = d 0 is L 2 -adjoint to minus the divergence -∇· = d ⊺ 0 , so that the two Laplace operators ∆ 0 = d ⊺ 0 d 0 and ∆ = ∇·∇ differ by a sign. This is why the same ℓ 2 -gradient flow of Dirichlet energy can be equivalently expressed as ˙ q = ∆ q = -∆ 0 q , but not by, e.g., ˙ q = ∆ 0 q .

This shows that, while they are not equivalent, there is a close relationship between attention and diffusion mechanisms on graphs. The closest analogue to the standard attention expressible in this format is perhaps the choice a 1 ,ij = f (˜ a ( q i , q j )) , a 0 ,ii = ∑ j ∈ ¯ N ( i ) a 1 ,ij , discussed in Section 4 and Appendix A.4, where f is any scalar-valued positive function. For example, when f ( x ) = e x , it follows that

which leads to the standard GAT propagation mechanism plus an extra term arising from the fact that the attention a is not symmetric.

Remark A.14. Practically, GATs and their variants typically make use of multi-head attention, defined in terms of an attention mechanism which is averaged over some number | h | of independent 'heads',

which are distinct only in their learnable parameters. While the results of this section were presented in terms of | h | = 1 , the reader can check that multiple attention heads can be used in this framework provided it is the pre-attention ˜ a that is averaged instead.

A.6 Bracket derivations and properties

Here the architectures in the body are derived in greater detail. First, it will be shown that L ∗ = -L , G ∗ = G , and M ∗ = M , as required for structure-preservation.

Proposition A.5. For L , G , M defined in Section 4, we have L ∗ = -L , G ∗ = G , and M ∗ = M .

Proof. First, denoting A = diag ( A 0 , A 1 ) , it was shown in section A.3 that B ∗ = A -1 B ⊺ A for any linear operator B of appropriate dimensions. So, applying this to L , it follows that

Similarly, it follows that

where the second-to-last equality used that A 1 A -1 1 = I

Remark A.15. Note that the choice of zero blocks in L , G is sufficient but not necessary for these adjointness relationships to hold. For example, one could alternatively choose the diagonal blocks of L to contain terms like B -B ∗ for an appropriate message-passing network B .

Next, we compute the gradients of energy and entropy with respect to ( · , · ) .

Proposition A.6. The A -gradient of the energy

satisfies

Moreover, given the energy and entropy defined as

where f E : R n V → R acts on node features, g E , g S : R n E → R act on edge features, and s denotes sum aggregation over nodes or edges, the A -gradients are

Proof. Since the theory of A -gradients in section A.3 establishes that ∇ E = A -1 δE , it is only necessary to compute the L 2 -gradients. First, letting x = ( q p ) ⊺ , it follows for the first definition of energy that

showing that δE ( q , p ) = ( q p ) ⊺ , as desired. Moving to the metriplectic definitions, since each term of E,S has the same functional form, it suffices to compute the gradient of f ( s ( Bq )) for some function f : R n f → R and matrix B : R |V| → R |V| . To that end, adopting the Einstein summation convention where repeated indices appearing up-and-down in an expression are implicitly summed, if 1 ≤ a, b ≤ n f and 1 ≤ i, j ≤ |V| , we have

implying that ∇ s ( q ) = 1 ⊗ I . Continuing, it follows that

showing that ∇ ( f ◦ s ◦ B ) decomposes into an outer product across modalities. Applying this formula to the each term of E,S then yields the L 2 -gradients,

from which the desired A -gradients follow directly.

Finally, we can show that the degeneracy conditions for metriplectic structure are satisfied by the network in Section 4.

Theorem A.16. The degeneracy conditions L ∇ S = M ∇ E = 0 are satisfied by the metriplectic bracket in Section A.6.

Proof. This is a direct calculation using Theorem 3.1 and Proposition A.6. In particular, it follows that

since d 1 d 0 = 0 as a consequence of the graph calculus. These calculations establish the validity of the energy conservation and entropy generation properties seen previously in the manuscript body.

Remark A.17. Clearly, this is not the only possible metriplectic formulation for GNNs. On the other hand, this choice is in some sense maximally general with respect to the chosen operators L , G , since only constants are in the kernel of d 0 (hence there is no reason to include a nodal term in S ), and only elements in the image of d ⊺ 1 (which do not exist in our setting) are guaranteed to be in the kernel of d ⊺ 0 for any graph. Therefore, M is chosen to be essentially G without the ∆ 0 term, whose kernel is graph-dependent and hence difficult to design.

B Experimental details and more results

This Appendix provides details regarding the experiments in Section 5, as well as any additional information necessary for reproducing them. We implement the proposed algorithms with PYTHON and PYTORCH [71] that supports CUDA. The experiments are conducted on systems that are equipped with NVIDIA RTX A100 and V100 GPUs. For NODEs capabilities, we use the TORCHDIFFEQ library [16].

B.1 Damped double pendulum

The governing equations for the damped double pendulum can be written in terms of four coupled first-order ODEs for the angles that the two pendula make with the vertical axis θ 1 , θ 2 and their associated angular momenta ω 1 , ω 2 (see [72]),

where m 1 , m 2 , l 1 , l 2 are the masses resp. lengths of the pendula, ∆ θ = θ 1 -θ 2 is the (signed) difference in vertical angle, g is the acceleration due to gravity, and

for damping constants k 1 , k 2 .

Dataset. Atrajectory of the damped double pendulum by solving an initial value problem associated with the ODE 2. The initial condition used is (1.0, π /2, 0.0, 0.0), and the parameters are m 1 = m 2 = 1 , g = 1 , l 1 = 1 , l 2 = 0 . 9 , k 1 = k 2 = 0 . 1 . For time integrator, we use the TorchDiffeq library [16] with Dormand-Prince 5 (DOPRI5) as the numerical solver. The total simulation time is 50 (long enough for significant dissipation to occur), and solution snapshots are collected at 500 evenly-spaced temporal indices.

To simulate the practical case where only positional data for the system is available, the double pendulum solution is integrated to time T = 50 (long enough for significant dissipation to occur) and post-processed once the angles and angular momenta are determined from the equations above, yielding the ( x, y ) -coordinates of each pendulum mass at intervals of 0 . 1 s. This is accomplished using the relationships

The double pendulum is then treated as a fully connected three-node graph with positional coordinates q i = ( x i , y i ) as nodal features, and relative velocities p α = ( d 0 q ) α as edge features. Note that the positional coordinates ( x 0 , y 0 ) = (0 , 0) of the anchor node are held constant during training. To allow for the necessary flexibility of coordinate changes, each architecture from Section 4 makes use of a message-passing feature encoder before time integration, acting on node features and edge features separately, with corresponding decoder returning the original features after time integration.

To elicit a fair comparison, both the NODE and NODE+AE architectures are chosen to contain comparable numbers of parameters to the bracket architectures ( ∼ 30 k), and all networks are trained for 100,000 epochs. For each network, the configuration of weights producing the lowest overall error during training is used for prediction.

Hyperparameters. The networks are trained to reconstruct the node/edge features in mean absolute error (MAE) using the Adam optimizer [73]. The NODEs and metriplectic bracket use an initial learning rate of 10 -4 , while the other models use an initial learning rate of 10 -3 . The width of the hidden layers in the message passing encoder/decoder is 64, and the number of hidden features for nodes/edges is 32. The time integrator used is simple forward Euler.

Network architectures. The message passing encoders/decoders are 3-layer MLPs mapping, in the node case, nodal features and their graph derivatives, and in the edge case, edge features and their graph coderivatives, to a hidden representation. For the bracket architectures, the attention mechanism used in the learnable coderivatives is scaled dot product. The metriplectic network uses 2-layer MLPs f E , g E , g S with scalar output and hidden width 64. For the basic NODE, node and edge features are concatenated, flattened, and passed through a 4-layer fully connected network of width 128 in each hidden layer, before being reshaped at the end. The NODE+AE architecture uses a 3-layer fully connected network which operates on the concatenated and flattened latent embedding of size 32 ∗ 6 = 192 , with constant width throughout all layers.

B.2 Mujoco

We represent an object as a fully-connected graph, where a node corresponds to a body part of the object and, thus, the nodal feature corresponds to a position of a body part or joint. To learn the dynamics of an object, we again follow the encoder-decoder-type architecture considered in the

(a)

q

(b)

p

(c)

E

and

S

1

0

0

-1

E

S

0

50.0

0

50.0

0

50.0

double-pendulum experiments. First we employ a node-wise linear layer to embed the nodal feature into node-wise hidden representations (i.e., the nodal feature q i corresponds to a position of a body part or an angle of a joint.). As an alternative encoding scheme for the nodal feature, in addition to the position or the angle, nodal velocities are considered as additional nodal features, i.e., q i = ( q i , v i ) . The experimental results of the alternative scheme is represented in the following section B.2.2.

The proposed dynamics models also require edge features (e.g., edge velocity), which are not presented in the dataset. Thus, to extract a hidden representation for an edge, we employ a linear layer, which takes velocities of the source and destination nodes of the edge as an input and outputs edge-wise hidden representations, i.e., the edge feature correspond to a pair of nodal velocities p α = ( v src ( α ) , v dst ( α ) ) , where v src ( α ) and v dst ( α ) denote velocities of the source and destination nodes connected to the edge.

The MuJoCo trajectories are generated in the presence of an actor applying controls. To handle the changes in dynamics due to the control input, we introduce an additive forcing term, parameterized by an MLP, to the dynamics models, which is a similar approach considered in dissipative SymODEN [32]. In dissipative SymODEN, the forcing term is designed to affect only the change of the generalized momenta (also known as the port-Hamiltonian dynamics [74]). As opposed to this approach, our proposed forcing term affects the evolution of both the generalized coordinates that are defined in the latent space. Once the latent states are computed at specified time indices, a node-wise linear decoder is applied to reconstruct the position of body parts of the object. Then the models are trained based on the data matching loss measured in mean-square errors between the reconstructed and the ground-truth positions.

B.2.1 Experiment details

We largely follow the experimental settings considered in [23].

Dataset. As elaborated in [23], the standard Open AI Gym [75] environments preprocess observations in ad-hoc ways, e.g., Hopper clips the velocity observations to [ -10 , 10] d . Thus, the authors in [23] modified the environments to simply return the position and the velocity ( q, v ) as the observation and we use the same dataset, which is made publicly available by the authors. The dataset consists of training and test data, which are constructed by randomly splitting the episodes in the replay buffer into training and test data. Training and test data consist of ∼ 40K and ∼ 300 or ∼ 85 trajectories, respectively. For both training and test data, we include the first 20 measurements (i.e., 19 transitions) in each trajectory.

Hyperparameters. For training, we use the Adam optimizer [73] with the initial learning rate 5e-3 and weight decay 1e-4. With the batch size of 200 trajectories, we train the models for 256 epochs. We also employ a cosine annealing learning rate scheduler with the minimum learning rate 1e-6. For time integrator, we use the Torchdiffeq library with the Euler method.

Network architectures. The encoder and decoder networks are parameterized as a linear layer and the dimension of the hidden representations is set to 80. For attention, the scaled dot-product attention is used with 8 heads and the embedding dimension is set to 16. The MLP for handling the

forcing term consists of three fully-connected layers (i.e., input, output layers and one hidden layer with 128 neurons). The MLP used for parameterizing the 'black-box' NODEs also consists of three fully-connected layers with 128 neurons in each layer.

B.2.2 Additional results

Figure 6 reports the loss trajectories for all considered dynamics models. For the given number of maximum epochs (i.e., 256), the Hamiltonian and double bracket models tend to reach much lower training losses (an order of magnitude smaller) errors than the NODE and Gradient models do. The metriplectic model produces smaller training losses compared to the NODE and gradient models after a certain number of epochs (e.g., 100 epochs for Hopper).

(a) HalfCheetah

(b) Hopper

(c) Swimmer

NODE

Hamiltonian

Gradient

Double

Metripletic

101

100

101

train MSE

train MSE

train MSE

100

100

10-1

10-1.

10-1

10-2

0

50

100

150

200

250

0

50

100

150

200

250

0

50

100

150

200

250

In the next set of experiments, we provide not only positions/angles of body parts as nodal features, but also velocities of the body parts as nodal features (i.e., q i = ( q i , v i ) ). Table 6 reports the relative errors measured in L2-norm; again, the Hamiltonian, double bracket, and metriplectic outperform other dynamics models. In particular, the metriplectic bracket produces the most accurate predictions in the Hopper and Swimmer environments. Figure 7 reports the loss trajectories for all considered models. Similar to the previous experiments with the position as the only nodal feature, the Hamiltonian and Double bracket produces the lower training losses than the NODE and Gradient models do. For the Hopper and Swimmer environments, however, among all considered models, the metriplectic model produces the lowest training MSEs after 256 training epochs.

B.3 Node classification

To facilitate comparison with previous work, we follow the experimental methodology of [61].

(a) HalfCheetah

(b) Hopper

(c) Swimmer

NODE

Hamiltonian

Gradient

Double

Metripletic

101

100

train MSE

train MSE

train MSE

100

10-1

10-1

F-0

10-2

0

50

100

150

200

250

0

50

100

150

200

250

0

50

100

150

200

250

B.3.1 Experiment details

Datasets. We consider the three well-known citation networks, Cora [58], Citeseer [59], and Pubmed [60]; the proposed models are tested on the datasets with the original fixed Planetoid traing/test splits, as well as random train/test splits. In addition, we also consider the coauthor graph, CoauthorCS [61] and the Amazon co-purchasing graphs, Computer and Photo [62]. Table 7 provides some basic statistics about each dataset.

Hyperparameters The bracket architectures employed for this task are identical to those in Section 4 except for that the right-hand side of the Hamiltonian, gradient, and double bracket networks is scaled by a learnable parameter Sigmoid ( α ) > 0 , and the matrix A 2 = I is used as the inner product on 3-cliques. It is easy to verify that this does not affect the structural properties or conservation character of the networks. Nodal features q i are specified by the datasets, and edge features p α = ( d 0 q ) α are taken as the combinatorial gradient of the nodal features. In order to determine good hyperparameter configurations for each bracket, a Bayesian search is conducted using Weights and Biases [76] for each bracket and each dataset using a random 80/10/10 train/valid/test split with random seed 123. The number of runs per bracket was 500 for CORA, CiteSeer, and PubMed, and 250 for CoauthorCS, Computer, and Photo. The hyperparameter configurations leading to the best validation accuracy are used when carrying out the experiments in Table 4 and Table 5.

Specifically, the hyperparameters that are optimized are as follows: initial learning rate (from 0.0005 to 0.05), number of training epochs (from 25 to 150), method of integration (rk4 or dopri5), integration time (from 1 to 5), latent dimension (from 10 to 150 in increments of 10), pre-attention mechanism ˜ a (see below), positive function f (either exp or Squareplus ), number of pre-attention heads (from 1 to 15, c.f. Remark A.14), attention embedding dimension (from 1 × to 15 × the number of heads), weight decay rate (from 0 to 0.05), dropout/input dropout rates (from 0 to 0.8), and the MLP activation function for the metriplectic bracket (either relu , tanh , or squareplus ). The pre-attention is chosen

from one of four choices, defined as follows:

Network architectures. The architectures used for this experiment follow that of GRAND [49], consisting of the learnable affine encoder/decoder networks ϕ, ψ and learnable bracket-based dynamics in the latent space. However, recall that the bracket-based dynamics require edge features, which are manufactured as p α = ( d 0 q ) α . In summary, the inference procedure is as follows:

Training is accomplished using the standard cross entropy

where t i is the one-hot truth vector corresponding to the i th node. In the case of the metriplectic network, the networks f E , g E , g S are 2-layer MLPs with hidden dimension equal to the latent feature dimension and output dimension 1.

B.3.2 Additional depth study

Here we report the results of a depth study on Cora with the Planetoid train/val/test split. Table 9 shows the train/test accuracy of the different bracket architectures under two different increased depth conditions, labeled Task 1 and Task 2, respectively. Task 1 refers to fixing the integration step-size at ∆ t = 1 and integrating to a variable final time T , while Task 2 instead fixes the final time T to the value identified by the hyperparameter search (see Table 8) and instead varies the step-size ∆ t . Notice that both tasks involve repeatedly composing the trained network and hence simulate increasing depth, so that any negative effects of network construction such as oversmoothing, oversquashing, or vanishing/exploding gradients should appear in both cases. For more information, Table 10 provides a runtime comparison corresonding to the depth studies in Table 9.

Observe that every bracket-based architecture exhibits very stable performance in Task 2, where the final time is held fixed while the depth is increased. This suggests that our proposed networks are dynamically stable and effectively mitigate negative effects like oversmoothing which are brought by repeated composition and often seen in more standard GNNs. Interestingly, despite success on Task 2, only the gradient and metriplectic architectures perfectly maintain or improve their performance during the more adversarial Task 1 where the final time is increased with a fixed step-size. This suggests that, without strong diffusion, the advection experienced during conservative dynamics has the potential to radically change label classification over time, as information is moved through the feature domain in a loss-less fashion.

Remark B.1. It is interesting that the architecture most known for oversmoothing (i.e., gradient) exhibits the most improved classification performance with increasing depth on Task 1. This is perhaps due to the fact that the gradient system decouples over nodes and edges, while the others

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a unified framework inspired the bracket-based dynamical system to analysis the oversmoothing problem in GNN. The past work may leverage the opposite physics concept such as the reversible processes, irreversible process and therefore it is not clear how such concept help to design GNNs, while the framework in this paper gives a deeper understanding. Leveraging the data-driven exterior calculus, this paper constructs four novel architectures which span the both reversibility and irreversibility spectrum using geometric brackets as a means of parameterizing dynamics abstractly without empirically assuming a physical model. Interestingly, it can reinterpretate the message-passing  and GAT as the fluxes and conservation balances of physics simulator.  It also generalize the attention mechanism which extends the nodal feature to higher order cliques and provide a unified evaluation of dissipation.  Emperically, the author compare the architectures derived from this framework with classical GNN,e.g., GAT, GDE over several benchmarks.\n\nSTRENGTHS:\n1. This paper is well-written and offers valuable insights into the construction of graph neural networks. As far as I am aware, the idea presented in this paper is innovative. However, since I lack a background in physics, I would appreciate hearing the suggestions of other reviewers who are familiar with bracket-based dynamical systems.\n\n2. The experimental result in damped double pendulum and MuJoCO dynamics looks good.\n\n3. The author made the code available, promoting reproducibility.\n\nWEAKNESSES:\n1. This paper may be a little hard to understand for the reader without physics background.\n\n2. It seems that the algorithm proposed by the author does not outperform the baselines in node classification problem.\n\nQUESTIONS:\nI am curious about the precise computational complexity of the architectures proposed by the author, specifically in the context of N nodes or N neighbors. It would be beneficial to have concrete values rather than just the order of complexity. Additionally, it would be valuable to understand the advantages of these architectures in comparison to traditional graph neural networks (GNNs).\n\nRegarding the node classification experiment, the algorithm's performance is generally comparable to the baseline, although it does appear weaker in some cases. However, it showcases excellent performance in physical systems like the double pendulum and MuJoCo dynamics. Could this be attributed to the notion that physics-inspired frameworks are inherently more suitable for describing and modeling physical systems rather than social networks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This work provides a comprehensive overview of graph attention networks (GATs), shedding light on their fundamental concepts and principles. Additionally, the authors introduce a set of novel GNN architectures that leverage structure-preserving bracket-based dynamical systems. By incorporating these systems into GNNs, the proposed architectures aim to enhance the modeling capabilities and performance of graph-based machine learning tasks. Overall, the paper presents both a broad perspective on GATs and innovative approaches to further advance the field.

STRENGTHS:
They give a detailed theoretical analysis of graph attention networks with exterior calculus techniques. They also provide a generalized attention mechanism and propose several novel structure-preserving GNNs with satisfying performance.

WEAKNESSES:
The table writing is not clear. Sometimes orange result is better than blue result while sometime blue is better.

QUESTIONS:
I'd like to know whether your analysis can explain why "Gradient“ model performs worse on heterophily graphs like Computer and Photo. Which physics system is suitable for heterophily graphs and which is suitable for homophily graphs?

LIMITATIONS:
They have already listed their limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes structure-preserving bracket-based dynamical systems to learn physical systems using GNNs. The authors proposed four formulations depending on completeness and character (conservative or dissipative). The models are demonstrated to be effective in physical system and node classification tasks.\n\nSTRENGTHS:\n* The method is constructed based on solid mathematics, and incorporating attention as a learnable inner product seems quite natural.\n* The authors suggested various formalisms depending on completeness and character under the notion of the abstract bracket formulations, which offers a general viewpoint for physical systems.\n* The method is quite adaptive because it is demonstrated to be effective in various tasks, including physical system and node classification.\n\nWEAKNESSES:\n* It is unclear to the reviewer where is the authors' contribution and where it is from prior works, particularly the mathematical part. The presentation could be more clear to distinguish existing work and the contribution of the work.\n* The authors claim, \"We use these architectures to systematically evaluate the role of (ir)reversibility in the performance of deep GNNs.\" However, the reviewer could not catch \"the role of (ir)reversibility\" from the experiments. The authors could elaborate on this point in the experiments part.\n* Despite the broad range of experiments considered, the performance of the models is moderate. Therefore, it is unclear in which condition the proposed models are effective (see the following weakness for more details).\n* In Section 5.1, the Hamiltonian model works better than the gradient model on the dissipative phenomena. The author could explain why the conservative works better here as it could be key to understanding \"the role of (ir)reversibility.\"\n* In addition, there are no experiments where the gradient model works best. With these results, it is unclear when the gradient model is effective. The authors could add experiments where the gradient works fine, e.g., incomplete and totally dissipative system.\n* In Section 5.3, the authors reported that the proposed model consumed much memory. In this case, computation time for prediction could also be reported because there are typically tradeoffs between speed and accuracy.\n\nMinor points:\n* Pointers to \"Table 3\" (49, 124, and 159) might be \"Table 1.\"\n\nQUESTIONS:\n* Does the method work well for PDEs, e.g., heat and Navier–Stokes equations?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a bracket-based dynamical system framework to design structure-preserving graph neural networks (GNNs). Specifically, the authors leverage four formalisms: Hamiltonian, Gradient, Double-Bracket, and Metriplectic, that model physical systems with different completeness and dissipation characteristics, and implement them via discretizations from exterior calculus. The authors parameterize GNNs with a chosen dynamic by specifying the bracket matrices and learning the energy/entropy functions via graph attention mechanisms. Experiments on real-world graph inference tasks show that structure-preserving GNNs can perform better than those that are not strictly structure-preserving, and the best dynamics depend on the downstream task.\n\nSTRENGTHS:\nStrengths:\n\n1. It is novel to use bracket-based parameterizations to design physics-inspired GNNs.\n\n2. It is fruitful to analyze existing GNNs (e.g., GAT, GRAND) under the bracket-based dynamical framework.\n\nWEAKNESSES:\nWeaknesses:\n\n1. Since the paper is intended for general audience in graph machine learning, it can be written in a clearer manner with more motivating examples and targeting applications. For starters, include a notation section.\n\n2. One of the key motivations of the paper is to improve performance of deep GNNs, yet all the experiments in the main manuscripts do not investigate explicitly the role of depth - are the GNNs used in these experiments actually deep? The more related experiments are described and discussed in Appendix B.3.2 as ablation study, but results (Table 8) show that for node classification task, only Gradient-based GNNs perform well when the number of layers increases, whereas other dynamics exhibit (significant) performance drop. If the paper intends to mitigate over-smoothing, the authors should provide more compelling experiments. On the other hand, if the paper simply intends to describe a physics-inspired framework to parameterize GNNs, then the authors should re-write the discussion of deep GNNs.\n\nQUESTIONS:\n1. What's the implications for practitioners? What applications would we expect imposing certain bracket structure outperforms the unconstrained ones? What structural requirement one should choose for different applications (e.g., add a column in Table 1 giving example applications)? The current set of experiments reads a bit confusing to me. For example, experiment 2 considers the node classification task, which requires some sense of clustering / smoothing, but not over-smoothing. So the findings in Table 4 showing that Double Bracket performs the best aligns with our expectations? But in Table 8, the Double Bracket formalism cannot be computed for deeper architectures - Is it merely a computational constraint? More explicit remarks and discussions will be great!\n\n2. The role of (learnable inner product) $A$: the authors connect this with graph attention mechanism; but a priori one can also choose other forms of $A$? Have the authors compare some simple baselines, say fixed $A$ or even just set $A$ as identity? This could also help address the memory issue in experiments (Sec 5.3) where the authors cannot compute $A$ (and thus $E, S$) due to the high-dimensionality of the node features?\n\n3. The authors discussed higher-order attention mechanism based on higher-order cliques (Appendix 4). Will this be useful in practice or is this a pure theoretical discussion? If the former, can the authors provide experimental demonstration? If the latter, the authors might consider removing this from the primary contributions since this does not seem like the focus of the paper?\n\nLIMITATIONS:\nThis work does not have negative societal impact. The limitations are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}