SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anthony Gruber
Center for Computing Research
Albuquerque, NM. USA
adgrube@sandia.gov
Kookjin Lee *
School of Computing and Augmented Intelligence
Tempe, AZ. USA
kookjin.lee@asu.edu
Nathaniel Trask†
Philadelphia, PA. USA
ntrask@seas.upenn.edu
Recent works have shown that physics-inspired architectures allow the training of deep graph neural networks (GNNs) without oversmoothing. The role of these physics is unclear, however, with successful examples of both reversible (e.g., Hamiltonian) and irreversible (e.g., diffusion) phenomena producing comparable results despite diametrically opposed mechanisms, and further complications arising due to empirical departures from mathematical theory. This work presents a series of novel GNN architectures based upon structure-preserving bracket-based dynamical systems, which are provably guaranteed to either conserve energy or generate positive dissipation with increasing depth. It is shown that the theoretically principled framework employed here allows for inherently explainable constructions, which contextualize departures from theory in current architectures and better elucidate the roles of reversibility and irreversibility in network performance. Code is available at the Github repository https://github.com/natrask/BracketGraphs.
* K. Lee acknowledges the support from the U.S. National Science Foundation under grant CNS2210137.
$^{\dagger}$ Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. This article has been co-authored by an employee of National Technology & Engineering Solutions of Sandia, LLC under Contract No. DE-NA0003525 with the U.S. Department of Energy (DOE). The employee owns all right, title and interest in and to the article and is solely responsible for its contents. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this article or allow others to do so, for United States Government purposes. The DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan https://www.energy.gov/downloads/doe-public-access-plan. The work of N. Trask and A. Gruber is supported by the U.S. Department of Energy, Office of Advanced Computing Research under the "Scalable and Efficient Algorithms - Causal Reasoning, Operators and Graphs" (SEA-CROGS) project, the DoE Early Career Research Program, and the John von Neumann fellowship at Sandia.
Graph neural networks (GNNs) have emerged as a powerful learning paradigm able to treat unstructured data and extract "object-relation"/causal relationships while imparting inductive biases which preserve invariances through the underlying graph topology [1, 2, 3, 4]. This framework has proven effective for a wide range of both graph analytics and data-driven physics modeling problems. Despite successes, GNNs have generally struggled to achieve the improved performance with increasing depth typical of other architectures. Well-known pathologies, such as oversmoothing, oversquashing, bottlenecks, and exploding/vanishing gradients yield deep GNNs which are either unstable or lose performance as the number of layers increase [5, 6, 7, 8].
To combat this, a number of works build architectures which mimic physical processes to impart desirable numerical properties. For example, some works claim that posing message passing as either a diffusion process or reversible flow may promote stability or help retain information, respectively. These present opposite ends of a spectrum between irreversible and reversible processes, which either dissipate or retain information. It is unclear, however, what role (ir)reversibility plays [9]. One could argue that dissipation entropically destroys information and could promote oversmoothing, so should be avoided. Alternatively, in dynamical systems theory, dissipation is crucial to realize a low-dimensional attractor, and thus dissipation may play an important role in realizing dimensionality reduction. Moreover, recent work has shown that dissipative phenomena can actually sharpen information as well as smooth it [10], although this is not often noticed in practice since typical empirical tricks (batch norm, etc.) lead to a departure from the governing mathematical theory.
In physics, Poisson brackets and their metriplectic/port-Hamiltonian generalization to dissipative systems provide an abstract framework for studying conservation and entropy production in dynamical systems. In this work, we construct four novel architectures which span the (ir)reversibility spectrum, using geometric brackets as a means of parameterizing dynamics abstractly without empirically assuming a physical model. This relies on an application of the data-driven exterior calculus (DDEC) [11], which allows a reinterpretation of the message-passing and aggregation of graph attention networks [12] as the fluxes and conservation balances of physics simulators [13], providing a simple but powerful framework for mathematical analysis. In this context, we recast graph attention as an inner-product on graph features, inducing graph derivative "building-blocks" which may be used to build geometric brackets. In the process, we generalize classical graph attention [12] to higher-order clique cochains (e.g., labels on edges and loops). The four architectures proposed here scale with identical complexity to classical graph attention networks, and possess desirable properties that have proven elusive in current architectures. On the reversible and irreversible end of the spectrum we have Hamiltonian and Gradient networks. In the middle of the spectrum, Double Bracket and Metriplectic architectures combine both reversibility and irreversibility, dissipating energy to either the environment or an entropic variable, respectively, in a manner consistent with the second law of thermodynamics. We summarize these brackets in Table 1, providing a diagram of their architecture in Figure 1.
Primary contributions:
Theoretical analysis of GAT in terms of exterior calculus. Using DDEC we establish a unified framework for construction and analysis of message-passing graph attention networks, and provide an extensive introductory primer to the theory in the appendices. In this setting, we show that with our modified attention mechanism, GATs amount to a diffusion process for a special choice of activation and weights.
Generalized attention mechanism. Within this framework, we obtain a natural and flexible extension of graph attention from nodal features to higher order cliques (e.g. edge features). We show attention must have a symmetric numerator to be formally structure-preserving, and introduce a novel and flexible graph attention mechanism parameterized in terms of learnable inner products on nodes and edges.
Novel structure-preserving extensions. We develop four GNN architectures based upon bracket-based dynamical systems. In the metriplectic case, we obtain the first architecture with linear complexity in the size of the graph while previous works are  $O(N^3)$ .
Unified evaluation of dissipation. We use these architectures to systematically evaluate the role of (ir)reversibility in the performance of deep GNNs. We observe best performance for partially
Formalism	Equation	Requirements	Completeness	Character
Hamiltonian	\(\dot{\mathbf{x}}=\{\mathbf{x},E\}\)	\(\mathbf{L}^{*}=-\mathbf{L},\)Jacobi's identity	complete	conservative
Gradient	\(\dot{\mathbf{x}}=-[\mathbf{x},E]\)	\(\mathbf{M}^{*}=\mathbf{M}\)	incomplete	totally dissipative
Double Bracket	\(\dot{\mathbf{x}}=\{\mathbf{x},E\}+\{\{\mathbf{x},E\}\}\)	\(\mathbf{L}^{*}=-\mathbf{L}\)	incomplete	partially dissipative
Metriplectic	\(\dot{\mathbf{x}}=\{\mathbf{x},E\}+\left[\mathbf{x},S\right]\)	\(\mathbf{L}^{*}=-\mathbf{L},\mathbf{M}^{*}=\mathbf{M},\) \(\mathbf{L}\nabla S=\mathbf{M}\nabla E=\mathbf{0}\)	complete	partially dissipative
Table 1: The abstract bracket formulations employed in this work. Here  $\mathbf{x}$  represents a state variable, while  $E = E(\mathbf{x}), S = S(\mathbf{x})$  are energy and entropy functions. "Conservative" indicates purely reversible motion, "totally dissipative" indicates purely irreversible motion, and "partially dissipative" indicates motion which either dissipates  $E$  (in the double bracket case) or generates  $S$  (in the metriplectic case).
dissipative systems, indicating that a combination of both reversibility and irreversibility are important. Pure diffusion is the least performant across all benchmarks. For physics-based problems including optimal control, there is a distinct improvement. All models provide near state-of-the-art performance and marked improvements over black-box GAT/NODE networks.
2 Previous works
Neural ODEs: Many works use neural networks to fit dynamics of the form  $\dot{x} = f(x,\theta)$  to time series data. Model calibration (e.g., UDE [14]), dictionary-based learning (e.g., SINDy [15]), and neural ordinary differential equations (e.g., NODE [16]) pose a spectrum of inductive biases requiring progressively less domain expertise. Structure-preservation provides a means of obtaining stable training without requiring domain knowledge, ideally achieving the flexibility of NODE with the robustness of UDE/SINDy. The current work learns dynamics on a graph while using a modern NODE library to exploit the improved accuracy of high-order integrators [17, 18, 19].
Structure-preserving dense networks: For dense networks, it is relatively straightforward to parameterize reversible dynamics, see for example: Hamiltonian neural networks [20, 21, 22, 23], Hamiltonian generative networks [24], Hamiltonian with Control (SymODEN) [25], Deep Lagrangian networks [26] and Lagrangian neural networks [27]. Structure-preserving extensions to dissipative systems are more challenging, particularly for metriplectic dynamics [28] which require a delicate degeneracy condition to preserve discrete notions of the first and second laws of thermodynamics. For dense networks such constructions are intensive, suffering from  $O(N^3)$  complexity in the number of features [29, 30, 31]. In the graph setting we avoid this and achieve linear complexity by exploiting exact sequence structure. Alternative dissipative frameworks include Dissipative SymODEN [32] and port-Hamiltonian [33]. We choose to focus on metriplectic parameterizations due to their broad potential impact in data-driven physics modeling, and ability to naturally treat fluctuations in multiscale systems [34].
Physics-informed vs structure-preserving: "Physics-informed" learning imposes physics by penalty, adding a regularizer corresponding to a physics residual. The technique is simple to implement and has been successfully applied to solve a range of PDEs [35], discover data-driven models to complement first-principles simulators [36, 37, 38], learn metriplectic dynamics [39], and perform uncertainty quantification [40, 41]. Penalization poses a multiobjective optimization problem, however, with parameters weighting competing objectives inducing pathologies during training, often resulting in physics being imposed to a coarse tolerance and qualitatively poor predictions [42, 43]. In contrast, structure-preserving architectures exactly impose physics by construction via carefully designed networks. Several works have shown that penalty-based approaches suffer in comparison, with structure-preservation providing improved long term stability, extrapolation and physical realizability.
Structure-preserving graph networks: Several works use discretizations of specific PDEs to combat oversmoothing or exploding/vanishing gradients, e.g. telegraph equations [44] or various reaction-diffusion systems [45]. Several works develop Hamiltonian flows on graphs [46, 47]. For metriplectic dynamics, [48] poses a penalty based formulation on graphs. We particularly focus on GRAND, which poses graph learning as a diffusive process [49], using a similar exterior calculus framework and interpreting attention as a diffusion coefficient. We show in Appendix A.5 that their
Figure 1: A diagrammatic illustration of the bracket-based architectures introduced in Section 4.
analysis fails to account for the asymmetry in the attention mechanism, leading to a departure from the governing theory. To account for this, we introduce a modified attention mechanism which retains interpretation as a part of diffusion PDE. In this purely irreversible case, it is of interest whether adherence to the theory provides improved results, or GRAND's success is driven by something other than structure-preservation.
3 Theory and fundamentals
Here we introduce the two essential ingredients to our approach: bracket-based dynamical systems for neural differential equations, and the data-driven exterior calculus which enables their construction. A thorough introduction to this material is provided in Appendices A.1, A.3, and A.2.
Bracket-based dynamics: Originally introduced as an extension of Hamiltonian/Lagrangian dynamics to include dissipation [50], bracket formulations are used to inform a dynamical system with certain structural properties, e.g., time-reversibility, invariant differential forms, or property preservation. Even without dissipation, bracket formulations may compactly describe dynamics while preserving core mathematical properties, making them ideal for designing neural architectures.
Bracket formulations are usually specified via some combination of reversible brackets  $\{F,G\} = \langle \nabla F,\mathbf{L}\nabla G\rangle$  and irreversible brackets  $[F,G] = \langle \nabla F,\mathbf{M}\nabla G\rangle ,\{\{F,G\} \} = \langle \nabla F,\mathbf{L}^2\nabla G\rangle$  for potentially state-dependent operators  $\mathbf{L}^{*} = -\mathbf{L}$  and  $\mathbf{M}^{*} = \mathbf{M}$ . The particular brackets which are used in the present network architectures are summarized in Table 1. Note that complete systems are the dynamical extensions of isolated thermodynamical systems: they conserve energy and produce entropy, with nothing lost to the ambient environment. Conversely, incomplete systems do not account for any lost energy: they only require that it vanish in a prescribed way. The choice of completeness is an application-dependent modeling assumption.
Exterior calculus: In the combinatorial Hodge theory [51], an oriented graph  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$  carries sets of  $k$ -cliques, denoted  $\mathcal{G}_k$ , which are collections of ordered subgraphs generated by  $(k + 1)$  nodes. This induces natural exterior derivative operators  $d_k:\Omega_k\to \Omega_{k + 1}$ , acting on the spaces of functions on  $\mathcal{G}_k$ , which are the signed incidence matrices between  $k$ -cliques and  $(k + 1)$ -cliques. An explicit representation of these derivatives is given in Appendix A.1, from which it is easy to check the exact sequence property  $d_{k + 1}\circ d_k = 0$  for any  $k$ . This yields a discrete de Rham complex on the graph  $\mathcal{G}$  (Figure 2). Moreover, given a choice of inner product (say,  $\ell^2$ ) on  $\Omega_k$ , there is an obvious dual de Rham complex which comes directly from adjointness. In particular, one can define dual derivatives  $d_k^*:\Omega_{k + 1}\to \Omega_k$  via the equality
$$
\langle d _ {k} f, g \rangle_ {k + 1} = \langle f, d _ {k} ^ {*} g \rangle_ {k},
$$
from which nontrivial results such as the Hodge decomposition, Poincaré inequality, and coercivity/invertibility of the Hodge Laplacian  $\Delta_k = d_k^* d_k + d_{k-1}d_{k-1}^*$  follow (see e.g. [11]). Using the derivatives  $d_k, d_k^*$ , it is possible to build compatible discretizations of PDEs on  $\mathcal{G}$  which are guaranteed to preserve exactness properties such as, e.g.,  $d_1 \circ d_0 = \operatorname{curl} \circ \operatorname{grad} = 0$ .
The choice of inner product  $\langle \cdot, \cdot \rangle_k$  thus induces a definition of the dual derivatives  $d_k^*$ . In the graph setting [52], one typically selects the  $\ell^2$  inner product, obtaining the adjoints of the signed incidence matrices as  $d_k^* = d_k^\intercal$ . By instead working with the modified inner product  $(\mathbf{v}, \mathbf{w}) = \mathbf{v}^\intercal \mathbf{A}_k \mathbf{w}$  for a machine-learnable  $\mathbf{A}_k$ , we obtain  $d_k^* = \mathbf{A}_k^{-1} d_k^\intercal \mathbf{A}_{k+1}$  (see Appendix A.3). This parameterization
Figure 2: A commutative diagram illustrating the relationship between the graph derivatives  $d_{k}$ , their  $\ell^2$  adjoints  $d_{k}^{\intercal}$ , and the learnable adjoints  $d_{k}^{*}$ . These operators form a de Rham complex due to the exact sequence property  $d_{i + 1} \circ d_i = d_i^\intercal \circ d_{i + 1}^\intercal = d_i^* \circ d_{i + 1}^* = 0$ . We show that the learnable  $\mathbf{A}_k$  may encode attention mechanisms, without impacting the preservation of exact sequence structure.
inhibits the exact sequence property from the graph topology encoded in  $d_k$  while allowing for incorporation of geometric information from data. This leads directly to the following result, which holds for any (potentially feature-dependent) symmetric positive definite matrix  $\mathbf{A}_k$ .
Theorem 3.1. The dual derivatives  $d_k^* : \Omega_{k+1} \to \Omega_k$  adjoint to  $d_k : \Omega_k \to \Omega_{k+1}$  with respect to the learnable inner products  $\mathbf{A}_k : \Omega_k \to \Omega_k$  satisfy an exact sequence property.
Proof.  $d_{k - 1}^{*}d_{k}^{*} = \mathbf{A}_{k - 1}^{-1}d_{k - 1}^{\intercal}\mathbf{A}_{k}\mathbf{A}_{k}^{-1}d_{k}^{\intercal}\mathbf{A}_{k + 1} = \mathbf{A}_{k - 1}^{-1}\left(d_{k}d_{k - 1}\right)^{\intercal}\mathbf{A}_{k + 1} = 0.$
As will be shown in Section 4, by encoding graph attention into the  $\mathbf{A}_k$ , we may exploit the exact sequence property to obtain symmetric positive definite diffusion operators, as well as conduct the cancellations necessary to enforce degeneracy conditions necessary for metriplectic dynamics.
For a thorough review of DDEC, we direct readers to Appendix A.1 and [11]. For exterior calculus in topological data analysis see [52], and an overview in the context of PDEs see [53, 54].
4 Structure-preserving bracket parameterizations
We next summarize properties of the bracket dynamics introduced in Section 3 and displayed in Table 1, postponing details and rigorous discussion to Appendices A.3 and A.6. Letting  $\mathbf{x} = (\mathbf{q},\mathbf{p})$  denote node-edge feature pairs, the following operators will be used to generate our brackets.
$$
\mathbf {L} = \left( \begin{array}{c c} 0 & - d _ {0} ^ {*} \\ d _ {0} & 0 \end{array} \right), \quad \mathbf {G} = \left( \begin{array}{c c} \Delta_ {0} & 0 \\ 0 & \Delta_ {1} \end{array} \right) = \left( \begin{array}{c c} d _ {0} ^ {*} d _ {0} & 0 \\ 0 & d _ {1} ^ {*} d _ {1} + d _ {0} d _ {0} ^ {*} \end{array} \right), \quad \mathbf {M} = \left( \begin{array}{c c} 0 & 0 \\ 0 & \mathbf {A} _ {1} d _ {1} ^ {*} d _ {1} \mathbf {A} _ {1} \end{array} \right).
$$
As mentioned before, the inner products  $\mathbf{A}_0$ ,  $\mathbf{A}_1$ ,  $\mathbf{A}_2$  on  $\Omega_k$  which induce the dual derivatives  $d_0^*$ ,  $d_1^*$  are chosen in such a way that their combination generalizes a graph attention mechanism. The precise details of this construction are given below, and its relationship to the standard GAT network from [12] is shown in Appendix A.5. Notice that  $\mathbf{L}^* = -\mathbf{L}$ , while  $\mathbf{G}^* = \mathbf{G}$ ,  $\mathbf{M}^* = \mathbf{M}$  are positive semi-definite with respect to the block-diagonal inner product  $(\cdot, \cdot)$  defined by  $\mathbf{A} = \mathrm{diag}(\mathbf{A}_0, \mathbf{A}_1)$  (details are provided in Appendix A.6). Therefore,  $\mathbf{L}$  generates purely reversible (Hamiltonian) dynamics and  $\mathbf{G}$ ,  $\mathbf{M}$  generate irreversible (dissipative) ones. Additionally, note that state-dependence in  $\mathbf{L}$ ,  $\mathbf{M}$ ,  $\mathbf{G}$  enters only through the adjoint differential operators, meaning that any structural properties induced by the topology of the graph  $\mathcal{G}$  (such as the exact sequence property mentioned in Theorem 3.1) are automatically preserved.
Remark 4.1. Strictly speaking,  $\mathbf{L}$  is guaranteed to be a truly Hamiltonian system only when  $d_0^*$  is state-independent, since it may otherwise fail to satisfy Jacobi's identity. On the other hand, energy conservation is always guaranteed due to the fact that  $\mathbf{L}$  is skew-adjoint.
In addition to the bracket matrices  $\mathbf{L},\mathbf{M},\mathbf{G}$ , it is necessary to have access to energy and entropy functions  $E,S$  and their associated functional derivatives with respect to the inner product on  $\Omega_0\oplus \Omega_1$  defined by A. For the Hamiltonian, gradient, and double brackets,  $E$  is chosen simply as the "total kinetic energy"
$$
E (\mathbf {q}, \mathbf {p}) = \frac {1}{2} \left(| \mathbf {q} | ^ {2} + | \mathbf {p} | ^ {2}\right) = \frac {1}{2} \sum_ {i \in \mathcal {V}} | \mathbf {q} _ {i} | ^ {2} + \frac {1}{2} \sum_ {\alpha \in \mathcal {E}} | \mathbf {p} _ {\alpha} | ^ {2},
$$
whose  $\mathbf{A}$ -gradient (computed in Appendix A.6) is just  $\nabla E(\mathbf{q},\mathbf{p}) = \left(\mathbf{A}_0^{-1}\mathbf{q}\quad \mathbf{A}_1^{-1}\mathbf{p}\right)^{\top}$ . Since the metriplectic bracket uses parameterizations of  $E, S$  which are more involved, discussion of this case is deferred to later in this Section.
Attention as learnable inner product: Before describing the dynamics, it remains to discuss how the matrices  $\mathbf{A}_i$ ,  $0 \leq i \leq 2$ , are computed in practice, and how they relate to the idea of graph attention. Recall that if  $n_V > 0$  denotes the nodal feature dimension, a graph attention mechanism takes the form  $a(\mathbf{q}_i, \mathbf{q}_j) = f(\tilde{a}_{ij}) / \sum_j f(\tilde{a}_{ij})$  for some differentiable pre-attention function  $\tilde{a}: n_V \times n_V \to \mathbb{R}$  (e.g., for scaled dot product [55]) one typically represents  $a(\mathbf{q}_i, \mathbf{q}_j)$  as a softmax, so that  $f = \exp(\mathbf{q})$ . This suggests a decomposition  $a(\mathbf{q}_i, \mathbf{q}_j) = \mathbf{A}_0^{-1} \mathbf{A}_1$  where  $\mathbf{A}_0 = (a_{0,ii})$  is diagonal on nodes and  $\mathbf{A}_1 = (a_{1,ij})$  is diagonal on edges,
$$
a _ {0, i i} = \sum_ {j \in \mathcal {N} (i)} f \left(\tilde {a} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right)\right), \quad a _ {1, i j} = f \left(\tilde {a} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right)\right).
$$
Treating the numerator and denominator of the standard attention mechanism separately in  $\mathbf{A}_0$ ,  $\mathbf{A}_1$  allows for a flexible and theoretically sound incorporation of graph attention directly into the adjoint differential operators on  $\mathcal{G}$ . In particular, if  $\mathbf{A}_1$  is symmetric with respect to edge-orientation and  $\mathbf{p}$  is an edge feature which is antisymmetric, it follows that
$$
\left(d _ {0} ^ {*} \mathbf {p}\right) _ {i} = \left(\mathbf {A} _ {0} ^ {- 1} d _ {0} ^ {\intercal} \mathbf {A} _ {1} \mathbf {p}\right) _ {i} = \sum_ {j \in \mathcal {N} (i)} a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) \mathbf {p} _ {j i},
$$
which is just graph attention combined with edge aggregation. This makes it possible to give the following informal statement regarding graph attention networks which is explained and proven in Appendix A.5.
Remark 4.2. The GAT layer from [12] is almost the forward Euler discretization of a metric heat equation.