SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore, we see that  $d_k^\intercal \mathbf{A}_{k + 1} = \mathbf{A}_k d_k^*$  and hence  $d_k^* = \mathbf{A}_k^{-1} d_k^\intercal \mathbf{A}_{k + 1}$ . Similarly, if  $\mathbf{q}, \mathbf{q}'$  denote vectors of  $k$ -clique features, it follows from the  $\ell^2$ -self-adjointness of  $\mathbf{A}_k$  that
$$
\left(\mathbf {q}, \mathbf {B q} ^ {\prime}\right) _ {k} = \left\langle \mathbf {q}, \mathbf {A} _ {k} \mathbf {B q} ^ {\prime} \right\rangle = \left\langle \mathbf {B} ^ {\intercal} \mathbf {A} _ {k} \mathbf {q}, \mathbf {q} ^ {\prime} \right\rangle = \left\langle \mathbf {A} _ {k} ^ {- 1} \mathbf {B} ^ {\intercal} \mathbf {A} _ {k} \mathbf {q}, \mathbf {A} _ {k} \mathbf {q} ^ {\prime} \right\rangle = \left(\mathbf {B} ^ {*} \mathbf {q}, \mathbf {q} ^ {\prime}\right) _ {k},
$$
establishing that  $\mathbf{B}^{*} = \mathbf{A}_{k}^{-1}\mathbf{B}^{\intercal}\mathbf{A}_{k}$
Remark A.6. It is common in graph theory to encounter the case where  $a_i > 0$  are nodal weights and  $w_{ij} > 0$  are edge weights. These are nothing more than the (diagonal) inner products  $\mathbf{A}_0, \mathbf{A}_1$  in disguise, and so Proposition A.3 immediately yields the familiar formula for the induced divergence
$$
\left(d _ {0} ^ {*} \mathbf {p}\right) _ {i} = \frac {1}{a _ {i}} \sum_ {j: (i, j) \in \mathcal {E}} w _ {i j} \left(\mathbf {p} _ {j i} - \mathbf {p} _ {i j}\right).
$$
Note that all of these notions extend to the case of block inner products in the obvious way. For example, if  $\mathbf{q}$ ,  $\mathbf{p}$  are node resp. edge features, it follows that  $\mathbf{A} = \mathrm{diag}(\mathbf{A}_0, \mathbf{A}_1)$  is an inner product on node-edge feature pairs, and the adjoints of node-edge operators with respect to  $\mathbf{A}$  are computed as according to Proposition A.3.
Remark A.7. For convenience, this work restricts to diagonal matrices  $\mathbf{A}_0, \mathbf{A}_1$ . However, note that a matrix which is diagonal in "edge space"  $\mathcal{G}_2$  is generally full in a nodal representation. This is because an (undirected) edge is uniquely specified by the two nodes which it connects, meaning that a purely local quantity on edges is necessarily nonlocal on nodes.
A.4 Higher order attention
As mentioned in the body, when  $f = \exp$  and  $\tilde{a}(\mathbf{q}_i, \mathbf{q}_j) = (1 / d) \langle \mathbf{W}_K \mathbf{q}_i, \mathbf{W}_Q \mathbf{q}_j \rangle$ , defining the learnable inner products  $\mathbf{A}_0 = (a_{0,ii}), \mathbf{A}_1 = (a_{1,ij})$  as
$$
a _ {0, i i} = \sum_ {j \in \mathcal {N} (i)} f \left(\tilde {a} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right)\right), \quad a _ {1, i j} = f \left(\tilde {a} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right)\right),
$$
recovers scaled dot product attention as  $\mathbf{A}_0^{-1}\mathbf{A}_1$
Remark A.8. Technically,  $\mathbf{A}_1$  is an inner product only with respect to a predefined ordering of the edges  $\alpha = (i,j)$ , since we do not require  $\mathbf{A}_1$  be orientation-invariant. On the other hand, it is both unnecessary and distracting to enforce symmetry on  $\mathbf{A}_1$  in this context, since any necessary symmetrization will be handled automatically by the differential operator  $d_0^*$ .
Similarly, other common attention mechanisms are produced by modifying the pre-attention function  $\tilde{\alpha}$ . While  $\mathbf{A}_0^{-1}\mathbf{A}_1$  never appears in the brackets of Section 4, letting  $\alpha = (i,j)$  denote a global edge with endpoints  $i,j$ , it is straightforward to calculate the divergence of an antisymmetric edge feature  $\mathbf{p}$  at node  $i$ ,
$$
\begin{array}{l} \left(d _ {0} ^ {*} \mathbf {p}\right) _ {i} = \left(\mathbf {A} _ {0} ^ {- 1} d _ {0} ^ {\intercal} \mathbf {A} _ {1} \mathbf {p}\right) _ {i} = a _ {0, i i} ^ {- 1} \sum_ {\alpha} \left(d _ {0} ^ {\intercal}\right) _ {i \alpha} \left(\mathbf {A} _ {1} \mathbf {p}\right) _ {\alpha} \\ = a _ {0, i i} ^ {- 1} \sum_ {\alpha \ni i} (\mathbf {A} _ {1} \mathbf {p}) _ {- \alpha} - (\mathbf {A} _ {1} \mathbf {p}) _ {\alpha} = - \sum_ {j \in \mathcal {N} (i)} \frac {a _ {1 , j i} + a _ {i , i j}}{a _ {0 , i i}} \mathbf {p} _ {i j}. \\ \end{array}
$$
This shows that  $b(\mathbf{q}_i, \mathbf{q}_j) = (a_{1,ij} + a_{1,ji}) / a_{0,ii}$  appears under the divergence in  $d_0^* = \mathbf{A}_0^{-1}d_0^\top \mathbf{A}_1$ , which is the usual graph attention up to a symmetrization in  $\mathbf{A}_1$ .
Remark A.9. While  $\mathbf{A}_1$  is diagonal on global edges  $\alpha = (i,j)$ , it appears sparse nondiagonal in its nodal representation. Similarly, any diagonal extension  $\mathbf{A}_2$  to 2-cliques will appear as a sparse 3-tensor  $\mathbf{A}_2 = (a_{2,ijk})$  when specified by its nodes.
This inspires a straightforward extension of graph attention to higher-order cliques. In particular, denote by  $K > 0$  the highest degree of clique under consideration, and define  $\mathbf{A}_{K-1} = (a_{K-1,i_1i_2\dots i_K})$  by
$$
a _ {K - 1, i _ {1} i _ {2} \dots i _ {K}} = f \left(\mathbf {W} \left(\mathbf {q} _ {i _ {1}}, \mathbf {q} _ {i _ {2}}, \dots , \mathbf {q} _ {i _ {K}}\right)\right),
$$
where  $\mathbf{W} \in \mathbb{R}^{\otimes Kn_V}$  is a learnable  $K$ -tensor. Then, for any  $0 \leq k \leq K-2$  define  $\mathbf{A}_k = (a_{k,i_1i_2\dots i_{k+1}})$  by
$$
a _ {k, i _ {1} i _ {2} \dots i _ {k + 1}} = \sum_ {i _ {K}, \dots , i _ {K - k - 1}} a _ {K - 1, i _ {1} i _ {2} \dots i _ {K}}.
$$
This recovers the matrices  $\mathbf{A}_0, \mathbf{A}_1$  from before when  $K = 2$ , and otherwise extends the same core idea to higher-order cliques. It's attractive that the attention mechanism captured by  $d_k^*$  remains asymmetric, meaning that the attention of any one node to the others in a  $k$ -clique need not equal the attention of the others to that particular node.
Remark A.10. A more obvious but less expressive option for higher-order attention is to let
$$
a _ {k, i _ {1} i _ {2} \dots i _ {k + 1}} = \frac {a _ {K - 1 , i _ {1} i _ {2} \dots i _ {K}}}{\sum_ {i _ {K} , \dots , i _ {K - k - 1}} a _ {K - 1 , i _ {1} i _ {2} \dots i _ {K}}},
$$
for any  $0 \leq k \leq K - 2$ . However, application of the combinatorial codifferential  $d_{k-1}^{\mathsf{T}}$  appearing in  $d_{k-1}^{*}$  will necessarily symmetrize this quantity, so that the asymmetry behind the attention mechanism is lost in this formulation.
To illustrate how this works more concretely, consider the extension  $K = 3$  to 2-cliques, and let  $\mathcal{N}(i,j) = \mathcal{N}(i)\cap \mathcal{N}(j)$ . We have the tensors  $\mathbf{A}_2 = (a_{2,ijk})$ ,  $\mathbf{A}_1 = (a_{1,ij})$ , and  $\mathbf{A}_0 = (a_{0,i})$  defined by
$$
a _ {2, i j k} = f \left(\mathbf {W} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}, \mathbf {q} _ {k}\right)\right), \quad a _ {1, i j} = \sum_ {k \in \mathcal {N} (i, j)} a _ {2, i j k}, \quad a _ {0, i} = \sum_ {j \in \mathcal {N} (i)} \sum_ {k \in \mathcal {N} (i, j)} a _ {2, i j k}.
$$
This provides a way for (features on) 3-node subgraphs of  $\mathcal{G}$  to attend to each other, and can be similarly built-in to the differential operator  $d_1^* = \mathbf{A}_1^{-1}d_0^\top \mathbf{A}_2$ .
A.5 Exterior calculus interpretation of GATs
Let  $\mathcal{N}(i)$  denote the one-hop neighborhood of node  $i$ , and let  $\overline{\mathcal{N}}(i) = \mathcal{N}(i) \cup \{i\}$ . Recall the standard (single-headed) graph attention network (GAT) described in [12], described layer-wise as
$$
\mathbf {q} _ {i} ^ {k + 1} = \sigma \left(\sum_ {j \in \overline {{\mathcal {N}}} (i)} a \left(\mathbf {q} _ {i} ^ {k}, \mathbf {q} _ {j} ^ {k}\right) \mathbf {W} ^ {k} \mathbf {q} _ {j} ^ {k}\right), \tag {1}
$$
where  $\sigma$  is an element-wise nonlinearity,  $\mathbf{W}^k$  is a layer-dependent embedding matrix, and  $a(\mathbf{q}_i,\mathbf{q}_j)$  denotes the attention node  $i$  pays to node  $j$ . Traditionally, the attention mechanism is computed through
$$
a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) = \mathrm {S o f t m a x} _ {j} \tilde {a} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) = \frac {e ^ {\tilde {a} \left(\mathbf {q} _ {i} , \mathbf {q} _ {j}\right)}}{\sigma_ {i}},
$$
where the pre-attention coefficients  $\tilde{a} (\mathbf{q}_i,\mathbf{q}_j)$  and nodal weights  $\sigma_{i}$  are defined as
$$
\tilde {a} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) = \text {L e a k y R e L U} \left(\mathbf {a} ^ {\intercal} \left(\mathbf {W} ^ {\intercal} \mathbf {q} _ {i} \mid \mid \mathbf {W} ^ {\intercal} \mathbf {q} _ {j}\right)\right), \quad \sigma_ {i} = \sum_ {j \in \overline {{N}} (i)} e ^ {\tilde {a} (\mathbf {q} _ {i}, \mathbf {q} _ {j})}.
$$
However, the exponentials in the outer Softmax are often replaced with other nonlinear functions, e.g. Squareplus, and the pre-attention coefficients  $\tilde{a}$  appear as variable (but learnable) functions of the nodal features. First, notice that (1) the attention coefficients  $a(\mathbf{q}_i,\mathbf{q}_j)$  depend on the node features  $\mathbf{q}$  and not simply the topology of the graph, and (2) the attention coefficients are not symmetric, reflecting the fact that the attention paid by node  $i$  to node  $j$  need not equal the attention paid by node  $j$  to node  $i$ . A direct consequence of this is that GATs are not purely diffusive under any circumstances, since it was shown in Appendix A.1 that the combinatorial divergence  $d_0^\top$  will antisymmetrize the edge features it acts on. In particular, it is clear that the product  $a(\mathbf{q}_i,\mathbf{q}_j)(\mathbf{q}_i - \mathbf{q}_j)$  is asymmetric in  $i,j$  under the standard attention mechanism, since even the pre-attention coefficients  $\tilde{a} (\mathbf{q}_i,\mathbf{q}_j)$  are not symmetric, meaning that there will be two distinct terms after application of the divergence. More precisely, there is the following subtle result.
Proposition A.4. Let  $\mathbf{q} \in \mathbb{R}^{|\mathcal{V}| \times n_V}$  denote an array of nodal features. The expression
$$
\sum_ {j \in \mathcal {N} (i)} a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) \left(\mathbf {q} _ {i} - \mathbf {q} _ {j}\right),
$$
where  $a = \mathbf{A}_0^{-1}\mathbf{A}_1$  is not the action of a Laplace operator whenever  $\mathbf{A}_1$  is not symmetric.
Proof. From Appendix A.3, we know that any Laplace operator on nodes is expressible as  $d_0^* d_0 = \mathbf{A}_0^{-1}d_0^\top \mathbf{A}_1d_0$  for some positive definite  $\mathbf{A}_0, \mathbf{A}_1$ . So, we compute the action of the Laplacian at node  $i$ ,
$$
\begin{array}{l} \left(\Delta_ {0} \mathbf {q}\right) _ {i} = \left(d _ {0} ^ {*} d _ {0} \mathbf {q}\right) _ {i} = \left(\mathbf {A} _ {0} ^ {- 1} d _ {0} ^ {\intercal} \mathbf {A} _ {1} d _ {0} \mathbf {q}\right) _ {i} = a _ {0, i i} ^ {- 1} \sum_ {\alpha} \left(d _ {0} ^ {\intercal}\right) _ {i \alpha} \left(\mathbf {A} _ {1} d _ {0} \mathbf {q}\right) _ {\alpha} \\ = a _ {0, i i} ^ {- 1} \sum_ {\alpha \ni i} \left(\mathbf {A} _ {1} d _ {0} \mathbf {q}\right) _ {- \alpha} - \left(\mathbf {A} _ {1} d _ {0} \mathbf {q}\right) _ {\alpha} = - \frac {1}{2} \sum_ {j \in \mathcal {N} (i)} \frac {a _ {1 , j i} + a _ {i , i j}}{a _ {0 , i i}} \left(\mathbf {q} _ {j} - \mathbf {q} _ {i}\right), \\ = \sum_ {j \in \mathcal {N} (i)} a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) \left(\mathbf {q} _ {j} - \mathbf {q} _ {i}\right), \\ \end{array}
$$
which shows that  $a(\mathbf{q}_i, \mathbf{q}_j) = (1/2)(a_{1,ji} + a_{1,ij}) / a_{0,ii}$  must have symmetric numerator.
While this result shows that GATs (and their derivatives, e.g., GRAND) are not purely diffusive, it also shows that it is possible to get close to GAT (at least syntactically) with a learnable diffusion mechanism. In fact, setting  $\sigma = \mathbf{W}^k = \mathbf{I}$  in (1) yields precisely a single-step diffusion equation provided that  $a(q_i^k,q_j^k)$  is right-stochastic (i.e.,  $\sum_{j}a(\mathbf{q}_{i},\mathbf{q}_{j})1_{j} = 1_{i}$ ) and built as dictated by Proposition A.4.
Theorem A.11. The GAT layer (1) is a single-step diffusion equation provided that  $\sigma = \mathbf{W}^k = \mathbf{I}$ , and the attention mechanism  $a(\mathbf{q}_i, \mathbf{q}_j) = (1/2)(a_{1,ji} + a_{1,ij}) / a_{0,ii}$  is right-stochastic.
Proof. First, notice that the Laplacian with respect to an edge set which contains self-loops is computable via
$$
\left(\Delta_ {0} \mathbf {q}\right) _ {i} = - \sum_ {j \in \overline {{\mathcal {N}}} (i)} a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) \left(\mathbf {q} _ {j} - \mathbf {q} _ {i}\right) = \mathbf {q} _ {i} - \sum_ {j \in \overline {{\mathcal {N}}} (i)} a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) \mathbf {q} _ {j}.
$$
Therefore, taking a single step of heat flow  $\dot{\mathbf{q}} = -\Delta_0\mathbf{q}$  with forward Euler discretization and time step  $\tau = 1$  is equivalent to
$$
\mathbf {q} _ {i} ^ {k + 1} = \mathbf {q} _ {i} ^ {k} - \tau \left(\Delta_ {0} \mathbf {q} ^ {k}\right) _ {i} = \sum_ {j \in \overline {{\mathcal {N}}} (i)} a \left(\mathbf {q} _ {i} ^ {k}, \mathbf {q} _ {j} ^ {k}\right) \mathbf {q} _ {j} ^ {k},
$$
which is just a modified and non-activated GAT layer with  $\mathbf{W}^k = \mathbf{I}$  and attention mechanism  $a$ .
Remark A.12. Since Softmax and its variants are right-stochastic, Theorem A.11 is what establishes equivalence between the non-divergence equation
$$
\dot {\mathbf {q}} _ {i} = \sum_ {j \in \overline {{\mathcal {N}}} (i)} a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) \left(\mathbf {q} _ {j} - \mathbf {q} _ {i}\right),
$$
and the standard GAT layer seen in, e.g., [49], when  $a(\mathbf{q}_i,\mathbf{q}_j)$  is the usual attention mechanism.
Remark A.13. In the literature, there is an important (and often overlooked) distinction between the positive graph/Hodge Laplacian  $\Delta_0$  and the negative "geometer's Laplacian"  $\Delta$  which is worth noting here. Particularly, we have from integration-by-parts that the gradient  $\nabla = d_0$  is  $L^2$ -adjoint to minus the divergence  $-\nabla \cdot = d_0^\intercal$ , so that the two Laplace operators  $\Delta_0 = d_0^\intercal d_0$  and  $\Delta = \nabla \cdot \nabla$  differ by a sign. This is why the same  $\ell^2$ -gradient flow of Dirichlet energy can be equivalently expressed as  $\dot{\mathbf{q}} = \Delta \mathbf{q} = -\Delta_0 \mathbf{q}$ , but not by, e.g.,  $\dot{\mathbf{q}} = \Delta_0 \mathbf{q}$ .
This shows that, while they are not equivalent, there is a close relationship between attention and diffusion mechanisms on graphs. The closest analogue to the standard attention expressible in this format is perhaps the choice  $a_{1,ij} = f(\tilde{a} (\mathbf{q}_i,\mathbf{q}_j))$ ,  $a_{0,ii} = \sum_{j\in \tilde{\mathcal{N}} (i)}a_{1,ij}$ , discussed in Section 4 and Appendix A.4, where  $f$  is any scalar-valued positive function. For example, when  $f(x) = e^{x}$ , it follows that
$$
\begin{array}{l} \left(\Delta_ {0} \mathbf {q}\right) _ {i} = - \frac {1}{2} \sum_ {j \in \mathcal {N} (i)} \frac {e ^ {\tilde {a} \left(\mathbf {q} _ {i} , \mathbf {q} _ {j}\right)} + e ^ {\tilde {a} \left(\mathbf {q} _ {j} , \mathbf {q} _ {i}\right)}}{\sigma_ {i}} \left(\mathbf {q} _ {j} - \mathbf {q} _ {i}\right) \\ = - \frac {1}{2} \sum_ {j \in \mathcal {N} (i)} \left(a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) + \frac {e ^ {\tilde {a} \left(\mathbf {q} _ {j} , \mathbf {q} _ {i}\right)}}{\sigma_ {i}}\right) \left(\mathbf {q} _ {j} - \mathbf {q} _ {i}\right), \\ \end{array}
$$
which leads to the standard GAT propagation mechanism plus an extra term arising from the fact that the attention  $a$  is not symmetric.
Remark A.14. Practically, GATs and their variants typically make use of multi-head attention, defined in terms of an attention mechanism which is averaged over some number  $|h|$  of independent "heads",
$$
a \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right) = \frac {1}{| h |} \sum_ {h} a ^ {h} \left(\mathbf {q} _ {i}, \mathbf {q} _ {j}\right),
$$
which are distinct only in their learnable parameters. While the results of this section were presented in terms of  $|h| = 1$ , the reader can check that multiple attention heads can be used in this framework provided it is the pre-attention  $\tilde{a}$  that is averaged instead.
A.6 Bracket derivations and properties
Here the architectures in the body are derived in greater detail. First, it will be shown that  $\mathbf{L}^{*} = -\mathbf{L}$ ,  $\mathbf{G}^{*} = \mathbf{G}$ , and  $\mathbf{M}^{*} = \mathbf{M}$ , as required for structure-preservation.
Proposition A.5. For  $\mathbf{L},\mathbf{G},\mathbf{M}$  defined in Section 4, we have  $\mathbf{L}^{*} = -\mathbf{L}$ ,  $\mathbf{G}^{*} = \mathbf{G}$ , and  $\mathbf{M}^{*} = \mathbf{M}$ .
Proof. First, denoting  $\mathbf{A} = \mathrm{diag}\left(\mathbf{A}_0,\mathbf{A}_1\right)$ , it was shown in section A.3 that  $\mathbf{B}^{*} = \mathbf{A}^{-1}\mathbf{B}^{\top}\mathbf{A}$  for any linear operator  $\mathbf{B}$  of appropriate dimensions. So, applying this to  $\mathbf{L}$ , it follows that
$$
\begin{array}{l} \mathbf {L} ^ {*} = \left( \begin{array}{c c} \mathbf {A} _ {0} ^ {- 1} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} ^ {- 1} \end{array} \right) \left( \begin{array}{c c} \mathbf {0} & - d _ {0} ^ {*} \\ d _ {0} & \mathbf {0} \end{array} \right) ^ {\intercal} \left( \begin{array}{c c} \mathbf {A} _ {0} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} \end{array} \right) \\ = \left( \begin{array}{c c} \mathbf {0} & \mathbf {A} _ {0} ^ {- 1} d _ {0} ^ {\intercal} \mathbf {A} _ {1} \\ - \mathbf {A} _ {1} ^ {- 1} (d _ {0} ^ {*}) ^ {\intercal} \mathbf {A} _ {0} & \mathbf {0} \end{array} \right) = \left( \begin{array}{c c} \mathbf {0} & d _ {0} ^ {*} \\ - d _ {0} & \mathbf {0} \end{array} \right) = - \mathbf {L}. \\ \end{array}
$$
Similarly, it follows that
$$
\begin{array}{l} \mathbf {G} ^ {*} = \left( \begin{array}{c c} \mathbf {A} _ {0} ^ {- 1} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} ^ {- 1} \end{array} \right) \left( \begin{array}{c c} d _ {0} ^ {*} d _ {0} & \mathbf {0} \\ \mathbf {0} & d _ {1} ^ {*} d _ {1} \end{array} \right) ^ {\intercal} \left( \begin{array}{c c} \mathbf {A} _ {0} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} \end{array} \right) \\ = \left( \begin{array}{c c} \mathbf {A} _ {0} ^ {- 1} d _ {0} ^ {\intercal} (d _ {0} ^ {*}) ^ {\intercal} \mathbf {A} _ {0} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} ^ {- 1} d _ {1} ^ {\intercal} (d _ {1} ^ {*}) ^ {\intercal} \mathbf {A} _ {1} \end{array} \right) = \left( \begin{array}{c c} d _ {0} ^ {*} d _ {0} & \mathbf {0} \\ \mathbf {0} & d _ {1} ^ {*} d _ {1} \end{array} \right) = \mathbf {G}, \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {M} ^ {*} = \left( \begin{array}{c c} \mathbf {A} _ {0} ^ {- 1} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} ^ {- 1} \end{array} \right) \left( \begin{array}{c c} \mathbf {0} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} d _ {1} ^ {*} d _ {1} \mathbf {A} _ {1} \end{array} \right) ^ {\intercal} \left( \begin{array}{c c} \mathbf {A} _ {0} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} \end{array} \right) \\ = \left( \begin{array}{c c} \mathbf {0} & \mathbf {0} \\ \mathbf {0} & d _ {1} ^ {\top}   (d _ {1} ^ {*}) ^ {\top}   \mathbf {A} _ {1} ^ {2} \end{array} \right) = \left( \begin{array}{c c} \mathbf {0} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} d _ {1} ^ {*} d _ {1} \mathbf {A} _ {1} \end{array} \right) = \mathbf {M}, \\ \end{array}
$$
where the second-to-last equality used that  $\mathbf{A}_1\mathbf{A}_1^{-1} = \mathbf{I}$
Remark A.15. Note that the choice of zero blocks in  $\mathbf{L}$ ,  $\mathbf{G}$  is sufficient but not necessary for these adjointness relationships to hold. For example, one could alternatively choose the diagonal blocks of  $\mathbf{L}$  to contain terms like  $\mathbf{B} - \mathbf{B}^*$  for an appropriate message-passing network  $\mathbf{B}$ .
Next, we compute the gradients of energy and entropy with respect to  $(\cdot ,\cdot)$
Proposition A.6. The A-gradient of the energy
$$
\nabla E (\mathbf {q}, \mathbf {p}) = \left( \begin{array}{c c} \mathbf {A} _ {0} ^ {- 1} & \mathbf {0} \\ \mathbf {0} & \mathbf {A} _ {1} ^ {- 1} \end{array} \right) \left( \begin{array}{c} \mathbf {q} \\ \mathbf {p} \end{array} \right) = \left( \begin{array}{c} \mathbf {A} _ {0} ^ {- 1} \mathbf {q} \\ \mathbf {A} _ {1} ^ {- 1} \mathbf {p} \end{array} \right).
$$
Moreover, given the energy and entropy defined as
$$
E (\mathbf {q}, \mathbf {p}) = f _ {E} (s (\mathbf {q})) + g _ {E} (s (d _ {0} d _ {0} ^ {\mathbf {T}} \mathbf {p})),
$$
$$
S (\mathbf {q}, \mathbf {p}) = g _ {S} \left(s \left(d _ {1} ^ {\intercal} d _ {1} \mathbf {p}\right)\right),
$$
where  $f_{E}:\mathbb{R}^{n_{V}}\to \mathbb{R}$  acts on node features,  $g_{E},g_{S}:\mathbb{R}^{n_{E}}\to \mathbb{R}$  act on edge features, and  $s$  denotes sum aggregation over nodes or edges, the A-gradients are
$$
\nabla E (\mathbf {q}, \mathbf {p}) = \left( \begin{array}{c} \mathbf {A} _ {0} ^ {- 1} \mathbf {1} \otimes \nabla f _ {E} (s (\mathbf {q})) \\ \mathbf {A} _ {1} ^ {- 1} d _ {0} d _ {0} ^ {\intercal} \mathbf {1} \otimes \nabla g _ {E} (s (d _ {0} d _ {0} ^ {\intercal} \mathbf {p})) \end{array} \right), \quad \nabla S (\mathbf {q}, \mathbf {p}) = \left( \begin{array}{c} \mathbf {0} \\ \mathbf {A} _ {1} ^ {- 1} d _ {1} ^ {\intercal} d _ {1} \mathbf {1} \otimes \nabla g _ {S} (s (d _ {1} ^ {\intercal} d _ {1} \mathbf {p})) \end{array} \right),
$$
Proof. Since the theory of  $\mathbf{A}$ -gradients in section A.3 establishes that  $\nabla E = \mathbf{A}^{-1}\delta E$ , it is only necessary to compute the  $L^2$ -gradients. First, letting  $\mathbf{x} = (\mathbf{q} \quad \mathbf{p})^\top$ , it follows for the first definition of energy that
$$
d E (\mathbf {x}) = \sum_ {i \in \mathcal {V}} \left\langle \mathbf {q} _ {i}, d \mathbf {q} _ {i} \right\rangle + \sum_ {\alpha \in \mathcal {E}} \left\langle \mathbf {p} _ {\alpha}, d \mathbf {p} _ {\alpha} \right\rangle = \left\langle \mathbf {q}, d \mathbf {q} \right\rangle + \left\langle \mathbf {p}, d \mathbf {p} \right\rangle = \left\langle \mathbf {x}, d \mathbf {x} \right\rangle ,
$$
showing that  $\delta E(\mathbf{q},\mathbf{p}) = (\mathbf{q}\quad \mathbf{p})^{\intercal}$ , as desired. Moving to the metriplectic definitions, since each term of  $E,S$  has the same functional form, it suffices to compute the gradient of  $f(s(\mathbf{Bq}))$  for some function  $f:\mathbb{R}^{n_f}\to \mathbb{R}$  and matrix  $\mathbf{B}:\mathbb{R}^{|\mathcal{V}|}\to \mathbb{R}^{|\mathcal{V}|}$ . To that end, adopting the Einstein summation convention where repeated indices appearing up-and-down in an expression are implicitly summed, if  $1\leq a,b\leq n_{f}$  and  $1\leq i,j\leq |\mathcal{V}|$ , we have
$$
d \left(s (\mathbf {q})\right) = \sum_ {i \in | \mathcal {V} |} d \mathbf {q} _ {i} = \sum_ {i \in | \mathcal {V} |} \delta_ {i} ^ {j} d \mathbf {q} _ {j} = \mathbf {1} ^ {j} d q _ {j} ^ {a} \mathbf {e} _ {a} = \left(\mathbf {1} \otimes \mathbf {I}\right): d \mathbf {q} = \nabla \left(s (\mathbf {q})\right): d \mathbf {q},
$$
implying that  $\nabla s(\mathbf{q}) = \mathbf{1} \otimes \mathbf{I}$ . Continuing, it follows that
$$
\begin{array}{l} d \left(f \circ s \circ \mathbf {B} \mathbf {q}\right) = f ^ {\prime} \left(s (\mathbf {B} \mathbf {q})\right) _ {a} s ^ {\prime} (\mathbf {B} \mathbf {q}) _ {i} ^ {a} B ^ {i j} d q _ {j} ^ {a} = f ^ {\prime} \left(s (\mathbf {B} \mathbf {q})\right) _ {a} \mathbf {e} _ {a} \left(B ^ {\intercal}\right) ^ {i j} 1 _ {j} d q _ {i} ^ {a} \\ = \left\langle \nabla f (s (\mathbf {B q})) \otimes \mathbf {B} ^ {\intercal} \mathbf {1}, d \mathbf {q} \right\rangle = \left\langle \nabla (f \circ s \circ \mathbf {B q}), d \mathbf {q} \right\rangle , \\ \end{array}
$$
showing that  $\nabla (f\circ s\circ \mathbf{B})$  decomposes into an outer product across modalities. Applying this formula to the each term of  $E,S$  then yields the  $L^2$  -gradients,