SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tingting Dan
University of North Carolina at Chapel Hill tdan@email.unc.edu
Ziquan Wei
University of North Carolina at Chapel Hill  
ziquanw@email.unc.edu
Minjeong Kim
University of North Carolina at Greensboro mkim@uncg.edu
Jiaqi Ding
University of North Carolina at Chapel Hill jiaqid@cs.unc.edu
Shahar Z Kovalsky
University of North Carolina at Chapel Hill shaharko@unc.edu
Won Hwa Kim
Pohang University of Science and Technology wonhwa@postech.ac.kr
Guorong Wu*
grwu@med.unc.edu
Graphs are ubiquitous in various domains, such as social networks and biological systems. Despite the great successes of graph neural networks (GNNs) in modeling and analyzing complex graph data, the inductive bias of locality assumption, which involves exchanging information only within neighboring connected nodes, restricts GNNs in capturing long-range dependencies and global patterns in graphs. Inspired by the classic Brachistochrone problem, we seek how to devise a new inductive bias for cutting-edge graph application and present a general framework through the lens of variational analysis. The backbone of our framework is a two-way mapping between the discrete GNN model and continuous diffusion functional, which allows us to design application-specific objective function in the continuous domain and engineer discrete deep model with mathematical guarantees. First, we address over-smoothing in current GNNs. Specifically, our inference reveals that the existing layer-by-layer models of graph embedding learning are equivalent to a  $\ell_2$ -norm integral functional of graph gradients, which is the underlying cause of the over-smoothing problem. Similar to edge-preserving filters in image denoising, we introduce the total variation (TV) to promote alignment of the graph diffusion pattern with the global information present in community topologies. On top of this, we devise a new selective mechanism for inductive bias that can be easily integrated into existing GNNs and effectively address the trade-off between model depth and over-smoothing. Second, we devise a novel generative adversarial network (GAN) to predict the spreading flows in the graph through a neural transport equation. To avoid the potential issue of vanishing flows, we tailor the objective function to minimize the transportation within each community while maximizing the inter-community flows. Our new GNN models achieve state-of-the-art (SOTA) performance on graph learning benchmarks such as Cora, Citeseer, and Pubmed.
Graph is a fundamental data structure that arises in various domains, including social network analysis [42], natural language processing [36], computer vision [30], recommender systems [37], and knowledge graphs [21] among others. Tremendous efforts have been made to operate machine learning on graph data (called graph neural networks, or GNNs) at the node [23], link [41], and graph level [27]. The common inductive bias used in GNNs is the homophily assumption that nodes that are connected in a graph are more likely to have similar features or labels. In this context, most GNN models deploy a collection of fully-connected layers to progressively learn graph embeddings by aggregating the nodal feature representations from its topologically-connected neighbors throughout the graph [19].
$\oplus$  group 1  $\ominus$  group 3  $\otimes$  group 2  $\bigcirc$  unlabeled
Figure 1: Demonstration of the root cause of over-smoothing in GNNs. Nodes #1 and #2 are located along the boundary of two communities. The locality assumption in GNNs steers the learning of the graph representations by constraining the information exchange via node-to-node connections. However, such link-wise inductive bias opts to neutralize the contrast of node embeddings between nodes #1 and #2, which might undermine the node classification accuracy. Our research framework yields the solution for the over-smoothing issue by enabling heat-kernel diffusion within each community while penalizing the excessive community-to-community information exchanges (highlighted in red).
Under the hood of GNNs, the graph representation learning process is achieved by various learnable operations, such as message passing [5] or graph convolution [23]. Due to the nature of exchanging information in a local graph neighborhood, however, it is challenging to capture global graph representations, which go beyond node-to-node relationship, by leveraging the deep architecture in GNNs while being free of overly smoothing the feature representations for the closely-connected nodes. Fig. 1 demonstrates the root cause of over-smoothing issue in current GNNs, where node color denotes the group label (no color means unlabeled) and edge thickness indicates connection strength. It is clear that nodes #1 and #2 are located at the boundary of two communities. The inductive bias of GNNs (i.e., locality assumption) enforces the node embedding vectors on node #1 and #2 becoming similar due to being strongly connected (highlighted in red), even though the insight of global topology suggests that their node embeddings should be distinct. As additional layers are added to GNNs, the node embeddings become capable of capturing global feature representations that underlie the entire graph topology. However, this comes at the cost of over-smoothing node embeddings across graph nodes due to (1) an increased number of node-to-node information exchanges, and (2) a greater degree of common topology within larger graph neighborhoods. In this regard, current GNNs only deploy a few layers (typically two or three) [26], which might be insufficient to characterize the complex feature representations on the graph.
It is evident that mitigating the over-smoothing problem in GNNs will enable training deeper models. From a network architecture perspective, skip connections [16; 39], residual connections [25; 18], and graph attention mechanisms [34; 33] have been proposed to alleviate the information loss in GNNs, by either preserving the local feature representation or making information exchange adaptive to the importance of nodes in the graph. Although these techniques are effective to patch the over-smoothing issue in some applications, the lack of an in-depth understanding of the root cause of the problem poses the challenge of finding a generalized solution that can be scaled up to current graph learning applications.
Inspired by the success of neural ordinary differential equations in computer vision [10], research focus has recently shifted to link the discrete model in GNNs with partial differential equation (PDE) based numerical recipes [38; 29; 6; 14]. For example, graph neural diffusion (GRAND) formulates GNNs as a continuous diffusion process [6]. In their framework, the layer structure of GNNs corresponds to a specific discretization choice of temporal operators. Since PDE-based model does not revolutionize the underlying inductive bias in current GNNs, it is still unable to prevent the excessive information change between adjacent nodes as in nodes #1 and #2 in Fig. 1. In this regard, using more advanced PDE solvers only can provide marginal improvements in terms of numerical stability over the corresponding discrete GNN models, while the additional computational cost, even in the feed-forward scenario, could limit the practical applicability of PDE-based methods for large-scale graph learning tasks.
Figure 2: Top: The Brachistochrone problem played a pivotal role in the development of classical mechanics and the powerful mathematical tool known as the calculus of variations. Bottom: We introduce a general framework to answer "Brachistochrone" problems regarding diffusion patterns on graphs that allow us to re-think and re-design application-specific deep model of GNNs with enhanced mathematical interpretability.
In this regard, pioneering work on continuous approaches has prompted to re-think GNN as a graph diffusion process governed by the Euler-Lagrange (E-L) equation of the heat kernel. This formulation is reminiscent of the Brachistochrone problem<sup>2</sup>, which emerged over 400 years ago and established the mathematical framework of classical mechanics. The powerful calculus of variations allows us to generate solutions for various mechanics questions (e.g., the slope that yields the fastest ball sliding down the curve is given by a cycloid) through the lens of E-L equation, as shown in Fig. 2 (top).
In a similar vein, the question that arises in the context of community detection is: What graph diffusion pattern is best suited for preserving community organizations? The question for graph classification would be: What graph diffusion pattern works best for capturing the system-level characteristics of graph topology? Following the spirit of Brachistochrone problem, we present a general research framework to customize application-specific GNNs in a continuous space of graph diffusion functionals. As shown in Fig. 2 (bottom), we have established a fundamental structure for our framework that involves a two-way mapping between a discrete GNN model and a continuous graph diffusion functional. This allows us to develop application-specific objective functions (with an explainable regularization term) in the continuous domain and construct a discrete deep model with mathematical guarantee. We demonstrate two novel GNN models, one for addressing over-smoothing and one for predicting the flows from longitudinal nodal features, both achieving state-of-the-art performance (Cora:  $85.6\%$ , CiteSeer:  $73.9\%$ , Pubmed:  $80.10\%$ , even in 128 network layers).
We have made four major contributions. (1) We establish a connection between the discrete model of GNNs and the continuous functional of inductive bias in graph learning by using the E-L equation as a stepping stone to bridge the discrete and continuous domains. (2) We introduce a general framework to re-think and re-design new GNNs that is less "black-box". (3) We devise a novel selective mechanism upon inductive bias to address the over-smoothing issue in current GNNs and achieve state-of-the-art performance on graph learning benchmarks. (4) We construct a novel GNN in the form of a generative adversarial network (GAN) to predict the flow dynamics in the graph by a neural transport equation.
In the following, we first elucidate the relationship between GNN, PDE, and calculus of variations (COV), which sets the stage for the GNN-PDE-COV framework for new GNN models in Section 2.2.
2.1 Re-think GNNs: Connecting dots across graph neural networks, graph diffusion process, Euler-Lagrange equation, and Lagrangian mechanics
Graph diffusion process. Given graph data  $\mathcal{G} = (V,W)$  with  $N$  nodes  $V = \{v_{i}|i = 1,\dots ,N\}$ , the adjacency matrix  $W = [w_{ij}]_{i,j = 1}^{N}\in \mathbb{R}^{N\times N}$  describes connectivity strength between any two nodes. For each node  $v_{i}$ , we have a graph embedding vector  $x_{i}\in \mathbb{R}^{m}$ . In the context of graph topology, the graph gradient  $(\nabla_{\mathcal{G}}x)_{ij} = w_{ij}(x_i - x_j)$  indicates the feature difference between  $v_{i}$  and  $v_{j}$  weighted
2The Brachistochrone problem is a classic physics problem that involves finding the curve down which a bead sliding under the influence of gravity will travel in the least amount of time between two points.
by the connectivity strength  $w_{ij}$ , where  $\nabla_{\mathcal{G}}$  is a  $\mathbb{R}^N \to \mathbb{R}^{N \times N}$  operator. Thus, the graph diffusion process can be formulated as  $\frac{\partial x(t)}{\partial t} = div(\nabla_{\mathcal{G}}x(t))$ , where the evolution of embedding vectors  $x = [x_i]_{i=1}^N$  is steered by the graph divergence operator.
Connecting GNN to graph diffusion. In the regime of GNN, the regularization in the loss function often measures the smoothness of embeddings  $x$  over the graph by  $x^T\Delta x$ , where  $\Delta = div(\nabla_{\mathcal{G}})$  is the graph Laplacian operator [23]. To that end, the graph smoothness penalty encourages two connected nodes to have similar embeddings by information exchange in each GNN layer. Specifically, the new graph embedding  $x^{l}$  in the  $l^{th}$  layer is essentially a linear combination of the graph embedding  $x^{l-1}$  in the previous layer, i.e.,  $x^{l} = A_{W,\Theta}x^{l-1}$ , where the matrix  $A$  depends on graph adjacency matrix  $W$  and trainable GNN parameter  $\Theta$ . After rewriting  $x^{l} = Ax^{l-1}$  into  $x^{l} - x^{l-1} = (A - I)x^{l-1}$ , updating graph embeddings  $x$  in GNN falls into a discrete graph diffusion process, where the time parameter  $t$  acts as a continuous analog of the layers in the spirit of Neural ODEs [10]. It has been shown in [6] that running the graph diffusion process for multiple iterations is equivalent to applying a GNN layer multiple times.
GNN is a discrete model of Lagrangian mechanics via E-L equation. The diffusion process  $\frac{\partial x(t)}{\partial t} = div\left(\nabla_{\mathcal{G}}x(t)\right)$  has been heavily studied in image processing for decades ago, which is the E-L equation of the functional  $\min_x\int_\Omega |\nabla x|^2 dx$ . By replacing the 1D gradient operator defined in the Euclidean space  $\Omega$  with the graph gradient  $(\nabla_{\mathcal{G}}x)_{ij}$ , it is straightforward to find that the equation governing the graph diffusion process  $\frac{\partial x(t)}{\partial t} = div\left(\nabla_{\mathcal{G}}x(t)\right)$  is the E-L equation of the functional  $\min_x\int_\mathcal{G}|\nabla_{\mathcal{G}}x|^2 dx$  over the graph topology. Since the heat kernel diffusion is essentially the mathematical description of the inductive bias in current GNNs [6; 14], we have established a mapping between the mechanics of GNN models and the functional of graph diffusion patterns in a continuous domain.
Tracing the smoking gun of over-smoothing in GNNs. In Fig. 1, we observed that the inductive bias of link-wise propagation is the major reason for excessive information exchange, which is attributed to the over-smoothing problem in GNNs. An intuitive approach is to align the diffusion process with high-level properties associated with graph topology, such as network communities. After connecting the GNN inductive bias to the functional of graph diffusion process, we postulate that the root cause of over-smoothing is the isotropic regularization mechanism encoded by the  $\ell_2$ -norm. More importantly, connecting GNN to the calculus of variations offers a more principled way to design new deep models with mathematics guarantees and model mechanistic explainability.
2.2 Re-design GNNs: Revolutionize inductive bias, derive new E-L equation, and construct deeper GNN
The general roadmap for re-designing GNNs typically involves three major steps: (1) formulating inductive bias into the functional of graph diffusion patterns; (2) deriving the corresponding E-L equation; and then (3) developing a new deep model of GNN based on the finite difference solution of E-L equation. Since the graph diffusion functional is application-specific, we demonstrate the construction of new GNN models in the following two graph learning applications.
2.2.1 Develop VERY deep GNNs with a selective mechanism for link-adaptive inductive bias
Problem formulation. Taking the feature learning component (learnable parameters  $\Theta$ ) out of GNNs, the graph embeddings  $x^{L}$  (output of an  $L$ -layer GNN) can be regarded as the output of an iterative smoothing process ( $L$  times) underlying the graph topology  $\mathcal{G}$ , constrained by the data fidelity  $\left\| x^{L} - x^{0}\right\|_{2}^{2}$  (w.r.t. the initial graph embeddings  $x^{0}$ ) and graph smoothness term  $\int_{\mathcal{G}}|\nabla_{\mathcal{G}}x|^{2}dx$ . Inspired by the great success of total variation (TV) for preserving edges in image denoising [31], reconstruction [35] and restoration [8], we proposed to address the over-smoothing issue in current GNN by replacing the quadratic Laplacian regularizer with TV on graph gradients, i.e.,  $\mathcal{J}_{TV}(x) = \int |\nabla_{\mathcal{G}}x|dx$ . Thus, the TV-based objective function for graph diffusion becomes:  $\min_{x}(\left\| x - x^{0}\right\|_{2}^{2} + \mathcal{J}_{TV}(x))$ .
However, the  $\ell_1$ -norm function, denoted by  $|\cdot|$  in the definition of the total variation functional  $\mathcal{I}_{TV}$ , is not differentiable at zero. Following the dual-optimization schema [4; 7], we introduce the latent
Figure 3: Our new deep model integrates a novel diffusion-gathering  $(DG)$  layer (for selective graph diffusion) after the conventional fully-connected (FC) layer (for graph representation learning).
auxiliary matrix  $z \in \mathbb{R}^{N \times N}$  and reformulate the TV-based functional as  $\min_x \max_z \mathcal{J}_{TV}(x,z) = \max_z \min_x \int (z \otimes \nabla_{\mathcal{G}} x) dx$ , subject to  $|z| \leq 1^{N \times N}$ , where  $\otimes$  is Hadamard operation between two matrices. Furthermore, we use an engineering trick of element-wise operation  $z_{ij}(\nabla_{\mathcal{G}} x)_{ij}$  to keep the degree always non-negative (same as we take the absolute value), which makes the problem solvable. In the end, we reformulate the minimization of  $\mathcal{J}_{TV}(x)$  into a dual min-max functional as  $\mathcal{J}_{TV}(x,z)$ , where we maximize  $z$  ( $z \to 1^{N \times N}$ ) such that  $\mathcal{J}_{TV}(x,z)$  is close enough to  $\mathcal{J}_{TV}(x)$ . Therefore, the new objective function is reformulated as:
$$
\mathcal {J} (x, z) = \max  _ {z} \min  _ {x} \| x - x ^ {0} \| _ {2} ^ {2} + \lambda \int (z \nabla_ {\mathcal {G}} x) d x, \tag {1}
$$
where  $\lambda$  is a scalar balancing the data fidelity term and regularization term. Essentially, Eq. 1 is the dual formulation with min-max property for the TV distillation problem [44].
Constructing E-L equations. To solve Eq. 1, we present the following two-step alternating optimization schema. First, the inner minimization problem (solving for  $x_{i}$ ) in Eq. 1 can be solved by letting  $\frac{\partial}{\partial x_i}\mathcal{J}(x_i,z_i) = 0$ :
$$
\frac {\partial}{\partial x _ {i}} \mathcal {J} \left(x _ {i}, z _ {i}\right) = 2 \left(x _ {i} - x _ {i} ^ {0}\right) + \lambda z _ {i} \nabla_ {\mathcal {G}} x _ {i} = 0 \Rightarrow \hat {x} _ {i} = x _ {i} ^ {0} - \frac {\lambda}{2} z _ {i} \nabla_ {\mathcal {G}} x _ {i} \tag {2}
$$
Replacing  $(\nabla_{\mathcal{G}}x)_{ij}$  with  $w_{ij}(x_i - x_j)$ , the intuition of Eq. 2 is that each element in  $\hat{x}_i$  is essentially the combination between the corresponding initial value in  $x_i^0$  and the overall graph gradients  $z_i\nabla_{\mathcal{G}}x_i = \sum_{j\in \mathcal{N}_i}w_{ij}(x_i - x_j)z_i$  within its graph neighborhood  $\mathcal{N}_i$ . In this regard, Eq. 2 characterizes the dynamic information exchange on the graph, which is not only steered by graph topology but also moderated by the attenuation factor  $z_i$  at each node.
Second, by substituting Eq. 2 back into Eq. 1, the objective function of  $z_{i}$  becomes  $\mathcal{J}(z_i) = \max_{|z_i|\leq 1}\left\| \frac{\lambda}{2} z_i\nabla_{\mathcal{G}}x_i\right\|_2^2 +\lambda z_i\nabla_{\mathcal{G}}(x_i^0 -\frac{\lambda}{2} z_i\nabla_{\mathcal{G}}x_i)$ . With simplification (in Eq. S1 to Eq. S3 of Supplementary), the optimization of each  $z_{i}$  is achieved by  $\arg \min_{|z_i|\leq 1}z_i\nabla_{\mathcal{G}}x_iz_i\nabla_{\mathcal{G}}x_i - \frac{4}{\lambda} z_i\nabla_{\mathcal{G}}x_i^0$ .
Specifically, we employ the majorization-minimization (MM) method [15] to optimize  $z_{i}$  by solving this constrained minimization problem (the detailed derivation process is given in Eq. S4 to S12 of Section S1.1 of Supplementary), where  $z_{i}$  can be iteratively refined by:
$$
z _ {i} ^ {l} = g \left(\underbrace {z _ {i} ^ {l - 1} + \frac {2}{\beta \lambda} \nabla_ {\mathcal {G}} x _ {i}} _ {b}, 1\right) = \left\{ \begin{array}{c c} b & | b | \leq 1 \\ 1 & b > 1 \\ - 1 & b <   - 1 \end{array} \right. \tag {3}
$$
$\beta$  is a hyper-parameter that is required to be no less than the largest eigenvalue of  $(\nabla_{\mathcal{G}}x_{i})(\nabla_{\mathcal{G}}x_{i})^{\top}$ .
Develop new GNN network architecture with a selective inductive bias. The building block in vanilla GNN [23] is a FC (fully-connected) layer where the input is the embedding vectors after isotropic graph diffusion (in  $\ell_{2}$ -norm). Since the estimation of graph embeddings  $x$  in Eq. 2 depends on the latest estimation of  $z^{(l)}$ , such recursive min-max solution for Eq. 1 allows us to devise a new network architecture that disentangles the building block in vanilla GNN into the feature representation learning and graph diffusion underlying TV. As shown in Fig. 3, we first deploy a FC layer to update the graph embeddings  $x^{(l)}$ . After that, we concatenate a diffusion-gathering  $(DG)$  layer for selective graph diffusion, which sequentially applies (1) node-adaptive graph diffusion (blue
Figure 4: (a) The illustration of the computational challenge for estimating the spreading flow. (b) The GAN architecture for min-max optimization in the network.
arrow in Fig. 3) on  $x^{(l)}$  by Eq. 2 $^{3}$ , and (2) gathering operation  $g(\cdot)$  (purple arrow in Fig. 3) to each  $x_{i}^{(l)}$  by Eq. 3.
Remarks. Eq. 3 indicates that larger connective degree results in larger value of  $z$ . Thus, the  $DG$  layer shifts the diffusion patterns by penalizing the inter-community information exchange (due to strong connections) while remaining the heat-kernel diffusion within the community. The preference of such link-adaptive diffusion can be adjusted by the hyper-parameter  $\lambda^4$  in Eq. 1. Recall our intuitive solution for over-smoothing problem in Fig. 1, the  $DG$  layer offers the exact global insight of graph topology to keep the node embeddings distinct between nodes #1 and #2. We demonstrate the effect of  $DG$  layer on the real-world graph data in Fig. S3 of Supplementary document.
2.2.2 Predict flow dynamics through graph neural transport equation
Problem formulation. We live in a world of complex systems, where everything is intricately connected in multiple ways. A holistic insight of how the system's components interact with each other and how changes in one part of the system can affect the behavior of the whole sheds new light on the dynamic behaviors of these complex systems over time. However, oftentimes it is an ill-posed problem. Taking the toy system in Fig. 4(a) as an example, while it is simple to calculate the future focal patterns based on the focal patterns at the current time point and the node-to-node flow information, determining flow dynamics based on longitudinal nodal observations is computationally hard since the solution is not unique.
The naive solution to predict the spreading flow is to (1) train a GNN to learn the intrinsic node embeddings and (2) predict the flow based on the difference of learned embeddings. However, this two-step approach might suffer from vanishing flow due to over-smoothing in GNNs. Following the spirit of Brachistochrone problem, we ask the question "What flow field  $f(t) = [f_{ij}(t)]_{i,j=1}^{N}$  underlines the system mechanics to the extent that it is able to predict the behaviors in the future?"
In this paper, we focus on the conservative system of energy transportation [2]. The system mechanics is formulated as:
$$
\frac {d x}{d t} + d i v (q) = 0 \tag {4}
$$
where  $q = [q_{ij}]_{i,j=1}^{N}$  is the flux field which propagates the potential energy  $u(t) = [u_i(t)]_{i=1}^{N}$  (conserved quantity) over time. Similar to a gravity field driving water flow, the intuition of Eq. 4 is that the change of energy density  $u$  (we assume there is a non-linear mapping  $\phi$  from external force  $x$  to  $u$ , i.e.,  $u_i = \phi(x_i)$ ) leads to energy transport throughout the entire graph. As flux is closely related to the difference of energy  $\nabla_{\mathcal{G}}u$  underlying the graph topology, we assume the energy flux  $q$  is regulated by the potential energy field  $\nabla_{\mathcal{G}}u$ , i.e.,  $q = \alpha \otimes \nabla_{\mathcal{G}}u$ , where  $\alpha = [\alpha_{ij}]_{i,j=1}^{N}$  is a learnable matrix characterizing the link-wise contribution of each energy potential  $\nabla_{\mathcal{G}}u_{ij}$  to the potential energy flux  $q_{ij}$ .