SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {z} _ {u} ^ {(k)} = \mathbf {z} _ {u} ^ {(k - 1)} - \tau \left. \frac {\partial E (\mathbf {Z} ; \mathbf {Z} ^ {(k - 1)})}{\partial \mathbf {z} _ {u}} \right| _ {\mathbf {Z} = \mathbf {Z} ^ {(k - 1)}} \\ = \mathbf {z} _ {u} ^ {(k - 1)} - 2 \frac {\tau}{2 \lambda} \left(\mathbf {z} _ {u} ^ {(k - 1)} - \mathbf {z} _ {u} ^ {(k - 1)}\right) - 2 \frac {\tau}{2 \lambda} \lambda \sum_ {v} c _ {u v} ^ {(k)} \left(\mathbf {z} _ {u} ^ {(k - 1)} - \mathbf {z} _ {v} ^ {(k - 1)}\right) \tag {11} \\ = (1 - \tau) \sum_ {v} c _ {u v} ^ {(k)} \mathbf {z} _ {u} ^ {(k - 1)} + \tau \sum_ {v} c _ {u v} ^ {(k)} \mathbf {z} _ {v} ^ {(k - 1)} \\ = (1 - \tau) \mathbf {z} _ {u} ^ {(k - 1)} + \tau \sum_ {v} c _ {u v} ^ {(k)} \mathbf {z} _ {v} ^ {(k - 1)}. \\ \end{array}
$$
The last step is due to the normalization of the attention scores, i.e.,  $\sum_{v}c_{uv}^{(k)} = 1$ . Eqn. 18 is the updating equation of the  $k$ -th layer of Transformers, i.e., Eqn. 5.
A.2 Proof for Theorem 2
$$
\min  _ {\mathbf {Z}} \sum_ {u} \| \mathbf {z} _ {u} - \mathbf {z} _ {u} ^ {(0)} \| _ {2} ^ {2} + \lambda \sum_ {u, v} c _ {u v} ^ {*} \| \mathbf {z} _ {u} - \mathbf {z} _ {v} \| _ {2} ^ {2}, \tag {12}
$$
Proof. Assume the  $K$ -layer model with the layer-wise updating rule defined by Eqn. 5 yields a sequence of attention matrices and node embeddings  $\mathbf{C}^{(1)}, \mathbf{Z}^{(1)}, \mathbf{C}^{(2)}, \mathbf{Z}^{(2)}, \dots, \mathbf{C}^{(K)}, \mathbf{Z}^{(K)}$ . We define a propagation matrix  $\mathbf{P}^{(k)}$ :
$$
\mathbf {P} ^ {(k)} = (1 - \alpha) \left[ \begin{array}{c c c c} 1 & 0 & \dots & 0 \\ 0 & 1 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & 1 \end{array} \right] + \alpha \left[ \begin{array}{c c c c} c _ {1 1} ^ {(k)} & c _ {1 2} ^ {(k)} & \dots & c _ {1 N} ^ {(k)} \\ c _ {2 1} ^ {(k)} & c _ {2 2} ^ {(k)} & \dots & c _ {2 N} ^ {(k)} \\ \vdots & \vdots & \ddots & \vdots \\ c _ {N 1} ^ {(k)} & c _ {N 2} ^ {(k)} & \dots & c _ {N N} ^ {(k)} \end{array} \right] = (1 - \alpha) \mathbf {I} + \alpha \mathbf {C} ^ {(k)}. \tag {13}
$$
Then Eqn. 5 can be equivalently written as
$$
\mathbf {Z} ^ {(k)} = \mathbf {P} ^ {(k)} \mathbf {Z} ^ {(k - 1)}. \tag {14}
$$
By stacking  $K$  layers of propagation, we can denote the output embeddings as
$$
\mathbf {Z} ^ {(K)} = \mathbf {P} ^ {(k)} \mathbf {Z} ^ {(K - 1)} = \mathbf {P} ^ {(K)} \mathbf {P} ^ {(k - 1)} \mathbf {Z} ^ {(k - 2)} = \dots = \mathbf {P} ^ {(K)} \dots \mathbf {P} ^ {(1)} \mathbf {Z} ^ {(0)} = \mathbf {P} ^ {*} \mathbf {Z} ^ {(0)}. \tag {15}
$$
We next conclude the proof by construction. Defining  $\mathbf{C}^{*} = \frac{1}{\tau^{*}}\left(\mathbf{P}^{*} - (1 - \tau^{*})\mathbf{I}\right)$ , where in specific,
$$
\mathbf {C} ^ {*} = \left[ \begin{array}{c c c c} c _ {1 1} ^ {*} & c _ {1 2} ^ {*} & \dots & c _ {1 N} ^ {*} \\ c _ {2 1} ^ {*} & c _ {2 2} ^ {*} & \dots & c _ {2 N} ^ {*} \\ \vdots & \vdots & \ddots & \vdots \\ c _ {N 1} ^ {*} & c _ {N 2} ^ {*} & \dots & c _ {N N} ^ {*} \end{array} \right] = \frac {1}{\tau^ {*}} \left(\left[ \begin{array}{c c c c} p _ {1 1} ^ {*} & p _ {1 2} ^ {*} & \dots & p _ {1 N} ^ {*} \\ p _ {2 1} ^ {*} & p _ {2 2} ^ {*} & \dots & p _ {2 N} ^ {*} \\ \vdots & \vdots & \ddots & \vdots \\ p _ {N 1} ^ {*} & p _ {N 2} ^ {*} & \dots & p _ {N N} ^ {*} \end{array} \right] - (1 - \tau^ {*}) \left[ \begin{array}{c c c c} 1 & 0 & \dots & 0 \\ 0 & 1 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & 1 \end{array} \right]\right) \tag {16}
$$
We can show that solving the denoising problem with gradient step size  $\frac{\tau^{*}}{2\lambda}$  w.r.t. the objective
$$
\min  _ {\mathbf {Z}} \sum_ {u} \| \mathbf {z} _ {u} - \mathbf {z} _ {u} ^ {(0)} \| _ {2} ^ {2} + \lambda \sum_ {u, v} c _ {u v} ^ {*} \| \mathbf {z} _ {u} - \mathbf {z} _ {v} \| _ {2} ^ {2}, \tag {17}
$$
will induce the output embeddings  $\mathbf{Z}^{(K)}$ , by noticing that
$$
\begin{array}{l} \mathbf {z} _ {u} ^ {(0)} - \frac {\tau^ {*}}{2 \lambda} \left. \frac {\partial E (\mathbf {Z} ; \mathbf {Z} ^ {(0)})}{\partial \mathbf {z} _ {u}} \right| _ {\mathbf {Z} = \mathbf {Z} ^ {(k - 1)}} \\ = \mathbf {z} _ {u} ^ {(0)} - 2 \frac {\tau^ {*}}{2 \lambda} (\mathbf {z} _ {u} ^ {(0)} - \mathbf {z} _ {u} ^ {(0)}) - 2 \frac {\tau^ {*}}{2 \lambda} \lambda \sum_ {v} c _ {u v} ^ {*} (\mathbf {z} _ {u} ^ {(0)} - \mathbf {z} _ {v} ^ {(0)}) \\ = (1 - \tau^ {*}) \sum_ {v} c _ {u v} ^ {*} \mathbf {z} _ {u} ^ {(0)} + \tau^ {*} \sum_ {v} c _ {u v} ^ {*} \mathbf {z} _ {v} ^ {(0)} (18) \\ = (1 - \tau^ {*}) \mathbf {z} _ {u} ^ {(0)} + \tau \sum_ {v} c _ {u v} ^ {*} \mathbf {z} _ {v} ^ {(0)} (18) \\ = (1 - \tau^ {*}) \mathbf {z} _ {u} ^ {(0)} + \tau^ {*} \left(\frac {1}{\tau^ {*}} p _ {u u} ^ {*} + 1 - \frac {1}{\tau^ {*}}\right) \mathbf {z} _ {u} ^ {(0)} + \tau^ {*} \sum_ {v \neq u} \frac {1}{\tau^ {*}} p _ {u v} ^ {*} \mathbf {z} _ {v} ^ {(0)} \\ = \sum_ {v} p _ {u v} ^ {*} \mathbf {z} _ {v} ^ {(0)}. \\ \end{array}
$$
And, we have  $\sum_{v}p_{uv}^{*}\mathbf{z}_{v}^{(0)} = \mathbf{z}_{u}^{(K)}$  since by definition  $\mathbf{Z}^{(K)} = \mathbf{P}^{*}\mathbf{Z}^{(0)}$ .
Table 5: Information for node property prediction datasets.
Dataset	Property	#Tasks	#Nodes	#Edges	#Feats	#Classes
Cora	homophilous	1	2,708	5,429	1,433	7
CiteSeer	homophilous	1	3,327	4,732	3,703	6
PubMed	homophilous	1	19,717	44,324	500	3
Actor	heterophilic	1	7,600	29,926	931	5
Squirrel	heterophilic	1	5,201	216,933	2,089	5
Chameleon	heterophilic	1	2,277	36,101	2,325	5
Deezer	heterophilic	1	28,281	92,752	31,241	2
ogbn-proteins	multi-task	112	132,534	39,561,252	8	2
Amazon2M	long-range dependence	1	2,449,029	61,859,140	100	47
pokec	heterophilic	1	1,632,803	30,622,564	65	2
ogbn-arxiv	homophilous	1	169,343	1,166,243	128	40
ogbn-papers100M	homophilous	1	111,059,956	1,615,685,872	128	172
Our experiments are based on 12 real-world datasets which are all publicly available with open access. The information of these datasets is presented in Table 5.
ARXIV papers, each of which is labeled with one subject area. The goal is to classify the paper into 172 ARXIV subject areas. We also follow the public split by OGB [19]. This dataset is nearly the largest public graph benchmark.
In this section, we provide more details regarding the model implementation, including model architectures, hyper-parameter settings and training details, to facilitate the reproducibility.
Our model is comprised of four modules: input layer, global attention, GNN model, and output layer. We present the details for each of them as follows.
- The input layer is a one-layer MLP with non-linear activation that transforms the input features  $\mathbf{X} \in \mathbb{R}^{N \times D}$  to the node embeddings in the latent space  $\mathbf{Z}^{(0)} \in \mathbb{R}^{N \times d}$ .
- The global attention computes the all-pair influence among  $N$  nodes through the attention mechanism as described in Eqn. 3. Specifically, it feeds the initial embeddings  $\mathbf{Z}^{(0)} \in \mathbb{R}^{N \times d}$  into an one-layer attention network and outputs the updated embeddings  $\mathbf{Z} \in \mathbb{R}^{N \times d}$  that absorbs the global information from other nodes.
- The GNN network GN is instantiated as a graph convolution network which takes the input graph  $\mathbf{A}$  and the initial embeddings  $\mathbf{Z}^{(0)} \in \mathbb{R}^{N \times d}$  as input and outputs the updated embeddings for each node. The latter is then linearly added with the output embeddings of the global attention, i.e.,  $\mathbf{Z}_O = (1 - \alpha)\mathbf{Z} + \alpha \mathrm{GN}(\mathbf{Z}^{(0)},\mathbf{A})$ , as the final representation for each node.
- The output layer is a feed-forward layer for prediction, which maps the node representations  $\mathbf{Z}_O \in \mathbb{R}^{N \times d}$  into the predicted labels  $\hat{Y}$ . The prediction  $\hat{Y}$  will be used for computing the standard loss function of the form  $\sum_{u} l(\hat{y}_u, y_u)$ , where  $l(\cdot, \cdot)$  can be cross-entropy for classification tasks or mean square error for regression tasks.
We use different training schemes for graph datasets of different scales. For relatively small graphs (e.g., from 1K to 0.1M nodes), we use full-graph training; for larger graphs that cannot be trained in a single GPU, we consider mini-batch training. The training details are introduced below.
Full-Graph Training. For all the datasets presented in Table 2 and ogbn-arxiv, we use the full-graph training. In specific, we feed the whole graph into the model and compute the all-pair attention among all the nodes and predict their labels for loss computation. For inference, similarly, we use the whole graph as input and compute the metric based on the prediction for the validation/testing nodes.
Mini-Batch Training. For datasets (except ogbn-arxiv) presented in Table 3, due to the large graph sizes, we consider the mini-batch training scheme used in [57]. Specifically, at the beginning of each training epoch, we randomly shuffle the nodes and partition the nodes into mini-batches with the size  $B$ . Then in each iteration, we feed one mini-batch (the input graph among these  $B$  nodes are directly extracted by the subgraph of the original graph) into the model for loss computation on the training nodes within this mini-batch. Then for inference on ogbn-proteins, Amazon2M andpokec (with 0.1M to 1M nodes), following the pipeline in [19] for evaluation, we can feed the whole graph into the model using CPU, which computes the all-pair attention among all the nodes in the dataset. For the gigantic graph ogbn-papers100M that cannot be fed as whole into the CPU with moderate memory during inference, we adopt the mini-batch partition strategy used in training to reduce the overhead. Since our model allows large batch sizes, we found the mini-batch partition can yield decent performance. In specific, we set the batch size as 10K, 0.1M, 0.1M and 0.4M for ogbn-proteins, Amazon2M, pokec and ogbn-papers100M, respectively.
Evaluation Protocol. We follow the common practice for node classification tasks and set a fixed number of training epochs: 300 for medium-sized graphs, 1000 for large-sized graphs, and 50 for the extremely large graph ogbn-papers100M. The testing accuracy achieved by the model that reports
the highest result on the validation set is used for evaluation. We run each experiment with five independent trials using different initializations, and report the mean and variance of the metrics.
We use the model performance on the validation set for hyper-parameter settings of all models including the competitors. Unless otherwise stated, the hyper-parameters are selected using grid search with the searching space:
- learning rate within  $\{0.001, 0.005, 0.01, 0.05, 0.1\}$ ;
- weight decay within  $\{1e - 5, 1e - 4, 5e - 4, 1e - 3, 1e - 2\}$ ;
- hidden size within  $\{32, 64, 128, 256\}$ ;
- dropout ratio within  $\{0, 0.2, 0.3, 0.5\}$ ;
number of layers within  $\{1,2,3\}$
The searching space for other hyper-parameters that differ in each model is as follows:
GAT: head number  $\in \{4,8,16\}$
- SGC: hop number  $\in \{1,2,3\}$ .
- APPNP:  $\alpha \in \{0.1, 0.2, 0.5, 0.9\}$ .
- JKNet: jumping knowledge type  $\in$  {max, concatenation}.
- CPGNN: pretraining epochs  $\in$  {100, 200, 300}.
- SIGN: hop number  $\in$  {1,2,3}.
- GloGNN:  $\alpha \in [0,1]$ ,  $\beta_{1} \in \{0,1,10\}$ ,  $\beta_{2} \in \{01,1,10,100,1000\}$ ,  $\gamma \in [0,0.9]$ , norm_layers  $\in \{1,2,3\}$ ,  $K \in [1,6]$ .
- Graphormer: embedding dimension  $\{64, 128, 256, 512\}$ .
- GraphTrans: The dimension of the attentive module  $\{64, 128, 256\}$ , the dimension of the GCN  $\{16, 32, 64, 128\}$ .
- NodeFormer: rb_order  $\in$  {1, 2, 3}, hidden channels  $\in$  {16, 32, 64, 128}, edge regularization weight  $\in$  [0, 1], head number  $\in$  {1, 2, 3, 4}.
- SGFormer: number of global attention layers is fixed as 1, number of GCN layers  $\in \{1,2,3\}$ , weight  $\alpha$  within  $\{0.5,0.8\}$ .
In this section, we supplement more experimental results to further study what patterns the global attention learn from data. We visualize the attention matrices given by SGFormer on different datasets, and the results are shown in Fig. 6. We found that overall, the attention graphs are quite different from the input graphs, which implies that the global attention mechanism indeed captures certain informative patterns from data that are helpful for downstream prediction. Furthermore, since our model uses one-layer attention, the attention scores estimated by the model can be used for interpreting the influence among samples. For instance, as shown by the results, there are some sharp lines in the heat map which corresponds to the nodes that are recognized as important 'information source' by the model and allocated with large attention weights. These nodes are akin to some hubs in the network that exert much influence on other nodes.
(b) citesee
(c) pubmed
(d) actor
(e) ogbn-proteins
(f) Amazon2M
Figure 6: Visualization of the attention weights produced by SGFormer.