SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Mean and standard deviation (with five independent runs using random initializations) of testing accuracy on medium-sized node classification benchmarks. We annotate the node and edge number of each dataset and OOM indicates out-of-memory when training on a GPU with 24GB memory. We highlight the best model with purple and the runner-up with brown in each dataset.
Dataset	Cora	CiteSeer	PubMed	Actor	Squirrel	Chameleon	Deezer
# nodes	2,708	3,327	19,717	7,600	2223	890	28,281
# edges	5,278	4,552	44,324	29,926	46,998	8,854	92,752
GCN	81.6 ± 0.4	71.6 ± 0.4	78.8 ± 0.6	30.1 ± 0.2	38.6 ± 1.8	41.3 ± 3.0	62.7 ± 0.7
GAT	83.0 ± 0.7	72.1 ± 1.1	79.0 ± 0.4	29.8 ± 0.6	35.6 ± 2.1	39.2 ± 3.1	61.7 ± 0.8
SGC	80.1 ± 0.2	71.9 ± 0.1	78.7 ± 0.1	27.0 ± 0.9	39.3 ± 2.3	39.0 ± 3.3	62.3 ± 0.4
JKNet	81.8 ± 0.5	70.7 ± 0.7	78.8 ± 0.7	30.8 ± 0.7	39.4 ± 1.6	39.4 ± 3.8	61.5 ± 0.4
APPNP	83.3 ± 0.5	71.8 ± 0.5	80.1 ± 0.2	31.3 ± 1.5	35.3 ± 1.9	38.4 ± 3.5	66.1 ± 0.6
H2GCN	82.5 ± 0.8	71.4 ± 0.7	79.4 ± 0.4	34.4 ± 1.7	35.1 ± 1.2	38.1 ± 4.0	66.2 ± 0.8
SIGN	82.1 ± 0.3	72.4 ± 0.8	79.5 ± 0.5	36.5 ± 1.0	40.7 ± 2.5	41.7 ± 2.2	66.3 ± 0.3
CPGNN	80.8 ± 0.4	71.6 ± 0.4	78.5 ± 0.7	34.5 ± 0.7	38.9 ± 1.2	40.8 ± 2.0	65.8 ± 0.3
GloGNN	81.9 ± 0.4	72.1 ± 0.6	78.9 ± 0.4	36.4 ± 1.6	35.7 ± 1.3	40.2 ± 3.9	65.8 ± 0.8
GraphormerSMALL	OOM	OOM	OOM	OOM	OOM	OOM	OOM
GraphormerSMALLER	75.8 ± 1.1	65.6 ± 0.6	OOM	OOM	40.9 ± 2.5	41.9 ± 2.8	OOM
GraphormerULTRASSMALL	74.2 ± 0.9	63.6 ± 1.0	OOM	33.9 ± 1.4	39.9 ± 2.4	41.3 ± 2.8	OOM
GraphTransSMALL	80.7 ± 0.9	69.5 ± 0.7	OOM	32.6 ± 0.7	41.0 ± 2.8	42.8 ± 3.3	OOM
GraphTransULTRASSMALL	81.7 ± 0.6	70.2 ± 0.8	77.4 ± 0.5	32.1 ± 0.8	40.6 ± 2.4	42.2 ± 2.9	OOM
NodeFormer	82.2 ± 0.9	72.5 ± 1.1	79.9 ± 1.0	36.9 ± 1.0	38.5 ± 1.5	34.7 ± 4.1	66.4 ± 0.7
SGFormer	84.5 ± 0.8	72.6 ± 0.2	80.3 ± 0.6	37.9 ± 1.1	41.8 ± 2.2	44.9 ± 3.9	67.1 ± 1.1
scalable GNNs and Transformers. Later in Sec. 5.3 and 5.4, we compare the time/space efficiency and analyze the impact of several key components in our model, respectively. More detailed descriptions for datasets and implementation are deferred to Appendix B and C, respectively.
5.1 Results on Medium-sized Graphs
Setup. We first evaluate the model on commonly used graph datasets, including three citation networks cora, citeseer and pubmed, where the graphs have high homophily ratios, and four heterophilic graphs actor, squirrel, chameleon and deezer-europe, where neighboring nodes tend to have distinct labels. These graphs have 2K-30K nodes. For citation networks, we follow the semi-supervised setting of [23] for data splits. For actor and deezer-europe, we use the random splits of the benchmarking setting in [32]. For squirrel and chameleon, we use the splits proposed by a recent evaluation paper [38] that filters the overlapped nodes in the original datasets.
Competitors. Given the moderate sizes of graphs where most of existing models can scale smoothly, we compare with multiple sets of competitors from various aspects. Basically, we adopt standard GNNs including GCN [23], GAT [51] and SGC [52] as baselines. On top of this, we compare with advanced GNN models, including JKNet [59], APPNP [24], SIGN [40], H2GCN [72], CPGNN [71] and GloGNN [27]. In terms of Transformers, we mainly compare with the state-of-the-art models designed for node classification NodeFormer [57]. Furthermore, we adapt two powerful Transformers tailored for graph classification, i.e., Graphormer [63] and GraphTrans [58], for comparison. In particular, since the original implementations of these models are of large sizes and are difficult to scale on all node classification datasets considered in this paper, we adopt their smaller versions for our experiments. We use Graphormer<sub>SMALL</sub> (6 layers and 32 heads), Graphormer<sub>SMALLER</sub> (3 layers and 8 heads) and Graphormer<sub>ULTRASMALL</sub> (2 layers and 1 head). As for GraphTrans, we use GraphTrans<sub>SMALL</sub> (3 layers and 4 heads) and GraphTrans<sub>ULTRASMALL</sub> (2 layers and 1 head).
Results. Table 2 reports the results of all the models. We found that SGFormer significantly outperforms three standard GNNs (GCN, GAT and SGC) by a large margin, with up to  $25.9\%$  impv. over GCN on actor, which suggests that our one-layer global attention model is indeed effective despite its simplicity. Moreover, we observe that the relative improvements of SGFormer over three standard GNNs are overall more significant on heterophilic graphs actor, squirrel1, chameleon and deezer. The possible reason is that in such cases the global attention could help to filter out spurious edges from neighboring nodes of different classes and accommodate dis-connected yet informative nodes in the graph. Compared to other advanced GNNs and graph Transformers (particularly NodeFormer), the performance of SGFormer is highly competitive and even superior with significant gains over the runner-ups in most cases. These results serve as concrete evidence for verifying the efficacy of SGFormer as a powerful learner for node-level prediction. We also found that both Graphormer and GraphTrans suffer from serious over-fitting, due to their relatively complex
Table 3: Testing results (ROC-AUC for ogbn-proteins and Accuracy for other datasets) on large-sized node property prediction benchmarks. The training of NodeFormer on ogbn-papers100M cannot be finished within an acceptable time budget.
Method	ogbn-proteins	Amazon2m	pokec	ogbn-arxiv	ogbn-papers100M
# nodes	132,534	2,449,029	1,632,803	169,343	111,059,956
# edges	39,561,252	61,859,140	30,622,564	1,166,243	1,615,685,872
MLP	72.04 ± 0.48	63.46 ± 0.10	60.15 ± 0.03	55.50 ± 0.23	47.24 ± 0.31
GCN	72.51 ± 0.35	83.90 ± 0.10	62.31 ± 1.13	71.74 ± 0.29	OOM
SGC	70.31 ± 0.23	81.21 ± 0.12	52.03 ± 0.84	67.79 ± 0.27	63.29 ± 0.19
GCN-NSampler	73.51 ± 1.31	83.84 ± 0.42	63.75 ± 0.77	68.50 ± 0.23	62.04 ± 0.27
GAT-NSampler	74.63 ± 1.24	85.17 ± 0.32	62.32 ± 0.65	67.63 ± 0.23	63.47 ± 0.39
SIGN	71.24 ± 0.46	80.98 ± 0.31	68.01 ± 0.25	70.28 ± 0.25	65.11 ± 0.14
NodeFormer	77.45 ± 1.15	87.85 ± 0.24	70.32 ± 0.45	59.90 ± 0.42	-
SGFormer	79.53 ± 0.38	89.09 ± 0.10	73.76 ± 0.24	72.63 ± 0.13	66.01 ± 0.37
Table 4: Efficiency comparison of SGFormer and graph Transformer competitors w.r.t. training time (ms) per epoch, inference time (ms) and GPU memory (GB) costs on a Tesla T4. We use the small model versions of Graphormer and GraphTrans. The missing results are caused by out-of-memory.
Method	Cora	PubMed	Amazon2M
Tr (ms)	Inf (ms)	Mem (GB)	Tr (ms)	Inf (ms)	Mem (GB)	Tr (ms)	Inf (ms)	Mem (GB)
Graphormer	563.5	537.1	5.0	-	-	-	-	-	-
GraphTrans	160.4	40.2	3.8	-	-	-	-	-	-
NodeFormer	68.5	30.2	1.2	321.4	135.5	2.9	5369.5	1410.0	4.6
SGFormer	15.0	3.8	0.9	15.4	4.4	1.0	2481.4	382.5	2.7
Figure 2: Scalability test of training time per epoch and GPU memory usage w.r.t. graph sizes (a.k.a. node numbers). NodeFormer suffers out-of-memory when # nodes reaches more than  $30\mathrm{K}$
architectures and limited ratios of labeled nodes. In contrast, the simple and lightweight architecture of SGFormer leads to its better generalization ability given the limited supervision in these datasets.
5.2 Results on Large-sized Graphs
Setup. We further evaluate the model on several large-graph datasets where the numbers of nodes range from millions to billions. The citation network ogbn-arxiv and the protein interaction network ogbn-proteins contain 0.16M and 0.13M nodes, respectively, and these two graphs have different edge sparsity. We use the public splits in OGB [19]. Furthermore, the item co-occurrence network Amazon2M and the social network pokec consist of 2.0M and 1.6M node, respectively, and the latter is a heterophilic graph. We follow the splits used in the recent work [57] for Amazon2M and use random splits with the ratio 1:1:8 forpokec. The largest dataset we demonstrate is ogbn-papers100M where the node number reaches 0.1B and we also follow the public OGB splits.
Competitors. Due to the large graph sizes, most of the expressive GNNs and Transformers compared in Sec. 5.1 are hard to scale within acceptable computation budgets. Therefore, we compare with MLP, GCN and two scalable GNNs, SGC and SIGN. We also compare with GNNs using the neighbor sampling [65]: GCN-NSampler and GAT-NSampler. Our main competitor is NodeFormer [57], the recently proposed scalable graph Transformer with all-pair attention.
Results. Table 3 presents the experimental results. We found that SGFormer yields consistently superior results across five datasets, with significant performance improvements over GNN competitors. This suggests the effectiveness of the global attention that can learn implicit inter-dependencies among a large number of nodes beyond input structures. Furthermore, SGFormer outperforms NodeFormer by a clear margin across all the cases, which demonstrates the superiority of SGFormer that uses simpler architecture and achieves better performance on large graphs. For the largest dataset ogbn-papers100M where prior Transformer models fail to demonstrate, SGFormer scales smoothly
Figure 3: Testing scores and training time per epoch of SGFormer w.r.t. # attention layers. More results on more datasets are deferred to Appendix D.
Figure 4: Testing performance of NodeFormer, SGFormer w/o self-loop and SGFormer w/ Softmax w.r.t. the number of attention layers. The missing results are caused by out-of-memory.
with decent efficiency and yields highly competitive results. Specifically, SGFormer reaches the testing accuracy of 66.0 with consumption of about 3.5 hours and 23.0 GB memory on a single GPU for training. This result provides strong evidence that shows the promising power of SGFormer on extremely large graphs, producing superior performance with limited computation budget.
5.3 Efficiency and Scalability
We next provide more quantitative comparisons w.r.t. the efficiency and scalability of SGFormer with the graph Transformer competitors Graphormer, GraphTrans, and NodeFormer.
Efficiency Comparison. Table 4 reports the training time per epoch, inference time and GPU memory costs on cora, pubmed and Amazon2M. Since the common practice for model training in these datasets is to use a fixed number of training epochs, we report the training time per epoch here for comparing the training efficiency. We found that notably, SGFormer is orders-of-magnitude faster than other competitors. Compared to Graphormer and GraphTrans that require quadratic complexity for global attention and are difficult for scaling to graphs with thousands of nodes, SGFormer significantly reduces the memory costs due to the simple global attention of  $O(N)$  complexity. In terms of time costs, SGFormer yields  $38\mathrm{x} / 141\mathrm{x}$  training/inference speedup over Graphormer on cora, and  $20\mathrm{x} / 30\mathrm{x}$  training/inference speedup over NodeFormer on pubmed. This is mainly because Graphormer requires the computation of quadratic global attention that is time-consuming, and NodeFormer additionally employs the augmented loss and Gumbel tricks. On the large graph Amazon2M, where NodeFormer and SGFormer both leverage mini-batch training, SGFormer is  $2\mathrm{x}$  and  $4\mathrm{x}$  faster than NodeFormer in terms of training and inference, respectively.
Scalability Test. We further test the model's scalability w.r.t. the numbers of nodes within one computation batch. We adopt the Amazon2M dataset and randomly sample a subset of nodes with the node number ranging from 10K to 100K. For fair comparison, we use the same hidden size 256 for all the models. In Fig. 2, we can see that the time and memory costs of SGFormer both scale linearly w.r.t. graph sizes. When the node number goes up to 40K, the model (SGFormer w/ Softmax) that replaces our attention with the Softmax attention suffers out-of-memory and in contrast, SGFormer costs only 1.5GB memory.
We proceed to analyze the impact of model layers on the testing performance and further discuss the performance variation of other models w.r.t. the number of attention layers. Due to space limit, we defer more results to Appendix D.
How does one-layer global attention perform compared to multi-layer attentions of Eqn. 5? Fig. 3 presents the testing performance and training time per epoch of SGFormer when the layer number of global attention increases from one to more. We found that using more layers does
not contribute to considerable performance boost and instead leads to performance drop on the heterophilic graph actor. Even worse, multi-layer attention requires more training time costs. Notably, using one-layer attention of SGFormer can consistently yield highly competitive performance as the multi-layer attention. These results verify the effectiveness of our one-layer attention that has desirable expressivity and superior efficiency. In Sec. 6 we will shed more insights into why the one-layer all-pair attention is an expressive model.
How does one-layer global attention of other models perform compared to the multi-layer counterparts? Fig. 4 presents the performance of NodeFormer, SGFormer w/o self-loop (removing the self-loop propagation in Eqn. 5) and SGFormer w/ Softmax (replacing our attention by the Softmax attention), w.r.t. different numbers of attention layers in respective models. We found that using one-layer attention for these models can yield decent results in quite a few cases, which suggests that for other implementations of global attention, using a single-layer model also has potential for competitive performance. In some cases, for instance, NodeFormer produces unsatisfactory results on ogbn-proteins with one layer. This is possibly because NodeFormer couples the global attention and local propagation in each layer, and using the one-layer model could sacrifice the efficacy of the latter. On top of all of these, we can see that it can be a promising future direction for exploring effective shallow attention models that can work consistently and stably well. We also found when using deeper model depth, SGFormer w/o self-loop exhibits clear performance degradation and much worse than the results in Fig. 3, which suggests that the self-loop propagation in Eqn. 5 can help to maintain the competitiveness of SGFormer with multi-layer attentions.
6 Theoretical Justifications
To resolve lingering opaqueness on the rationale of our methodology, we attempt to shed some insights on why the one-layer attention model can be expressive enough for learning desirable representations from global interactions. We mainly consider the feature propagation step of one Transformer layer:
$$
\mathbf {z} _ {u} ^ {(k)} = (1 - \tau) \mathbf {z} _ {u} ^ {(k - 1)} + \tau \sum_ {v = 1} ^ {N} c _ {u v} ^ {(k)} \mathbf {z} _ {v} ^ {(k - 1)}, \tag {5}
$$
where  $c_{uv}^{(k)}$  is the attention score between node  $u$  and  $v$  at the  $k$ -th layer and  $0 < \tau \leq 1$  is a weight. For example, in the original Transformers [50],  $c_{uv}^{(k)}$  is the Softmax-based attention score. In our model Eqn. 3, we correspondingly have:  $c_{uu}^{(k)} = \frac{N + \tilde{\mathbf{q}}_u^\top\tilde{\mathbf{k}}_u}{N + \sum_w\tilde{\mathbf{q}}_u^\top\tilde{\mathbf{k}}_w}$  and  $c_{uv}^{(k)} = \frac{\tilde{\mathbf{q}}_u^\top\tilde{\mathbf{k}}_v}{N + \sum_w\tilde{\mathbf{q}}_u^\top\tilde{\mathbf{k}}_w}$  for  $v \neq u$ . Our analysis is mainly based on the interpretation of one Transformer layer as an optimization step for a principled graph signal denoising problem [47; 21; 13] with a particular objective. The denoising problem defined over  $N$  nodes in a system aims at smoothing the node features by adaptively absorbing the information from other nodes. The denoised features synthesize the initial information and the global information under the guidance of the given objective.
Figure 5: Illustration of the theoretical analysis showing the equivalence between the multi-layer and one-layer attention models. The one-layer attention model can produce the same effect on the smoothness criteria and help to save potential redundancy.
Theorem 1. For any given attention matrix  $\mathbf{C}^{(k)} = [c_{uv}^{(k)}]_{N\times N}$ , Eqn. 5 is equivalent to a gradient descent operation with step size  $\frac{\tau}{2\lambda}$  for an optimization problem with the cost function:
$$
\min  _ {\mathbf {Z}} \sum_ {u} \| \mathbf {z} _ {u} - \mathbf {z} _ {u} ^ {(k - 1)} \| _ {2} ^ {2} + \lambda \sum_ {u, v} c _ {u v} ^ {(k)} \| \mathbf {z} _ {u} - \mathbf {z} _ {v} \| _ {2} ^ {2}, \tag {6}
$$
where  $\lambda$  is a trading weight parameter for the local smoothness and global smoothness criteria [69].
For analysis and comparing the one-layer and multi-layer attention models, we extend our model to have arbitrary  $K$  layers, where each layer comprises an attention network of Eqn. 3 with independent parameterization, and use the superscript  $k$  to index each layer.
Eqn. 6 can be seen as a generalization of the Dirichlet energy [70] defined over a dense attention graph, which is also utilized as the optimization target in the energy-constrained graph diffusion framework for designing principled and scalable attention layers [54]. Specifically, the pair-wise attention score  $c_{uv}^{(k)}$  reflects the proximity between nodes  $u$  and  $v$ , and the attention matrix  $\mathbf{C}^{(k)}$  enforces adaptive denoising effect at the  $k$ -th layer. Since  $\mathbf{C}^{(k)}$  varies layer by layer, multi-layer attention models can be seen as a cascade of descent steps on layer-dependent denoising objectives. As we will show in the next result, the multi-layer model can be simplified as a single-layer one, while the latter contributes to the same denoising effect.
Theorem 2. For any  $K$ -layer attention model (where  $K$  is an arbitrary positive integer) with the layer-wise updating rule defined by Eqn. 5, there exists  $\mathbf{C}^{*} = [c_{uv}^{*}]_{N\times N}$  such that one gradient descent step for the optimization problem (from the initial embeddings  $\mathbf{Z}^{(0)} = [\mathbf{z}_u^{(0)}]_{u = 1}^N$ )
$$
\min  _ {\mathbf {Z}} \sum_ {u} \| \mathbf {z} _ {u} - \mathbf {z} _ {u} ^ {(0)} \| _ {2} ^ {2} + \lambda \sum_ {u, v} c _ {u v} ^ {*} \| \mathbf {z} _ {u} - \mathbf {z} _ {v} \| _ {2} ^ {2}, \tag {7}
$$
can yield the output embeddings  $\mathbf{Z}^{(K)} = [\mathbf{z}_u^{(K)}]_{u=1}^N$  of the  $K$ -layer model.
The above result indicates that for any multi-layer attention model, one can always find a single-layer attention model that behaves in the same way aggregating the global information. Moreover, the multi-layer model optimizes different objectives at each layer, which suggests potential redundancy compared to the one-layer model that pursues a fixed objective in one step. And, according to Theorem 1, the one-layer model contributes to a steepest decent step on this objective. We present an illustration of our theoretical results in Fig. 5.
7 Conclusions and Outlooks
This paper explores the potential of simple Transformer-style architectures for learning large-graph representations where the scalability challenge plays a bottleneck. Through extensive experiments across diverse node property prediction benchmarks whose graph sizes range from thousands to billions, we demonstrate that a one-layer attention model combined with a vanilla GCN can surprisingly produce highly competitive performance. The simple and lightweight architecture enables our model, dubbed as SGFormer, to scale smoothly to an extremely large graph with 0.1B nodes and yields  $30\mathrm{x}$  acceleration compared to state-of-the-art Transformers on medium-sized graphs. We also provide accompanying theoretical justification for our methodology. On top of our technical contributions, we believe the results in this paper could shed lights on a new promising direction for building powerful and scalable Transformers on large graphs, which is largely under-explored.
Future Works. One limitation of the present work lies in the tacit assumption that the training and testing data comes from the identical distribution. Though this assumption is commonly adopted for research on representation learning where the main focus is the expressiveness and representation power of the models, in practical scenarios, testing data may come from previously unseen distributions, resulting in the challenge of out-of-distribution learning. Promising future directions could be to explore the potential of graph Transformers in out-of-distribution generalization tasks where distribution shifts exist [56; 62], and the capabilities of graph Transformers for detecting out-of-distribution samples [53; 30]. We leave more studies along these under-explored, orthogonal areas as future works.
Furthermore, in this work, we follow the common practice and mainly focus on the benchmark settings for evaluating the model w.r.t. a fundamental challenge of learning representations on large graphs. While in principle Transformers can be applied for other graph-based tasks and more broadly domain-specific applications, such as recommender systems [55], circuit designs [6] and combinatorial optimization [28; 29], these scenarios often require extra task-dependent designs. We thereby leave exploration along this orthogonal direction aiming to unlock the potential of graph Transformers for various specific applications as future works.
[1] Anson Bastos, Abhishek Nadgeri, Kuldeep Singh, Hiroki Kanezashi, Toyotaro Suzumura, and Isaiah Onando Mulang'. How expressive are transformers in spectral domain for graphs? Trans. Mach. Learn. Res., 2022, 2022.
[69] Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In Advances in Neural Information Processing Systems, pages 321-328, 2004.
[70] Dengyong Zhou and Bernhard Scholkopf. Regularization on discrete spaces. In Pattern Recognition, 27th DAGM Symposium, volume 3663, pages 361-368. Springer, 2005.
[71] Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In AAAI Conference on Artificial Intelligence, 2021.
[72] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In Advances in Neural Information Processing Systems, 2020.
A Proof for Technical Results
A.1 Proof for Theorem 1
Theorem 1. For a given attention matrix  $\mathbf{C}^{(k)} = [c_{uv}^{(k)}]_{N\times N}$ , Eqn. 5 is equivalent to a gradient descent with step size  $\frac{\tau}{2\lambda}$  for an optimization problem with the cost function:
$$
\min  _ {\mathbf {Z}} \sum_ {u} \| \mathbf {z} _ {u} - \mathbf {z} _ {u} ^ {(k - 1)} \| _ {2} ^ {2} + \lambda \sum_ {u, v} c _ {u v} ^ {(k)} \| \mathbf {z} _ {u} - \mathbf {z} _ {v} \| _ {2} ^ {2}, \tag {8}
$$
where  $\lambda$  is a trading weight for the local smoothness and global smoothness criteria [69].
Proof. We denote the cost function at the  $k$ -th layer as
$$
E (\mathbf {Z}; \mathbf {Z} ^ {(k - 1)}) = \sum_ {u} \| \mathbf {z} _ {u} - \mathbf {z} _ {u} ^ {(k - 1)} \| _ {2} ^ {2} + \lambda \sum_ {u, v} c _ {u v} ^ {(k)} \| \mathbf {z} _ {u} - \mathbf {z} _ {v} \| _ {2} ^ {2}. \tag {9}
$$
The gradient of  $E(\mathbf{Z}; \mathbf{Z}^{(k-1)})$  w.r.t.  $\mathbf{z}_u$  can be computed by
$$
\frac {\partial E (\mathbf {Z} ; \mathbf {Z} ^ {(k - 1)})}{\partial \mathbf {z} _ {u}} = 2 \left(\mathbf {z} _ {u} - \mathbf {z} _ {u} ^ {(k - 1)}\right) + 2 \lambda \sum_ {u, v} c _ {u v} ^ {(k)} \left(\mathbf {z} _ {u} - \mathbf {z} _ {v}\right). \tag {10}
$$
The gradient descent with step size  $\frac{\tau}{2\lambda}$  that minimizes the cost function  $E(\mathbf{Z};\mathbf{Z}^{(k - 1)})$  at the current layer is