SlowGuess/NeurIPS_2023_OCR huggingface dataset:
TpuGraphs is an internal large scale graph regression dataset, whose goal is to predict an execution time of an XLA's HLO graph with a specific compiler configuration on a Tensor Processing Unit (TPU). XLA [28] is a production backend compiler for various machine learning frameworks, including TensorFlow, PyTorch, and JAX. In this particular dataset, the compiler configuration controls physical layouts of tensors in the graph, and the runtime is measured on TPU v3 [15]. This dataset cares more about the ranking of the configurations for each graph than the absolute runtimes, since the ultimate goal is to use a model to select the best configuration for each graph.4 TpuGraphs
4We have made public a dataset [24] that closely parallels our internal dataset.
Table 1: Test accuracy on MalNet-Tiny and MalNet-Large. We report the standard deviation over five runs. GST+EFD achieves better accuracy than Full Graph Training, and GST, while being much more memory efficient and computationally faster.
Dataset Backbone	GCN	MalNet-Tiny SAGE	GraphGPS	GCN	MalNet-Large SAGE	GraphGPS
Full Graph Training	87.84±1.37	88.08±1.68	90.82±0.59	OOM	OOM	OOM
GST	88.26±0.80	88.42±1.03	91.03±0.81	88.35±1.14	88.62±0.82	91.39±0.85
GST-One	71.62±3.85	72.64±4.73	77.63±3.15	60.41±6.29	57.13±7.36	66.82±4.71
GST+E	86.53±1.18	86.82±0.93	89.75±0.89	48.42±6.61	43.28±7.01	62.47±3.19
GST+EF	87.67±0.78	87.83±0.81	90.52±0.71	84.83±0.96	85.26±0.87	91.33±0.65
GST+ED	88.18±0.48	88.50±0.74	90.96±0.68	82.17±4.74	71.83±6.31	89.46±1.36
GST+EFD	88.78±0.45	89.24±0.53	92.46±0.66	89.67±0.71	89.78±0.68	92.52±0.58
is similar to the dataset used in [17], but the runtime prediction is at the entire graph level rather than at the kernel (subgraph) level. TpuGraphs contains 5,153 HLO graphs and a total of 757,375 unique pairs of graphs and configurations. The average graph size is  $38\mathrm{k}$ , and the maximum is  $615\mathrm{k}$ . From the perspective of GST, a graph together with a configuration defines one  $\mathcal{G}^{(i)}$  because the configuration is featurized as parts of input node features to the GNN.
Please refer to Table 4 in Appendix for detailed statistics.
Methods. We test combinations of the following proposed techniques and some baselines. (1) Full Graph Training: we train on all graphs in their original scale without applying any partitioning beforehand. (2) GST-One: we partition the original graph into a collection of graph segments  $\mathcal{G}^{(i)} \approx \bigoplus \mathcal{G}_j^{(i)}$ , but we randomly select only one segment  $\mathcal{G}_j^{(i)}$  for each graph to train every iteration. (3) GST: following the general GST framework described in Algorithm 1, we replace ProduceEmbedding( $\cdot$ ) by using the same feature encoder  $F$  to forward all the segments in  $\{\mathcal{G}_j^{(i)}\}_{j \notin S^{(i)}}$  without storing any intermediate activation. We set  $S^{(i)} = 1$  in our experiments. (4) E: we introduce an embedding table  $\tilde{\pmb{h}}_j^{(i)} = \mathcal{T}(i,j)$  to store the historical embedding of each graph segment, and we fetch the embedding from  $\mathcal{T}$  if we do not need to calculate gradient for the corresponding segment. (5) F: in addition to introducing the embedding table  $\mathcal{T}$ , we finetune the prediction head  $F'$  with all up-to-date segment embeddings at the end of training. (6) D: we apply SED defined in Eq. 1 during training.
When these techniques are combined, we concatenate the acronyms with a "+" to GST as an abbreviation. We conduct all the experiments on MalNet with a single NVIDIA-V100 GPU with 16GB of memory, and four NVIDIA-V100 GPUs (for data parallelism) with 16GB of memory for TpuGraphs. Please refer to Appendix B for additional implementation details.
5.2 Empirical Results on MalNet
To demonstrate the general applicability of our proposed GST framework, we consider three backbones, namely, GCN [19], SAGE [11], and GraphGPS [25]. GCN and SAGE are two popular GNN architectures. GraphGPS is a Graph Transformer that recently achieves state-of-the-art performance on many graph-level tasks, but is well-known for its issue on scalability. We report the top-1 test accuracy of various methods on MalNet-Tiny and MalNet-Large in Table 1. We include MalNet-Tiny in this study because its graphs are relatively small so that it is still possible to run Full Graph Training.
Notably, we observe that GST slightly outperforms Full Graph Training in terms of test accuracy on MalNet-Tiny. GST has exactly the same number of weight parameters with Full Graph Training. This implies that GST potentially has a better hierarchical graph pooling mechanism that leads to better generalization. As we step from MalNet-Tiny to MalNet-Large, Full Graph Training strategy can no longer fit the large graphs on a GPU, so we report OOM in the table. GST's estimation on graph segment embeddings  $\bar{h}_j^{(i)}$  that do not require gradients is accurate, and thus does not suffer from staleness issues. Therefore, we use GST as an estimation for the performance of Full Graph Training on MalNet-Large.
Naively training on only one graph segment (GST-One) yields inferior performance than Full Graph Training and GST, showing that it is essential to aggregate embeddings from all graph segments.
Figure 2: Accuracy curve on MalNet-Large of GST+EFD with SAGE backbone. We start Prediction Head Finetuning at epoch 600.
Figure 3: Ablation study on the keep ratio  $p$  in SED. We report test accuracy of GST+EFD with SAGE backbone on MalNet-Large for 5 runs.
Figure 4: Ablation study on maximum segment size. We report test accuracy of GST+EFD with SAGE backbone on MalNet-Large for 5 runs.
Solely introducing the historical embedding table (GST+E) significantly deteriorates the optimization due to the staleness issue. Each of the proposed techniques (Prediction Head Finetuning and SED) individually is beneficial in combating the staleness issue. The combination of our two techniques (GST+EFD) achieves the best performance, slightly outperforming GST by another  $1 - 2\%$  in terms of final evaluation metric.
5.3 Empirical results on TpuGraphs dataset
Table 2: Training and test ordered pair accuracy Table 3: Runtime analysis (average training time (OPA) on TpuGraphs Dataset. per iteration in ms) on MalNet-Large dataset.
Train OPA	Test OPA
Full Graph Training	OOM	OOM
GST	87.43	86.84
GST-One	67.19	81.03
GST+E	74.02	82.62
GST+EFD	71.40	89.20
GCN	SAGE	GraphGPS
Full Graph Training	OOM	OOM	OOM
GST	720.8	706.3	1285.7
GST-One	242.2	239.6	441.4
GST+E	258.4	253.6	451.9
GST+EFD	247.9	244.7	448.2
As mentioned in Section 5.1, we care more about the ranking of configurations for each graph than the absolute runtimes in this dataset. Thus, we report the ordered pair accuracy (OPA) averaged over all computational graphs in Table 2. OPA metric is defined as:
$$
\mathrm {O P A} (y, \widehat {y}) = \frac {\sum_ {i} \sum_ {j} \mathbb {I} [ \widehat {y} _ {i} > \widehat {y} _ {j} ] \cdot \mathbb {I} [ y _ {i} > y _ {j} ]}{\sum_ {i} \sum_ {j} \mathbb {I} [ y _ {i} > y _ {j} ]}
$$
With some compiler domain knowledge, we found it better to predict each graph segment's runtime first and then aggregate them using sum pooling. This means the prediction head is part of  $F$ , and  $F'$  is simply a summation function. Since there are no learnable weights in  $F'$ , we omit Prediction Head Finetuning in this experiment, and GST+EFD in Table 2 excludes the finetuning stage. We observe a clear tradeoff between fitting training examples and generalization. First, GST has a much higher training OPA than the other methods, indicating accurate estimation on segments that do not require gradient is essential for training OPA. Training with one segment only or with the embedding table yields lower training OPA, and consequently lower test OPA as well. SED in GST+EFD functions as a regularization technique. Although it slightly lowers the training OPA compared to GST+E, it achieves better test OPA, even than GST, due to bias mitigation.
Effect of finetuning. We visualize the training/test accuracy curve of GST+EFD over time in Figure 2. The staleness introduced by historical embeddings drastically hurts generalization, as shown for the first 600 epochs. We start finetuning at epoch 600, and the gap between training and test accuracy decreases by a large margin instantly.
Ablation study on segment dropout ratio. To analyze the effect of the keep ratio  $p$  in SED, we vary its value from 0 to 1 and visualize the results in Figure 3. When  $p = 1$ , GST+EFD degrades back to using the historical embedding table without SED, as the performance decreases due to staleness. When  $p = 0$ , GST+EFD becomes GST-One, where we drop all the stale historical embeddings. This extreme case introduces too heavy regularization that impedes the model from fitting the training data, leading to a decrease in test performance ultimately. We found that  $p = 0.5$  achieves a satisfactory tradeoff between fitting the training data and adding a proper amount of regularization.
Ablation study on segment size. We also alter the maximum segment size and visualize the results in Figure 4. A smaller maximum segment size will result in much more number of segments. Interestingly, we found that the proposed GST+EFD is very robust to the choice of the maximum segment size, as long as the segment size is reasonably large.
Ablation study on partition algorithms. Please refer to Appendix C.
5.5 Runtime Analysis
Next, we empirically compare runtime of different variants under the proposed GST framework. We summarize an average time for one forward-backward pass during training on MalNet-Large dataset in Table 3. Since GST runs inference for the graph segments that do not require gradients, the runtime of GST is significantly higher than others'. We also found that  $\mathrm{GST + E}$  's and  $\mathrm{GST + EFD}$  's runtime are very close to GST-One's; this means the overhead of fetching embeddings from the embedding table  $\mathcal{T}$  is minimal. Moreover, GST+EFD's runtime is slightly lower than  $\mathrm{GST + E}$  's because in the implementation, we can skip the fetching process if an embedding is set to be dropped. This result demonstrates that our proposed GST+EFD not only is efficient in terms of memory usage but also reduces training time significantly.
Graph property prediction. In the context of graph property prediction, a model must predict a certain characteristic associated with the whole graph. Standard graph neural networks produce node embeddings as outputs [19, 11, 34]. To create a graph embedding (a vector representing the entire graph) out of the node embeddings, pooling methods are usually applied at the end. Common approaches to this problem include simply summing up or averaging all the node embeddings [8], or introducing a "virtual node" connected to all the other nodes [23]. Fully-connected Graph Transformer was recently proposed with an outstanding success on existing graph property prediction benchmarks [35, 25]. However, the fully-connected attention matrix limits the applicability of Graph Transformer to only small graphs with limited number of nodes [26].
GNN with graph partitioning. ClusterGCN [7] is designed for node-level tasks by training on graph segments. It partitions a graph into graph segments and randomly selects graph segments to form a minibatch during training. ROC [14], PipeGCN [29] and BNS-GCN [30] achieve distributed node-level GCN training through partitioning a graph into small segments such that each could be fitted into a single GPU memory, and then training multiple segments in parallel. All the above graph partitioning techniques for GNNs rely on the fact that an ego-subgraph (a subgraph centered around a node) contains sufficient information to make a prediction for the centered node. This is not true for graph property prediction tasks where we need to aggregate information from the whole graph to make an accurate prediction.
GNN with historical embeddings. The idea of historical embeddings was first introduced in VR-GCN [4], which uses historical embeddings to control the variance of neighborhood sampling. GNNAutoScale [9] incorporates historical embeddings to recover a more accurate neighborhood estimation in a scalable fashion. Developed upon GNNAutoScale, Yu et al. [37] uses a momentum step to incorporate historical embeddings when updating feature representations to further alleviate the staleness issue. These prior works maintain a historical embedding for each node because they consider node-level tasks. As we consider graph property prediction tasks, we record a historical embedding for each graph segment rather than each node.
We study how to train a GNN model for large graph property prediction tasks. We propose Graph Segment Training (GST), a general framework for learning large graph property prediction tasks with a constant memory footprint. We further introduce a historical embedding table to efficiently produce embeddings for graph segments that do not require gradients, and design two novel techniques — Prediction Head Finetuning and Stale Embedding Dropout — to mitigate the staleness issue. In conclusion, we believe that our proposed method is a step toward making graph property prediction learning more practical and scalable.
KC and JL acknowledge the support of DARPA under Nos. HR00112190039 (TAMI), N660011924033 (MCS); ARO under Nos. W911NF-16-1-0342 (MURI), W911NF-16-1-0171 (DURIP); NSF under Nos. OAC-1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), NIH under No. 3U54HG010426-04S1 (HuBMAP), Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba. CM's contributions were partially supported by ACE, one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA and NSF under grant CCF-2316233. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
[1] Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143-153, 2019.
[2] Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Gunnemann. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2464-2473, 2020.
[3] Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and Kevin Murphy. Machine learning on graphs: A model and comprehensive taxonomy. Journal of Machine Learning Research, 23(89):1-64, 2022.
[4] Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. In International Conference on Machine Learning, pages 942-950. PMLR, 2018.
[5] Jie Chen, Tengfei Ma, and Cao Xiao. Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations, 2018.
[6] Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Learning to Optimize Tensor Programs. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NeurIPS'18, page 3393-3404, Red Hook, NY, USA, 2018. Curran Associates Inc.
[7] Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 257-266, 2019.
[8] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.
[9] Matthias Fey, Jan E Lenssen, Frank Weichert, and Jure Leskovec. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In International Conference on Machine Learning, pages 3294-3304. PMLR, 2021.
Theorem A.1. Under proper a condition that  $W\cdot \tilde{\pmb{h}}_j^{(i)}\approx 0$  and  $W\cdot \pmb {h}_j^{(i)}\approx 0$  , where  $W$  is the first linear transformation in  $F^{\prime}$  , SED with a keep ratio p ensures to reduce bias term introduced by historical embeddings by a factor of  $p$  , while introducing another regularization term.
Proof. Let  $\delta^{(i)}\triangleq \pmb{h}_s^{(i)}\oplus \tilde{\pmb{h}}_j^{(i)} - \bigoplus_{j}h_{j}^{(i)}$  be the perturbation on the graph embedding. We use ET to denote using the embedding table without applying SED. For GST+E, we have
$$
\delta_ {j} ^ {(i) ^ {\mathrm {E T}}} = \left\{ \begin{array}{l l} 0 & \text {w i t h p r o b .} \frac {S ^ {(i)}}{J ^ {(i)}} \\ \tilde {\boldsymbol {h}} _ {j} ^ {(i)} - \boldsymbol {h} _ {j} ^ {(i)} & \text {w i t h p r o b .} \frac {J ^ {(i)} - S ^ {(i)}}{J ^ {(i)}} \end{array} \right.
$$
The randomness above comes from the fact that each segment  $\mathcal{G}_j^{(i)}$  is selected for backpropagation with probability  $\frac{S^{(i)}}{J^{(i)}}$ .
For SED, there are two folds of randomness when training on graph  $\mathcal{G}^{(i)}$ : randomly selecting  $S^{(i)}$  segments to train and randomly select stale embedding  $\tilde{\pmb{h}}_j^{(i)}$  to drop. Thus we can rewrite  $\delta_j^{(i)}$  as
$$
\delta_ {j} ^ {(i) \text {S E D}} = \left\{ \begin{array}{l l} \frac {(1 - p) (J ^ {(i)} - S ^ {(i)})}{S ^ {(i)}} \boldsymbol {h} _ {j} ^ {(i)} & \text {w i t h p r o b .} \frac {S ^ {(i)}}{J ^ {(i)}} \\ - \boldsymbol {h} _ {j} ^ {(i)} & \text {w i t h p r o b .} \frac {(1 - p) (J ^ {(i)} - S ^ {(i)})}{J ^ {(i)}} \\ \tilde {\boldsymbol {h}} _ {j} ^ {(i)} - \boldsymbol {h} _ {j} ^ {(i)} & \text {w i t h p r o b .} \frac {p (J ^ {(i)} - S ^ {(i)})}{J ^ {(i)}} \end{array} \right.
$$
We apply Taylor expansion around  $\delta_j^{(i)} = 0$  on the final loss to analyze the effect of this perturbation. In Section A.2 of Wei et al. [31], when  $W\cdot \tilde{\pmb{h}}_j^{(i)}\approx 0$  and  $W\cdot \pmb {h}_j^{(i)}\approx 0$  , the perturbation of  $\delta^{(i)}$  to the loss function might not be too large, so it supports the use of Taylor Expansion.
$$
\begin{array}{l} \mathcal {L} \left(F ^ {\prime} \left(\boldsymbol {h} _ {s} ^ {(i)} \bigoplus \tilde {\boldsymbol {h}} _ {j} ^ {(i)}\right)\right) - \mathcal {L} \left(F ^ {\prime} \left(\bigoplus \boldsymbol {h} _ {j} ^ {(i)}\right)\right) \\ = \mathcal {L} \left(F ^ {\prime} \left(\bigoplus \left(\boldsymbol {h} _ {j} ^ {(i)} + \delta_ {j} ^ {(i)}\right)\right)\right) - \mathcal {L} \left(F ^ {\prime} \left(\bigoplus \boldsymbol {h} _ {j} ^ {(i)}\right)\right) \\ \approx \sum_ {j} D _ {\boldsymbol {h} _ {j} ^ {(i)}} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ] \delta_ {j} ^ {(i)} + \frac {1}{2} \delta_ {j} ^ {(i)} ^ {\top} (D _ {\boldsymbol {h} _ {j} ^ {(i)}} ^ {2} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ]) \delta_ {j} ^ {(i)} \\ \end{array}
$$
Note that we randomly select segments with index  $s$  during training, we can then derive an approximation of the expected difference during training as
$$
\begin{array}{l} \mathbb {E} _ {s} \mathcal {L} \left(F ^ {\prime} \left(\boldsymbol {h} _ {s} ^ {(i)} \bigoplus \tilde {\boldsymbol {h}} _ {j} ^ {(i)}\right)\right) - \mathcal {L} \left(F ^ {\prime} \left(\bigoplus \boldsymbol {h} _ {j} ^ {(i)}\right)\right) \tag {3} \\ \approx \sum_ {j} \mathbb {E} _ {\delta_ {j} ^ {(i)}} \underbrace {D _ {\boldsymbol {h} _ {j} ^ {(i)}} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ] \delta_ {j} ^ {(i)}} _ {B} + \underbrace {\frac {1}{2} \delta_ {j} ^ {(i)} ^ {\top} (D _ {\boldsymbol {h} _ {j} ^ {(i)}} ^ {2} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ]) \delta_ {j} ^ {(i)}} _ {R} \\ \end{array}
$$
We can then compare the effect of SED by substituting the two versions of  $\delta_j^{(i)}$  into Eq. 3.
So for the first term, we have
$$
\begin{array}{l} \mathbb {E} _ {\delta_ {j} ^ {(i)} ^ {\mathrm {E T}}} [ B ] = \langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ], \mathbb {E} \delta_ {j} ^ {(i)} \rangle \\ = \left\langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} \left(\mathcal {L} \circ F ^ {\prime}\right) \left[ \boldsymbol {h} _ {j} ^ {(i)} \right], \frac {J ^ {(i)} - S ^ {(i)}}{J ^ {(i)}} \mathbb {E} \left(\tilde {\boldsymbol {h}} _ {j} ^ {(i)} - \boldsymbol {h} _ {j} ^ {(i)}\right) \right\rangle \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\delta_ {j} ^ {(i)}} ^ {\mathrm {S E D}} [ B ] = \langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ], \mathbb {E} \delta_ {j} ^ {(i)} \rangle \\ = \left\langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} \left(\mathcal {L} \circ F ^ {\prime}\right) \left[ \boldsymbol {h} _ {j} ^ {(i)} \right], \frac {J ^ {(i)} - S ^ {(i)}}{J ^ {(i)}} \mathbb {E} \left(\tilde {\boldsymbol {h}} _ {j} ^ {(i)} - \boldsymbol {h} _ {j} ^ {(i)}\right) * p \right\rangle \\ \end{array}
$$
whereas for the second term, we have
$$
\begin{array}{l} \mathbb {E} _ {\delta_ {j} ^ {(i) \mathrm {E T}}} [ R ] = \langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} ^ {2} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ], \frac {\mathbb {E} \delta_ {j} ^ {(i)} \delta_ {j} ^ {(i) ^ {\top}}}{2} \rangle \\ = \left\langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} ^ {2} \left(\mathcal {L} \circ F ^ {\prime}\right) [ \boldsymbol {h} _ {j} ^ {(i)} ], \frac {J ^ {(i)} - S ^ {(i)}}{2 J ^ {(i)}} \left(\tilde {\boldsymbol {h}} _ {j} ^ {(i)} - \boldsymbol {h} _ {j} ^ {(i)}\right) \odot^ {2} \right\rangle \\ \mathbb {E} _ {\delta_ {j} ^ {(i)} ^ {\mathrm {S E D}}} [ R ] = \left\langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} ^ {2} (\mathcal {L} \circ F ^ {\prime}) [ \boldsymbol {h} _ {j} ^ {(i)} ], \frac {\mathbb {E} \delta_ {j} ^ {(i)} \delta_ {j} ^ {(i) ^ {\top}}}{2} \right\rangle \\ = \left\langle D _ {\boldsymbol {h} _ {j} ^ {(i)}} ^ {2} \left(\mathcal {L} \circ F ^ {\prime}\right) \left[ \boldsymbol {h} _ {j} ^ {(i)} \right], \left(\frac {\left(J ^ {(i)} - S ^ {(i)}\right) p}{2 J ^ {(i)}} \left(\tilde {\boldsymbol {h}} _ {j} ^ {(i)} - \boldsymbol {h} _ {j} ^ {(i)}\right) \odot {} ^ {2} \right. \right. \\ + \frac {\left(J ^ {(i)} - S ^ {(i)}\right) (1 - p) \left(J ^ {(i)} - p J ^ {(i)} + p S ^ {(i)}\right)}{2 J ^ {(i)} S ^ {(i)}} \left. \boldsymbol {h} _ {j} ^ {(i) \odot 2}\right) \rangle \\ \end{array}
$$
It is easy to check that the statement satisfies given the value calculated.
B.1 Missing Algorithm
Algorithm 2 Pipeline of GST+EFD
Require: A preprocessed training graph dataset  $\mathcal{D}_{\mathrm{train}} = \{(\bigoplus \mathcal{G}_j^{(i)},y^{(i)})\}_{i = 1}^n$ . A parameterized backbone  $F$  and a prediction head  $F^{\prime}$ . A historical segment embedding table  $\mathcal{T}$ .
1: for  $t = 1$  to  $T_0$  do
2:  $\mathcal{B}\gets$  SampleMiniBatch(Dtrain)
3: for  $(\mathcal{G}^{(i)},y^{(i)})$  in  $\mathcal{B}$  do
4:  $\{\mathcal{G}_s^{(i)}\}_{s\in \mathcal{S}^{(i)}}\gets$  SampleGraphSegments  $(\mathcal{G}^{(i)})$
5:  $\tilde{\pmb{h}}_j^{(i)}\gets \mathcal{T}.\mathrm{LookUp}\left((i,j)\right)$  for  $j\notin S^{(i)}$
6:  $\pmb{h}_s^{(i)} \gets F(\mathcal{G}_s^{(i)})$  for  $s \in S^{(i)}$ .
7:  $\mathcal{T}$  .InsertOrUpdate((i,s),h(s))
8: end for
9: SGD on  $\frac{1}{|\mathcal{B}|}\sum_{i}\mathcal{L}\left(F^{\prime}(\eta_{s}^{(i)}\cdot \pmb{h}_{s}^{(i)}\bigoplus (\eta_{j}^{(i)}\cdot \tilde{\pmb{h}}_{j}^{(i)}),y^{(i)}\right)$  {SED with  $\eta$  defined in Equation 1}
0: end for
11: # Prediction Head Finetuning
12:  $\mathcal{T}$ .InsertOrUpdate((i,j),F( $\mathcal{G}_j^{(i)}$ )) for every  $\mathcal{G}_j^{(i)}$
13: for  $t = T_0$  to  $T_1$  do
14: for  $\mathcal{G}^{(i)}$  in SampleMiniBatch  $(\mathcal{D}_{\mathrm{train}})$  do
15:  $\bar{\pmb{h}}_j^{(i)}\gets \mathcal{T}.\mathrm{LookUp}\left((i,j)\right)$  for  $j\leq J^{(i)}$
16: end for
17: SGD with loss  $\leftarrow \frac{1}{|\mathcal{B}|}\sum_{i}\mathcal{L}\left(F^{\prime}(\bigoplus \bar{\boldsymbol{h}}_{j}^{(i)})\right)$
18: end for