SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7: The source code and major hyper-parameters used in the baselines.
Method	Hyper-parameter	Code
DeepWalk	The dimension of output representations is 64, walk length = 10, number of walks = 80	https://github.com/shenweichen/GraphEmbedding
struc2vec	The dimension of output representations is 32, walk length = 80, number of walks = 10, window size = 5	https://github.com/leoribeiro/struc2vec
DGI	512 hidden units per GNN layer, learning rate = 0.001	https://github.com/PetarV-/DGI
GAE	32 hidden units per GNN layer, learning rate = 0.01	https://github.com/zfjsail/gae-pytorch
graph2vec	The dimension of output representations is 128	https://github.com/benedekrozemberczki/graph2vec
InfoGraph	32 hidden units per GNN layer, 5 layers	https://github.com/fanyun-sun/InfoGraph
DGCNN	32 hidden units per GNN layer, learning rate = 0.001, batch size = 50	https://github.com/leftthomas/DGCNN
GIN	64 hidden units per GNN layer, 5 layers, learning rate = 0.01, sum pooling	https://github.com/weihua916/powerful-gnns
GraphCL	300 hidden units per GNN layer, 5 layers, learning rate = 0.001	https://github.com/Shen-Lab/GraphCL
GCC	64 hidden units per GNN layer, 5 layers, learning rate = 0.005, number of samples per epoch = 20000	https://github.com/THUDM/GCC
H Additional Experimental Results and Analysis
Analysis of ablation studies of APT. Table 1 presents the node classification results of APT and its 4 variants, i.e., APT-G, APT-P, APT-R, and APT-L2. We here analyze the performance of these ablation studies from the following three points of view: (1) APT-L2 and APT, the variants with the proximity regularization w.r.t old knowledge, perform the best in most cases. This suggests that catastrophic forgetting of previous trained graphs could occur during pre-training, and it is necessary to add the proximity regularization to prevent this; (2) Sometimes APT-R (the variant without proximity term) achieves the best performance on the test data dd687 and cornell. One potential reason is that the last data to be selected for pre-training is dblp, while the test data dd687 and cornell exhibit some properties similar to dblp. Therefore, even excluding the proximity term, the pre-training model can still remember knowledge from the last pre-trained data, and achieve good performance on the test data that is similar to the last pre-trained data; (3) APT-G and APT-P present suboptimal performance among all variants, which suggests the utility of graph properties and predictive uncertainty.
Impact of five graph properties combination. We study the effect of the strategy of utilizing only one graph property in Table 8. We find that the five properties used in our model are all indispensable, and the most important one probably varies for different datasets. That's why we choose to combine all graph properties.
Moreover, these case studies may provide some clues of how to select pre-training graphs when some knowledge of the downstream tasks is known. For example, if the downstream dataset is extremely dense (like imdb-binary), the density property dominates among the selection criteria (such that the probability of encountering very dense out-of-distribution samples during testing can be reduced). If the entropy of downstream dataset is very high (like brazil), it is perhaps better to choose graphs with high entropy for pre-training. But still, when the downstream task is unknown, using the combination of five metrics often leads to the most satisfactory and robust results.
Table 8: The effect of different graph properties on downstream performance (micro F1 is reported) under APT-L2 (fine-tune). The last row is our strategy of combining all the graph properties, and each of the first five rows is the strategy of only utilizing one graph property.
Dataset Method	node classification	graph classification
brazil	dd242	dd68	dd687	wisconsin	cornell	cora	pubmed	imdb-binary	dd	msrc-21
Entropy	80.04(2.15)	25.79(0.94)	16.31(0.81)	11.08(0.82)	67.01(2.00)	52.80(2.46)	45.41(0.85)	50.85(0.19)	76.78(0.84)	75.56(0.84)	24.34(1.50)
Density	79.23(1.92)	27.29(0.62)	19.89(0.95)	12.22(1.06)	65.58(1.87)	51.15(1.59)	46.18(0.71)	50.74(0.15)	77.20(0.66)	75.29(0.54)	24.20(1.31)
Average degree	79.22(1.65)	24.99(0.67)	16.56(1.01)	11.67(1.18)	67.02(1.86)	51.43(4.16)	46.38(0.48)	50.99(0.31)	76.87(0.93)	75.46(0.53)	25.14(1.54)
Degree variance	78.44(2.24)	24.94(0.61)	16.62(1.04)	11.51(1.17)	65.65(1.28)	50.45(2.14)	45.76(0.65)	50.70(0.21)	76.39(1.04)	75.47(0.67)	25.22(1.51)
Scale-free exponent	79.70(2.71)	24.94(0.68)	17.26(0.63)	12.03(1.41)	64.77(2.31)	51.37(2.70)	45.18(0.52)	50.84(0.26)	75.24(0.62)	75.52(1.24)	23.19(1.39)
Our combination	78.75(1.63)	24.62(0.90)	17.83(1.35)	12.26(0.78)	67.04(1.50)	52.94(1.95)	47.48(0.46)	51.25(0.21)	75.93(0.84)	75.58(1.06)	25.58(1.57)
Analysis of the selected graphs. The data sequentially selected via our graph selector are uillinois, soc-sign0811, msu, michigan, wiki-vote, soc-sign0902 and dblp. To further analyze why these graphs are chosen, we present their detailed structural properties in Table 4 in the Appendix D. We first observe that uillinois, michigan and msu have the largest value of the term related to graph properties (i.e., MEAN  $(\hat{\phi}_{entropy}, \hat{\phi}_{density}, \hat{\phi}_{avg\_deg}, \hat{\phi}_{deg\_var}, -\hat{\phi}_{\alpha}))$ , while dblp has the smallest. This indicates that the selection of data is influenced by a combination of factors, including graph properties and potentially other factors such as predictive uncertainty. Moreover, it is also interesting to see that the selected graph wiki-vote is the smallest graph among all the pre-training graphs, but it still contributes to the performance. This observation again verifies the curse of big data phenomenon in graph pre-training, which suggests that having more training samples does not necessarily result in improved downstream performance.
Results of APT under different backbone models. We here include GraphCL [39] and JOAO [38] as our backbone models. Table 9 shows our superiority under most cases. The datasets used for pre-training and testing are the same as those employed in the experiments where GCC was the backbone model (refer to Table 7). Since the downstream tasks of GraphCL and JOAO are limited to graph classification, we can directly adapt them to our setting with graph classification as downstream task. For the node classification task, we simply take the RWR subgraphs as the input of GraphCL and JOAO, and treat the learned subgraph representation as node representation. We directly adopt GraphCL and JOAO with their default hyper-parameters. The backbone GNN model is GIN with 5 layers.
Table 9: Micro F1 scores of different models in the node classification task and graph classification task, with GraphCL and JOAO as our backbone model, respectively.
Dataset Method	node classification	graph classification
brazil	dd242	dd68	dd687	wisconsin	cornell	cora	pubmed	imdb-binary	dd	msrc-21
GraphCL (rand, finetune)	30.45(0.37)	40.73(0.66)	64.43(14.95)	15.04(0.85)	14.69(2.48)	10.99(0.58)	63.85(2.18)	44.21(10.58)	63.60(3.61)	58.15(4.60)	8.25(2.94)
GraphCL (fine-tune)	30.81(0.36)	42.91(0.91)	73.57(10.33)	15.35(0.99)	13.51(2.57)	10.66(1.04)	63.85(4.42)	51.05(2.41)	66.90(4.39)	65.55(5.14)	8.77(2.60)
GraphCL-APT (fine-tune)	30.63(0.49)	42.93(1.01)	75.14(10.02)	15.42(1.49)	14.51(2.72)	10.88(0.84)	64.46(3.22)	52.63(2.63)	67.55(3.37)	67.23(4.07)	10.02(2.78)
JOAO (rand, finetune)	29.93(2.84)	42.01(0.68)	72.14(6.74)	10.93(2.85)	8.08(2.15)	7.40(3.48)	45.38(13.30)	45.26(10.31)	67.70(3.35)	62.10(4.31)	11.40(3.06)
JOAO (fine-tune)	29.34(3.04)	42.21(0.88)	75.00(5.76)	10.54(3.07)	7.56(1.94)	8.77(2.37)	50.0(12.28)	42.11(10.26)	68.50(3.61)	62.61(4.99)	10.18(1.72)
JOAO-APT (fine-tune)	30.45(0.56)	42.80(1.05)	72.29(11.03)	10.71(2.85)	9.32(4.32)	10.90(0.67)	51.92(1.98)	46.51(2.69)	63.9(3.48)	63.42(3.61)	12.82(0.70)
Results of APT under molecular pre-training. We also evaluate APT under molecular pretraining setting. We adopt the state-of-art model, Mole-BERT [35] and a commonly used strategy, EdgePred [12] as backbone molecular pre-training models.
The predictive uncertainty mentioned before is based on contrastive loss, however, the loss function of Mole-BERT and EdgePred are different from that used in GCC. To adapt these two models, we need to make corresponding changes to our predictive uncertainty. More specifically, for Mole-BERT, we take triplet masked contrastive learning  $\mathcal{L}_{\mathrm{TMCL}}$ , which is the combination of triplet loss  $\mathcal{L}_{\mathrm{tri}}$  with commonly-used contrastive loss  $\mathcal{L}_{\mathrm{con}}$  in the original paper, as the predictive uncertainty. For EdgePred, we take the loss function, i.e., negative log likelihood in the original paper as the predictive uncertainty
Besides, we adjust the data selection process in molecular pre-training. This is because each molecular graph is taken as a sample in molecular pre-training, so we can directly select graphs. The overall algorithm for APT under molecular setting is given in Algorithm 2.
Algorithm 2 Overall algorithm for APT (Mole).
Input: A collection of graphs  $\mathcal{G} = \{G_1, \dots, G_N\}$ , the number of graphs in the pre-training dataset  $|\mathcal{G}^0|$ , the maximal period  $F$  of training a set of graphs, trade-off parameter  $\gamma_t = 0$ , ratio of the number of selected graphs each time to the number of graphs in the pre-training dataset  $\eta$ , hyperparameter  $\{\beta_t\}$ , the learning rate  $\mu$ , the predictive uncertainty threshold of moving to a new set of graphs  $T_q$ , and the maximum iteration number  $T$ .
1: Choose a set of graphs  $\mathcal{G}^* (|\mathcal{G}^*| = \eta * |\mathcal{G}^0|)$  from  $\mathcal{G}$  via the graph selector, and  $\mathcal{G} \gets \mathcal{G} \setminus \{\mathcal{G}^*\}$ .
2: while The iteration number reaches  $T$  do
3: Update model parameters  $\theta \gets \theta -\mu \nabla_{\theta}\mathcal{L}(\theta)$
4: if  $\phi_{\mathrm{uncertain}}(\mathcal{G}^*) < T_g$  or the model has been trained on  $\mathcal{G}^*$  by  $F$  iterations then
5: Update the trade-off parameter  $\gamma_{t} \sim \mathrm{Beta}(1, \beta_{t})$ .
6: Choose a set of graphs  $\mathcal{G}^*$  ( $|\mathcal{G}^*| = \eta * |\mathcal{G}^0|$ ) from  $\mathcal{G}$ , and  $\mathcal{G} \gets \mathcal{G} \setminus \{\mathcal{G}^*\}$ .
7: end if
8: end while
For pre-training datasets, we use 2 million molecules sampled from the ZINC15 database following [12, 35]. For downstream datasets, we adopt the widely-used 8 binary classification datasets contained in MoleculeNet [33] following [12, 35]. For implementation,  $T_{g}$  is set as -2.5 for Mole-BERT and 0.5 for EdgePred. For both Mole-BERT and EdgePred, the ratio of the number of selected graphs each time to the number of the graphs in original pre-training dataset  $\eta$  is 0.2. Other parameters related to APT are set the same as those in APT with GCC as pre-training backbone model. The other parameters related to molecular pre-training are the same as those employed in Mole-BERT and EdgePred, respectively.
The results of APT under molecular pre-training with Mole-BERT and EdgePred as backbone models are reported in Table 10. We can see that our APT adoptions show significant superiority compared to the backbones. Moreover, APT (Mole-BERT) achieves the best performance on 6 out of 8 downstream datasets among all the baseline models, which shows the effectiveness of our pipeline.
Training time. As empirically noted in Table 11, the total training time of APT-L2 and APT is 18321.39 seconds and 18592.01 seconds respectively (including the time consumed in graph selection
Table 10: Results for molecular property classification. We report the mean (standard deviation) ROC-AUC of 10 random seeds with scaffold splitting. 'No pre-train' means training from scratch. The baseline results are directly obtained from the reported results in Mole-BERT [35].
Dataset Method	Tox21	ToxCast	Sider	ClinTox	MUV	HIV	BBBP	Bace	Average
# Molecules	7,831	8,575	1,427	1,478	93,087	41,127	2,039	1,513	-
No pretrain	74.6(0.4)	61.7(0.5)	58.2(1.7)	58.4(6.4)	70.7(1.8)	75.5(0.8)	65.7(3.3)	72.4(3.8)	67.15
InfoGraph	73.3(0.6)	61.8(0.4)	58.7(0.6)	75.4(4.3)	74.4(1.8)	74.2(0.9)	68.7(0.6)	74.3(2.6)	70.10
GPT-GNN	74.9(0.3)	62.5(0.4)	58.1(0.3)	58.3(5.2)	75.9(2.3)	65.2(2.1)	64.5(1.4)	77.9(3.2)	68.45
ContextPred	73.6(0.3)	62.6(0.6)	59.7(1.8)	74.0(3.4)	72.5(1.5)	75.6(1.0)	70.6(1.5)	78.8(1.2)	70.93
GraphLoG	75.0(0.6)	63.4(0.6)	59.6(1.9)	75.7(2.4)	75.5(1.6)	76.1(0.8)	68.7(1.6)	78.6(1.0)	71.56
G-Contextual	75.0(0.6)	62.8(0.7)	58.7(1.0)	60.6(5.2)	72.1(0.7)	76.3(1.5)	69.9(2.1)	79.3(1.1)	69.34
G-Motif	73.6(0.7)	62.3(0.6)	61.0(1.5)	77.7(2.7)	73.0(1.8)	73.8(1.2)	66.9(3.1)	73.0(3.3)	70.16
AD-GCL	74.9(0.4)	63.4(0.7)	61.5(0.9)	77.2(2.7)	76.3(1.4)	76.7(1.2)	70.7(0.3)	76.6(1.5)	72.16
JOAO	74.8(0.6)	62.8(0.7)	60.4(1.5)	66.6(3.1)	76.6(1.7)	76.9(0.7)	66.4(1.0)	73.2(1.6)	69.71
SimGRACE	74.4(0.3)	62.6(0.7)	60.2(0.9)	75.5(2.0)	75.4(1.3)	75.0(0.6)	71.2(1.1)	74.9(2.0)	71.15
GraphCL	75.1(0.7)	63.0(0.4)	59.8(1.3)	77.5(3.8)	76.4(0.4)	75.1(0.7)	67.8(2.4)	74.6(2.1)	71.16
GraphMAE	75.2(0.9)	63.6(0.3)	60.5(1.2)	76.5(3.0)	76.4(2.0)	76.8(0.6)	71.2(1.0)	78.2(1.5)	72.30
3D InfoMax	74.5(0.7)	63.5(0.8)	56.8(2.1)	62.7(3.3)	76.2(1.4)	76.1(1.3)	69.1(1.2)	78.6(1.9)	69.69
GraphMVP	74.9(0.8)	63.1(0.2)	60.2(1.1)	79.1(2.8)	77.7(0.6)	76.0(0.1)	70.8(0.5)	79.3(1.5)	72.64
MGSSL	75.2(0.6)	63.3(0.5)	61.6(1.0)	77.1(4.5)	77.6(0.4)	75.8(0.4)	68.8(0.6)	78.8(0.9)	72.28
AttrMask	75.1(0.9)	63.3(0.6)	60.5(0.9)	73.5(4.3)	75.8(1.0)	75.3(1.5)	65.2(1.4)	77.8(1.8)	70.81
MAM (with vanilla VQ-VAE)	75.8(0.6)	63.1(0.5)	60.7(1.5)	74.2(2.7)	76.5(1.6)	76.2(0.9)	66.4(0.7)	78.2(0.8)	71.39
TMCL (w/o Lcon)	73.5(1.0)	61.8(0.3)	58.7(1.6)	61.1(4.1)	71.6(1.3)	73.5(1.3)	65.4(2.6)	73.7(2.4)	67.41
TMCL (w/o Ltri)	74.1(0.4)	62.4(0.8)	58.7(3.0)	75.6(2.2)	75.7(1.1)	74.6(1.1)	66.8(1.4)	74.2(1.3)	70.26
MAM	76.2(0.5)	63.9(0.3)	61.4(1.9)	75.1(3.0)	77.4(2.1)	77.5(1.0)	66.8(1.5)	78.9(1.1)	72.16
TMCL	74.9(0.7)	63.2(0.7)	59.6(1.4)	77.0(4.2)	77.2(0.3)	75.3(1.1)	67.6(1.3)	75.1(1.2)	71.24
EdgePred	76.0(0.6)	64.1(0.6)	60.4(0.7)	64.1(3.7)	75.1(1.2)	76.3(1.0)	67.3(2.4)	77.3(3.5)	70.08
APT (EdgePred)	76.5(0.4)	64.3(0.3)	61.4(0.8)	77.9(3.0)	75.2(1.7)	76.6(0.9)	70.7(2.0)	81.8(1.4)	73.05
Mole-BERT	76.8(0.5)	64.3(0.2)	62.8(1.1)	78.9(3.0)	78.6(1.8)	78.2(0.8)	71.9(1.6)	80.8(1.4)	74.04
APT (Mole-BERT)	77.0(0.4)	65.0(0.3)	60.8(0.4)	79.9(1.0)	75.9(1.0)	78.9(0.6)	73.1(0.4)	82.3(1.4)	74.11
and regularization term), while the competitive graph pre-training model GCC takes 40161.68 seconds for the same number of training epochs on the same datasets.
- The time spent on the inference on all graphs during graph selection (which is the main time spent for graph selection) only accounts for  $3.95\%$  and  $3.87\%$  of the total time under APT-L2 and APT respectively. Note that this step is executed only in a few epochs (around  $6\%$  in our current model).
- The time cost of the L2 regularization term only accounts for  $0.08\%$  of the total time and the EWC regularization term only accounts for  $0.45\%$  of the total time, which is calculated by the runtime gap between the models with and without the regularization term. Note that the regularization term is imposed on the first two layers of the GNN encoder, which only accounts for  $12.4\%$  of the total number of parameters.
The efficiency of our model is due to a much smaller number of carefully selected training graphs and samples at each epoch. In addition, the number of parameters in our model is 190,544, which is the same order of magnitude as classical GNNs like GraphSAGE, GraphSAINT, etc. and is relatively small among models in open graph benchmark [11].
Table 11: Training time (sec) comparison between our model and GCC. All the models are trained under the same number of epochs, which is set as 100 in practice. (The difference in time cost of inference on all graphs is due to different runs.)
Time	GCC	APT-L2	APT
inference on all graphs	-	723.92	719.64
proximity term	-	15.98	83.58
total	40161.68	18321.39	18592.01
Effects of hyper-parameters. Here we show that the effect of the hyper-parameter  $\{\lambda\}$  in the proximity term, the maximal number of training iterations on one graph  $F$ , the predictive uncertainty threshold of moving to a new graph  $T_{g}$ , and the predictive uncertainty threshold of choosing training samples  $T_{s}$  on performance in Figure 10 and Figure 11.
The hyper-parameter  $\lambda$  is the trade-off parameters between the knowledge learnt from new data and that from previous data in Eq. (5). We use the dataset dd242 as an example to find the suitable values of the hyper-parameter under the L2 and EWC regularization setting respectively, and present here for reference (see Figure 10). Clearly, a too small or too large  $\lambda$  would deteriorate the performance. Thus, an appropriate value of  $\lambda$  is preferred to ensure that the graph pre-training model can learn from new data as well as remember previous knowledge. We leave changing  $\lambda$  as the future work.
(a) Ablation study on APTL2 (freeze)
(b) Ablation study on APT (freeze)
Figure 10: Performance of our model on dd242 w.r.t varying  $\{\lambda\}$ .
Our model training involves three hyper-parameter  $F, T_{g}, T_{s}$ , where  $F$  controls the largest number of epochs training on each graph,  $T_{g}$  is the predictive uncertainty threshold of moving to a new graph,  $T_{s}$  is the predictive uncertainty threshold of choosing training samples. We use grid search to show  $F \in \{4,5,6\}$ 's,  $T_{g} \in \{3,3.5,4\}$ 's and  $T_{s} \in \{1,2,3\}$ 's role in the pre-training.  $F$  remains at 5 while studying  $T_{g}$  and  $T_{s}$ ,  $T_{g}$  remains at 3.5 while studying  $F$  and  $T_{s}$ , and  $T_{s}$  remains at 2 while studying  $F$  and  $T_{g}$ . Figure 11 presents the effect of these parameters. We find that if the value of  $F$  is set too small or that of  $T_{g}$  is too large, the model cannot learn sufficient knowledge from each graph, leading to suboptimal results. Too large  $F$  or small  $T_{g}$  also lead to poor performance. This indicates that instead of training on a graph for a large period, it would be better to switch to training on various graphs in different domains to gain diverse and comprehensive knowledge. Regarding the hyper-parameter  $T_{s}$ , we observe that large  $T_{s}$  would make the model having too few training samples to learn knowledge, and small  $T_{s}$  could not select the most uncertain and representative samples, thus both cases achieve suboptimal performance.
(a) Ablation study on  $F$
Figure 11: Performance of our model on dd242 w.r.t varying  $F, T_{g}, T_{s}$ .
(b) Ablation study on  $T_{g}$
(c) Ablation study on  $T_{s}$
The choice of  $\beta_{t}$ , its alternatives, and ablation study. At the beginning of the pre-training, the model is less accurate and needs more guidance from graph properties. We therefore set  $\gamma_{t}$  as larger at the beginning and gradually decrease it. To simplify this process, we follow [3] to use the exponential formula of  $\beta_{t} = c_{1} - c_{2}^{t}$  to set the expectation of  $\gamma_{t}$  to be strictly decreasing (where  $\gamma_{t} \sim \mathrm{Beta}(1, \beta_{t})$ ).
The parameters  $c_{1}$  and  $c_{2}$  in the exponential formula of  $\beta_{t} = c_{1} - c_{2}^{t}$  are suggested as 1.005 and 0.995 in [3]. We simply perform grid search on  $c_{1}$  in  $\{1.005, 3, 5\}$ ; see the effect of  $c_{1}$  in the Figure 12.
We then illustrate that the choice of the decay function of  $\beta_{t}$  is robust. Table 12 and Table 13 below show the effect of linear decay, step decay and exponential decay on  $\beta_{t}$ . (The function for linear decay and step decay are designed as  $\beta_{t} = 2.001 + 0.004t$ ,  $\beta_{t} = 2.005 + \mathrm{floor}(t / 20)$ , respectively. The initial value  $\beta_{1}$  is set the same as ours.) While there is no universally better decay function, the performance of our method is not significantly impacted by the choice of different decay functions, and our performance is better than the baselines in most cases regardless of the choices of specific decay functions.
(a) Wisconsin on node classification
(b) Imdb-binary on graph classification
Figure 12: Performance of APT-L2 (freeze) w.r.t varying  $c_{1}$ .
Table 12: Micro F1 of APT-L2 (freeze) with the different decay functions in the node classification task.
Method\Dataset	brazil	dd242	dd68	dd687	wisconsin	cornell	cora	pubmed
linear	72.30(1.37)	16.28(0.57)	12.44(0.72)	10.29(0.87)	54.20(1.50)	47.66(1.53)	35.74(0.52)	46.49(0.19)
step	68.70(3.95)	16.74(0.45)	12.86(1.07)	10.09(0.76)	52.55(2.39)	48.08(1.28)	35.50(0.46)	46.58(0.21)
exponential	69.82(2.32)	16.79(0.88)	12.68(0.81)	10.34(1.12)	55.11(1.74)	48.76(2.20)	34.27(0.43)	46.21(0.15)
Table 13: Micro F1 of APT-L2 (freeze) with the different decay functions in the graph classification task.
Dataset Method	imdb-binary	dd	msrc-21
linear	73.66(0.34)	75.47(0.26)	13.01(0.78)
step	72.99(0.40)	75.41(0.41)	14.13(0.56)
exponential	73.54(0.40)	75.81(0.30)	13.16(0.77)
Results under GCC's original experimental setting. For a more comprehensive comparison, we also adopt the same pretraining datasets and downstream datasets as GCC. Table 14 and table 15 shows the performance of our model and the strongest competitor GCC, under the experiment setting of GCC in node classification and graph classification task, respectively. (The performance of GCC is directly taken from its paper.) The results indicate that our model still outperforms GCC under the experimental setting of GCC. The results still suggest ours superiority in most cases.
Table 14: Micro F1 scores of GCC and our model in the node classification task, under the experimental setting of GCC.
Dataset Method	US-Airport	H-index
GCC (freeze)	65.6	75.2
Ours (freeze)	66.12(5.31)	75.9(3.01)
GCC (fine-tune)	67.2	80.6
Ours (fine-tune)	70.50(6.08)	82.28(1.48)
Table 15: Micro F1 scores of GCC and our model in the graph classification task, under the experimental setting of GCC.
Dataset Method	IMDB-B	IMDB-M	COLLAB	RDT-B	RDT-M
GCC (freeze)	72.0	49.4	78.9	89.8	53.7
Ours (freeze)	73.2(0.27)	49.6(1.17)	79.12(0.77)	89.6(2.51)	54.1(2.45)
GCC (fine-tune)	73.8	50.3	81.1	87.6	53.0
Ours (fine-tune)	76.27(1.20)	50.50(1.08)	81.23(0.86)	92.20(2.43)	53.28(2.33)
The justification of input graphs' learning order. Table 16 reveals the downstream performance can be affected by the learning order of input training graphs. With the guidance of graph selector, the pre-training model is encouraged to first learn the graphs and samples with higher predictive uncertainty and graph properties. Such learning order accomplishes better downstream performance compared to the reverse or random one.
Table 16: The effect of input graphs' learning order on downstream performance (micro F1 is reported) under freezing mode in the node classification task. The first row is the order learnt from APT-L2, and the second and third rows are the reverse and random order of the first row, respectively.
Method\Dataset	brazil	dd242	dd68	dd687	wisconsin	cornell	cora	pubmed
Our order	69.82(2.32)	16.79(0.88)	12.68(0.81)	10.34(1.12)	55.11(1.74)	48.76(2.20)	34.27(0.43)	46.21(0.15)
Reverse order	69.60(2.71)	16.00(0.47)	11.41(0.91)	10.65(0.65)	51.46(1.64)	44.36(1.38)	35.66(0.62)	45.92(0.14)
Random order	67.25(2.40)	16.11(0.79)	12.57(1.17)	11.06(0.75)	53.06(2.41)	46.76(1.95)	35.90(0.72)	46.36(0.20)
The choice of the "difficult" data. Among all the data, "difficult" samples contribute the most to the loss function, and thus they can yield gradients with large magnitude. Comparatively, training with easy samples may suffer from inefficiency and poor performance as these data points produce gradients with magnitudes close to zero [14, 30]. In addition, learning from difficult samples has proven to be able to accelerate convergence and enhance the expressive power of the learnt representations [28, 31]. For our model, the importance of learning from difficult samples is also justified empirically, as shown in Table 17.