SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The empirical result indicates that the knowledge or information contained in previous input data will be forgotten or covered by newly incoming data. This phenomenon, called catastrophic forgetting, was first noticed in continual learning [35] and also
Figure 4: Predictive uncertainty versus training epoch.
appears in our case. Intuitively, when the training data is taken in a progressive or iterative manner, the learned parameters will cater to the newly incoming data and forget the old, previous data points.
One remedy for this issue is adding a proximal term to the objective. The additional proximal term (i.e., the regularization) guarantees the proximity between the new parameters and the model parameters learned from previous graphs. Therefore, when the model is learning the  $k$ -th input graph, the loss function for our pre-training model in APT is
$$
\mathcal {L} (\theta) = \sum_ {i} \mathcal {L} _ {i} (\theta) + \frac {\lambda}{2} \sum_ {j} F _ {j j} ^ {(k - 1)} \left(\theta_ {j} - \theta_ {j} ^ {(k - 1)}\right) ^ {2}, \tag {5}
$$
where  $\mathcal{L}_i$  is given in Eq. (1), the first summation is taken over the subgraph instances sampled from the latest input graph,  $\theta^{(k - 1)}$  is the model parameters after learning from the first  $k - 1$  graphs, and the parameter  $\lambda$  describes the trade-off between the knowledge learnt from new data and that from previous data. Here,  $F^{(k - 1)}$  is the Fisher information matrix of  $\theta^{(k - 1)}$ , and  $F_{jj}^{(k)}$  is its  $j$ -th diagonal element. When  $F$  is set as an identity matrix, the second term degenerates to the L2 regularization (which serves as one of our variants). The proximal term in Eq. (5) is absent when the first input graph is introduced to the model, and the term is applied on the first three layers of the pre-training model. Finally, we note that the total number of parameters in the pre-training model is in the same order of magnitude as classical GNNs, so the memory cost would not be a bottleneck.
3.3 Training and fine-tuning
Integrating the graph selector and the pre-training model forms the entire APT framework, and the overall algorithm is presented in Appendix C. After the training phase, the APT framework returns a pre-trained GNN model, and then the pre-trained model is applied to various downstream tasks from a wide spectrum of domains. In the so-called freezing mode, the pre-trained model returned by APT is directly applied to downstream tasks, without any changes in parameters. Alternatively, the fine-tuning mode uses the pre-trained graph encoder as initialization, and offers the flexibility of training the graph encoder and the downstream classifier together in an end-to-end manner.
In the experiments, we pre-train a graph representation model via the proposed APT framework, and then evaluate the transferability of our pre-trained model on multiple unseen graphs in the node classification and graph classification task. Lastly, we include the applicable scope of our pre-trained model. Additional experiments can be found in Appendix H, including our adaptation to backbone pre-training models GraphCL, JOAO and Mole-BERT, training time, impact of different graph properties, analysis of the ablation studies and selected pre-training graphs, hyper-parameter sensitivity, explorations of various combinations of graph properties.
Datasets. The datasets for pre-training and testing, along with their statistics, are listed in Appendix D. Pre-training datasets are collected from different domains, including social, citation, and movie networks. Testing datasets comprise both in-domain (e.g., citation, movie) and cross-domain (e.g., image, web, protein, transportation and others) datasets to evaluate transferability comprehensively, also including large-scale datasets with millions of edges sourced from [28].
Baselines. We evaluate our model against the following baselines for node and graph classification tasks. For node classification, ProNE [80], DeepWalk [49], struc2vec [53], DGI [65], GAE [34], and GraphSAGE [23] are used as baselines, and then the learned representations are fed into the logistic regression, as most of baselines did. As for graph classification, we take graph2vec [46], InfoGraph [59], DGCNN [81] and GIN [71] as baselines, using SVM as the classifier, which aligns with the methodology of most baselines. For both tasks, we also compare our model with (1) Random: random vectors are generated as representations; (2) GraphCL [76]: a GNN pre-training scheme based on contrastive learning with augmentations; (3) JOAO [75]: a GNN pre-training scheme that automatically selects data augmentations; (4) GCC [50]: the state-of-the-art cross-domain graph pre-training model (which is our model's version without the data selector, trained on all pre-training
data). GCC, GraphCL and JOAO are trained on the entire collected input data, and the suffix (rand, fine-tune) indicates whether the model is trained from scratch. We also include 9 variants of our model: (1) APT-G: removes the criteria of graph properties in the graph selector; (7) APT-P: removes the criteria of predictive uncertainty in the graph selector; (8) APT-R: removes the regularization w.r.t old knowledge in Eq. (5); (8) APT-L2: degenerates the second term in Eq. (5) with L2 regularization.
Experimental settings. In the training phase, we aim to utilize data from different domains to pre-train one graph model. We iteratively select graphs for pre-training until the predictive uncertainty of any candidate graph is below 3.5. For each selected graph, we choose samples with predictive uncertainty higher than 3. We set the number of subgraph instances queried in the graph for uncertainty estimation  $M$  as 500. The time-adaptive parameter  $\gamma_{t}$  in Eq. (4) follows a  $\gamma_{t} \sim \mathrm{Beta}(1, \beta_{t})$ , where  $\beta_{t} = 3 - 0.995^{t}$ . We set the trade-off parameter  $\lambda = 10$  for APT-L2, and  $\lambda = 500$  for APT. The total iteration number is 100. We adopt GCC as the backbone pre-training model with its default hyper-parameters, including their subgraph instance definition. In the fine-tuning phase, we select logistic regression or SVM as the downstream classifier and adopt the same setting as GCC. Due to space limit, results of our adaption on other backbones like GraphCL, JOAO and Mole-BERT, as well as details can be found in Appendices H and G. See Appendix G for details.
We provide an open-source implementation of our model APT at https://github.com/galina0217/APT.
Node classification. Table 1 presents the micro F1 score of different methods over 10 unseen graphs from a wide spectrum of domain for node classification task. We observe that our model beats the graph pre-training competitor by an average of  $9.94\%$  and  $17.83\%$  under freezing and fine-tuning mode respectively. This suggests that instead of pre-training on all the collected graphs (like GCC), it is better to choose a part of graphs better suited for pre-training (like our model APT).
Moreover, compared with the traditional models without pre-training, the performance gain of our model is attributed to the transferable knowledge learned by pre-training strategies. We also find that some proximity-based models, like ProNE, enforce neighboring nodes to share similar representations, leading to superior performance on graphs with strong homophily rather than weak homophily. Nonetheless, when applied to different graphs, these models require re-training, which makes them non-transferable. In contrast, we target at a general, transferable model free from specific assumptions on graph types, and thus applicable across wide scenarios (including both homophilic and heterophilic graphs). Ablation study reveals the necessity of all the components in APT (e.g.,
the proximity regularization with respect to old knowledge, proximity term, graph properties and predictive uncertainty). We also explore the impacts of the five graph properties used in our model, and demonstrate their indispensability by experiments. Thus, combining all graph properties is essential to boost the performance of APT. Details in ablation study and more experimental results can be found in Appendix H.
Graph classification. The micro F1 score on unseen test data in the graph classification task is summarized in Table 2. Especially, our model is  $7.2\%$  and  $1.3\%$  on average better than the graph pre-training backbone model under freezing and fine-tuning mode, respectively. Interestingly, we observe that all variants of APT perform quite well, and thus, a simpler yet well-performed version of APT could be used in practice. This phenomenon could happen since "graph pre-train and fine-tune" is an extremely complicated non-convex optimization problem. Another observation is that in specific cases, such as dd dataset, a decrease in downstream performance after fine-tuning is observed, as compared to the freezing mode. This could happen since "graph pre-train and fine-tune" is an extremely complicated non-convex optimization problem. A similar observation has been made in previous work [37] as well.
Finally, it is worth noting that our APT model achieves a training time  $2.2 \times$  faster than the competitive model GCC, achieved through a reduced number of carefully selected training graphs and samples. More specifically, we carefully selected only 7 datasets out of the available 11 and performed pre-training using at most  $24.92\%$  of the samples in each selected dataset. Moreover, for each newly added dataset, our model only needs a few more training iterations to convergence, rather than being trained from scratch.
4.3 Discussion: scope of application
The transferability of the pre-trained model comes from the learned representative structural patterns and the ability to distinguish these patterns (as discussed in  $\S 2$ ). Therefore, our pretraining model is more suitable for the datasets where the target (e.g., labels) is correlated with
Table 2: Micro F1 of different models in graph classification.
Dataset Method	imdb-binary	dd	msrc-21	A.R.
Random	49.30(4.82)	52.72(4.34)	4.49(2.14)	11
graph2vec	56.20(5.33)	59.16(3.47)	8.22(3.67)	7.7
InfoGraph	66.58(0.63)	58.66(0.23)	6.01(0.59)	7.7
GraphCL (freeze)	55.10(3.18)	57.82(4.71)	5.44(2.77)	9.3
JOAO (freeze)	63.90(3.48)	55.97(3.61)	5.09(2.65)	9.3
GCC (freeze)	73.09(0.55)	75.16(0.53)	11.61(1.33)	5.3
APT-G (freeze)	73.10(0.39)	75.24(0.42)	12.81(0.74)	4.3
APT-P (freeze)	72.83(0.81)	76.38(0.32)	13.30(0.57)	3.0
APT-R (freeze)	73.98(0.21)	75.32(0.34)	12.90(0.57)	3.0
APT-L2 (freeze)	73.54(0.40)	75.81(0.30)	13.16(0.77)	2.3
APT (freeze)	73.00(0.50)	75.83(0.31)	13.81(1.06)	3.0
DGCNN	71.00(4.69)	58.63(4.46)	6.01(0.59)	11.3
GIN	72.00(2.41)	77.61(1.47)*	10.54(5.08)	6.0
GraphCL (rand, fine-tune)	63.60(3.61)*	58.15(4.60)*	8.25(2.94)	12.7
JOAO (rand, fine-tune)	67.70(3.35)	62.10(4.31)	11.40(3.06)	10.0
GCC (rand, fine-tune)	75.80(1.37)	74.26(0.59)	17.18(1.43)	7.3
GraphCL (fine-tune)	66.90(4.39)	65.55(5.14)	8.77(2.60)	10.7
JOAO (fine-tune)	68.50(3.61)	62.61(4.99)	10.18(1.72)	10.0
GCC (fine-tune)	76.19(0.90)	75.32(1.77)	24.90(1.65)	4.7
APT-G (fine-tune)	76.29(0.89)	75.46(0.77)	21.94(0.73)	4.7
APT-P (fine-tune)	76.70(1.01)*	75.34(0.88)	24.32(1.22)	3.7
APT-R (fine-tune)	76.60(1.02)	75.64(0.70)	24.09(2.12)	3.3
APT-L2 (fine-tune)	75.93(0.84)	75.58(1.06)	25.58(1.57)*	3.7
APT (fine-tune)	76.27(1.20)	75.69(1.42)	24.41(1.82)	3.0
subgraph patterns or structural properties (e.g., motifs, triangles, betweenness, stars). For example, for node classification on heterophilous graphs (e.g., winconsin, cornell), our model performs very well because in these graphs, nodes with the same label are not directly connected, but share similar structural properties and behavior (or role, position). On the contrary, graphs with strong homophily (like cora, pubmed, ogbarxiv and ogbproteins) may not benefit too much from our models. Similar observation can also be made on graph classification: our model could also benefit the graphs whose label has a strong relationship with their structure, like molecular, chemical, and protein networks (e.g., dd in our experiments) [17, 66].
Graph pre-training. Inspired by pre-training in CV/NLP, recent efforts have shed light on pre-training on GNNs. Initially, unsupervised graph representation learning is used for graph pretraining [13, 21, 23, 46, 53, 63, 80]. The design of unsupervised models is largely based on the neighborhood similarity assumption, and thus cannot generalize to unseen graphs. More recently, self-supervised graph learning emerges as another line of research, including graph generative and contrastive models. Graph generative models aim to capture the universal graph patterns by recovering certain parts of input graphs [7, 27, 30, 34, 67], but they rely heavily on domain-specific knowledge. In comparison, contrastive models maximize the agreement between positive pairs and minimize that between negative pairs [24, 25, 29, 39, 40, 44, 59, 60, 65, 69, 70, 76, 78, 83, 85-87]. Some work in
this direction takes subgraph sampling as augmentation, in the hope that the transferable subgraph patterns can be captured during pre-training. However, all the aforementioned studies only focus on the design of pre-training models, rather than suitable selection of data for pre-training.
Uncertainty-based sample selection. The terminology uncertainty is widely used in machine learning, without a universally-accepted definition. In general, this term refers to the lack of confidence of an ML model in certain model parameters. The majority of existing works define uncertainty in the label space, such as taking the uncertainty as the confidence level about the prediction [16, 18, 42, 61]. Only a few works define uncertainty in the representation space [52, 68]. In [52], uncertainty is measured based on the representations of an instance's nearest neighbors with the same label. However, this approach requires access to the label information of the neighbors, and thus cannot be adapted in pre-training with unlabeled data. [68] introduces a pretext task for training a model of uncertainty over the learned representations, but this method assumes a well-pre-trained model is already available. Such a post processing manner is not applicable to our scenario, because we need an uncertainty that can guide the selection of data during pre-training rather than after pre-training.
Some works on active learning and hard example mining (HSM) have also introduced concept similar to uncertainty. In active learning, uncertainty is measured via classification prediction, and the active learning model focuses on those samples which the model is least certain about [4, 74, 82, 84]. However, these techniques all rely on the labels and cannot be adapted in pre-training with unlabeled data. As another line of work, HSM introduces similar strategies and works on those samples with the greatest loss, which can also be regarded as a kind of uncertainty [36, 43, 55, 56]. Nevertheless, existing HSM approaches do not meet the following two requirements needed in our setting. (1) The chosen instances should follow a joint distribution that reflects the topological structures of real-world graphs. This is satisfied by our use of graph-level predictive uncertainty and graph properties, but is not met in HSM. (2) The chosen set of graphs should include informative and sufficiently diverse instances. This is only achieved by the proposed APT framework while existing HSM methods fail to consider this requirement.
Pre-training in CV and NLP. For pre-training in CV and NLP, scaling up the pre-training data size often results in a better or saturating performance in the downstream [1, 15, 33, 51, 62]. In view of this, data selection is not an active research direction for CV and NLP. Existing studies mainly focus on selecting pre-training data that closely matches the downstream domain [2, 5, 8-10, 38, 72]. The assumption on downstream domain knowledge differs from our graph pre-training setting, making such data selection less relevant to our work.
Data-centric AI. This recently introduced concept emphasizes the enhancement of data quality and quantity, rather than model design [73, 79]. Following-up works in graph pre-training [24, 64] exploits the data-centric idea to design data augmentation. For example, [24] introduces a graph data augmentation method by interpolating the generator of different classes of graphs. [64] mainly focuses on the theoretical analysis of data-centric properties of data augmentation. While many of these works advocate for shifting the focus to data, they do not consider the co-evolution of data and model, as is the case in our work.
In this paper, we identify the curse of big data phenomenon for pre-training graph neural networks (GNNs). This observation then motivates us to choose a few suitable graphs and samples for GNN pre-training rather than training on a massive amount of unselected data. Without any knowledge of the downstream tasks, we propose a novel graph selector to provide the most instructive data for pre-training. The pre-training model is then encouraged to learn from the data in a progressive and iterative way, reinforce itself on newly selected data, and provide instructive feedback to the graph selector for further data selection. The integration of the graph selector and the pre-training model into a unified framework forms a data-active graph pre-training (APT) paradigm, in which the two components are able to mutually boost the capability of each other. Extensive experimental results verify that the proposed APT framework indeed enhances model capability with fewer input data.
This work was partially supported by NSFC (62206056, 62322606). Xin Jiang and Carl Yang were not supported by any fund from China.
[87] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, pages 2069-2080, 2021.
A Theoretical Connection Between Network Entropy and Typical Graph Properties
Many interesting graph structural properties from basic graph theory give rise to a graph with high network entropy [19]. We here theoretically show some connections between the proposed network entropy and typical structural properties.
To make theoretical analysis, we consider connected, unweighted and undirected graph, whose network entropy depends solely on its degree distribution (see Eq. (3)). Considering a random graph  $G$  with a fixed node set, we suppose that the degree of any node  $v_{i}$  independently follows distribution  $p$ , which is a common setting in random graph theory [8]. Then the expected network entropy of  $G$  is
$$
\langle \mathcal {H} (G) \rangle = \frac {1}{2 | E |} \sum_ {i} \left\langle \boldsymbol {d} _ {i} \log \boldsymbol {d} _ {i} \right\rangle = \frac {\left\langle \boldsymbol {d} \log \boldsymbol {d} \right\rangle}{\langle \boldsymbol {d} \rangle}. \tag {6}
$$
where every  $\pmb{d}_i$  (and  $\pmb{d}$ ) is an independent random variable follows the distribution  $p$ .
Now we are ready to discuss the connection between network entropy  $\langle \mathcal{H}(G)\rangle$  and some typical graph properties (i.e., average degree  $\langle d\rangle$ , degree variance  $\operatorname{Var}(d)$  and the scale-free exponent  $\alpha$ ).
Average degree. Given that the function  $x \log x$  is convex in  $x$ , we have
$$
\langle \mathcal {H} (G) \rangle \geq \frac {\langle d \rangle \log \langle d \rangle}{\langle d \rangle} = \log \langle d \rangle . \tag {7}
$$
It is clear that average degree is the lower bound of network entropy. Based on our discussion on §3.1, we conclude that when used for pre-training, an input graph with higher average degree would in general result in better performance of the pre-trained model.
Degree variance. The Taylor expansion of  $\langle d\log d\rangle$  in Eq. (6) at  $\langle d\rangle$  gives
$$
\langle \mathcal {H} (G) \rangle = \log \langle \boldsymbol {d} \rangle + \frac {\operatorname {V a r} (\boldsymbol {d})}{2 \langle \boldsymbol {d} \rangle^ {2}} + o \left(\frac {1}{\langle \boldsymbol {d} \rangle^ {2}}\right).
$$
where  $\operatorname{Var}(d)$  is the variance of  $d$ . We find that  $\log \langle d\rangle$  is exactly the zeroth-order term in the expansion. When average degree is fixed, the network entropy and the degree variance  $\operatorname{Var}(d)$  are positively correlated. This in turn implies a positive correlation between degree variance and the test performance of the model.
Scale-free exponent. Most real-world networks exhibit an interesting scale-free property (i.e., only a few nodes have high degrees), and thus the degree distribution often follows a power-law distribution. That is, we can just write the degree distribution as  $p(x) \sim x^{-\alpha}$ , where  $\alpha$  is called the scale-free exponent. For a real-world network, the scale-free exponent  $\alpha$  is usually larger than 2 [4]. Suppose the degrees of a random graph  $G$  with  $N$  nodes follows a power-law distribution  $p(x) = Cx^{-\alpha}$  where  $C$  is a normalization constant. When  $\alpha > 2$ , we could approximately have [8]
$$
\langle \mathcal {H} (G) \rangle = \frac {1}{\alpha - 2}, \quad \text {i f} N \to \infty .
$$
Clearly, a smaller scale-free exponent  $\alpha$  results in a higher network entropy.
Remark 1 (Connection between network entropy and typical structural properties). A graph with high network entropy arises from graphs with typical structural characteristics like large average degree, large degree variance, and scale-free networks with low scale-free exponent.
Besides the above theoretical analysis. The motivation of choosing density, average degree, degree variance and scale-free exponent is similar to that of network entropy. Intuitively, graphs with larger average degree and higher density have more interactions among the nodes, thus providing more topological information to graph pre-training. Also, the larger the diversity of node degrees, the more diverse the subgraph samples. The diversity of node degrees can be measured by degree variance and scale-free exponent. (A smaller scale-free exponent indicates the length of the tail of degree distribution is relatively longer, i.e., the degree distribution spreads out wider.)
B Theoretical Analysis and Advantages of Predictive Uncertainty
In this section, we provide the proofs of Theorem 1 and demonstrate the advantages of our proposed uncertainty.
Proofs of Theorem 1 Before we start the proof for Theorem 1, we introduce three useful lemmas that are adopted to prove the theorem as follows.
Lemma 1 (Fano's Inequality.). Let  $X$  be a random variable uniformly distributed over a finite set of outcomes  $\mathcal{X}$ . For any estimator  $\hat{X}$  such that  $X \to Y \to \hat{X}$  forms a Markov chain, we have
$$
\Pr (\widehat {X} \neq X) \geq 1 - \frac {\operatorname {I} (X ; \widehat {X}) - \log 2}{\log | \mathcal {X} |}. \tag {8}
$$
Lemma 2 (Data-Processing Inequality.). For any Markov chain  $X \to Y \to Z$ , we have
$$
\mathrm {I} (X; Y) \geq I (X; Z) a n d \mathrm {I} (Y; Z) \geq \mathrm {I} (X; Z). \tag {9}
$$
The proof of lemma 1 and lemma 2 can be found under Chapter 2 in [5]. The above lemmas establish a connection between mutual information and the accuracy of downstream task. Next, we further establish the connection between mutual information and ours uncertainty measure.
Lemma 3 (Connection between InfoNCE and mutual information). Assume that graph encoder  $g$  is a scaling function, namely  $g(\pmb{x}) = k\pmb{x}$ . Hence, the mutual information  $\mathrm{I}(X;Z)$  can be expressed by the InfoNCE loss  $\phi_{\text{uncertain}}$  for node in Eq.(1) as follows:
$$
- \phi_ {\text {u n c e r t a i n}} \geq - \mathrm {I} (X; f (X)) = - \mathrm {I} (X; Z). \tag {10}
$$
Proof: We follow [20] to prove the theorem. The mutual information  $\mathrm{I}(X;Z)$  can be expressed as: