SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiarong Xu $^{1*}$ , Renhong Huang $^{2}$ , Xin Jiang $^{3}$ , Yuxuan Cao $^{2}$ , Carl Yang $^{4}$ , Chunping Wang $^{5}$ , Yang Yang $^{2}$
$^{1}$ Fudan University,  $^{2}$ Zhejiang University,  $^{3}$ Lehigh University  $^{4}$ Emory University,  $^{5}$ Finvolution Group
jiarongxu@fudan.edu.cn, {renh2, caoyx, yangya}@zju.edu.cn xjiang@lehigh.edu, j.carlyang@emory.edu, wangchunping02@xinye.com
Pre-training on graph neural networks (GNNs) aims to learn transferable knowledge for downstream tasks with unlabeled data, and it has recently become an active research area. The success of graph pre-training models is often attributed to the massive amount of input data. In this paper, however, we identify the curse of big data phenomenon in graph pre-training: more training data do not necessarily lead to better downstream performance. Motivated by this observation, we propose a better-with-less framework for graph pre-training: fewer, but carefully chosen data are fed into a GNN model to enhance pre-training. The proposed pre-training pipeline is called the data-active graph pre-training (APT) framework, and is composed of a graph selector and a pre-training model. The graph selector chooses the most representative and instructive data points based on the inherent properties of graphs as well as predictive uncertainty. The proposed predictive uncertainty, as feedback from the pre-training model, measures the confidence level of the model in the data. When fed with the chosen data, on the other hand, the pretraining model grasps an initial understanding of the new, unseen data, and at the same time attempts to remember the knowledge learned from previous data. Therefore, the integration and interaction between these two components form a unified framework (APT), in which graph pre-training is performed in a progressive and iterative way. Experiment results show that the proposed APT is able to obtain an efficient pre-training model with fewer training data and better downstream performance.
Pre-training Graph neural networks (GNNs) shows great potential to be an attractive and competitive strategy for learning from graph data without costly labels [29, 50]. Recent advancements have been made in developing various graph pre-training strategies, which aim to capture transferable patterns from a diverse set of unlabeled graph data [22, 30, 31, 40, 44, 60, 76, 77]. Very often, the success of a graph pre-training model is attributed to the massive amount of unlabeled training data, a well-established consensus for pre-training in computer vision [12, 19, 26] and natural language processing [11, 45].
In view of this, contemporary research almost has no controversy on the following issue: Is a massive amount of input data really necessary, or even beneficial, for pre-training GNNs? Yet, two simple experiments regarding the number of training samples (and graph datasets) seem to doubt the positive
Figure 1: Top row: The effect of scaling up sample size (log scale) on the downstream performance based on a group of GCCs [50] under different configurations (the graphs used for pre-training are kept as all eleven pre-training data in Table 3, and the samples are taken from the backbone pre-training model according to its sampling strategy). The results for different downstream graphs (and tasks) are presented in separate figures. To better show the changing trend, we fit a curve to the best performing models (i.e., the convex hull fit as [1] does). Bottom row: The effect of scaling up the number of graph datasets on the downstream performance based on GCC. For a fixed horizontal coordinate, we run 5 trials. For each trial, we randomly choose a combination of input graphs. The shaded area indicates the standard deviation over the 5 trials. See Appendix E for more observations on other graph pre-training models and detailed settings.
answer to this question. We observe that scaling pre-training samples does not result in a one-model-fits-all increase in downstream performance (top row in Figure 1), and that adding input graphs (while fixing sample size) does not improve and sometimes even deteriorates the generalization of the pre-trained model (bottom row in Figure 1). Moreover, even if the number of input graphs (the horizontal coordinate) is fixed, the performance of the model pre-trained on different combinations of inputs varies dramatically; see the standard deviation in blue in Figure 1. As the first contribution of this work, we identify the curse of big data phenomenon in graph pre-training: more training samples (and graph datasets) do not necessarily lead to better downstream performance.
Therefore, instead of training on a massive amount of data, it is more appealing to choose a few suitable samples and graphs for pre-training. However, without the knowledge of downstream tasks, it is difficult to design new data selection criteria for the pre-training model. To this end, we propose the graph selector which is able to provide the most instructive data for the model by incorporating two criteria: predictive uncertainty and graph properties. On one hand, predictive uncertainty is introduced to measure the level of confidence (or certainty) in the data. On the other hand, some graphs are inherently more informative and representative than others, and thus the fundamental properties should help in the selection process.
Apart from the graph selector, the pre-training model is also designed to co-evolve with the data. Instead of swallowing data as a whole, the pre-training model is encouraged to learn from the data in a progressive way. After learning a certain amount of training data, the model receives feedback (from predictive uncertainty) on what kind of data the model has least knowledge of. Then the pre-training model is able to reinforce itself on highly uncertain data in next training iterations.
Putting together, we integrate the graph selector and the pre-training model into a unified paradigm and propose a data-active graph pre-training (APT) framework. The term "data-active" is used to emphasize the co-evolution of data and model, rather than mere data selection before model training. The two components in the framework actively cooperate with each other. The graph selector recognizes the most instructive data for the model; equipped with this intelligent selector, the pre-training model is well-trained and in turn provides better guidance for the graph selector.
The rest of the paper is organized as follows. In §2 we present the basic graph pre-training framework and review existing work on this topic. Then in §3 we describe in detail the proposed data-active graph pre-training (APT) paradigm. §4 contains numerical experiments, demonstrating the superiority of APT in different downstream tasks, and also includes the applicable scope of our pre-trained model.
2 Basic Graph Pre-training Framework
This section reviews the basic framework of graph pre-training commonly used in the literature. The backbone of our graph pre-training model also follows this framework.
We start with a natural question: What does graph pre-training actually learn? On one hand, graph pre-training tries to learn transferable semantic meaning associated with structural patterns. For example, both in citation networks and social networks, the closed triangle structure  $(\square)$  is interpreted as a stable relationship, while the open triangle  $(\square)$  indicates an unstable relationship. In comparison, these semantic meanings can be quite different in other networks, e.g., molecular networks. On the other hand, however, the distinction (or relationship) between different structural patterns is still transferable. Taking the same example, the closed and open triangles might yield different interpretations in molecular networks (stability of certain chemical property) from those in social networks (stability in social intimacy), but the distinction between these two structures remains the same: they indicate opposite (or contrastive) semantic meanings [32, 41, 57]. Therefore, the graph pre-training either learns representative structural patterns (when semantic meanings are present), or more importantly, obtains the capability of distinguishing these patterns. This observation in graph pre-training is not only different from that in other areas (e.g., computer vision and natural language processing), but may also explain why graph pre-training is effective, especially when some downstream information is absent.
With the hope to learn the transferable structural patterns or the ability to distinguish them, the graph pre-training model is fed with a diverse collection of input graphs, and the learned model, denoted by  $f_{\theta}$  (or simply  $f$  if the parameter  $\theta$  is clear from context), maps a node to a low-dimensional representation. Unaware of the specific downstream task as well as task-specific labels, one typically designs a self-supervised task for the pre-training model. Such self-supervised information for a node is often hidden in its neighborhood pattern, and thus the structure of its ego network can be used as the transferable pattern. Naturally, subgraph instances sampled from the same ego network  $\Gamma_{i}$  are considered similar while those sampled from different ego networks are rendered dissimilar. Therefore, the pre-training model attempts to capture the similarities (and dissimilarities) between subgraph instances, and such a self-supervised task is called the subgraph instance discrimination task. More specifically, given a subgraph instance  $\zeta_{i}$  from an ego network  $\Gamma_{i}$  centered at node  $v_{i}$  as well as its representation  $x_{i} = f(\zeta_{i})$ , the model  $f$  aims to encourage higher similarity between  $x_{i}$  and the representation of another subgraph instance  $\zeta_{i}^{+}$ sampled from the same ego network. This can be done by minimizing, e.g., the InfoNCE loss [48],
$$
\mathcal {L} _ {i} = - \log \frac {\exp \left(\boldsymbol {x} _ {i} ^ {\top} f \left(\zeta_ {i} ^ {+}\right) / \tau\right)}{\exp \left(\boldsymbol {x} _ {i} ^ {\top} f \left(\zeta_ {i} ^ {+}\right) / \tau\right) + \sum_ {\zeta_ {i} ^ {\prime} \in \Omega_ {i} ^ {-}} \exp \left(\boldsymbol {x} _ {i} ^ {\top} f \left(\zeta_ {i} ^ {\prime}\right) / \tau\right)}, \tag {1}
$$
where  $\Omega_{i}^{-}$  is a collection of subgraph instances sampled from different ego networks  $\Gamma_{j}$  ( $j \neq i$ ), and  $\tau > 0$  is a temperature hyper-parameter. Here the inner product is used as a similarity measure between two instances. One common strategy to sample these subgraph instances is via random walks on graphs, as used in GCC [50], and other sampling methods as well as loss functions are also valid.
3 Data-Active Graph Pre-training
In this section, we present the proposed APT framework for graph pre-training, and the overall pipeline is illustrated in Figure 2. The APT framework consists of two major components: a graph selector and a graph pre-training model. The technical core is the interaction between these two components: The graph selector feeds suitable data for pre-training, and the graph pre-training model learns from the carefully chosen data. The feedback of the pre-training model in turn helps select the needed data tailored to the model.
The rest of this section is organized as follows. We describe the graph selector in §3.1 and the graph pre-training model in §3.2. The overall pre-training and fine-tuning strategy is presented in §3.3.
Figure 2: Overview of the proposed data-active graph pre-training paradigm. The graph selector provides the graph and samples suitable for pre-training, while the graph pre-training model learns from the incoming data in a progressive way and in turn better guides the selection process. In the graph selector component, Part (a) provides an illustrating example on the predictive uncertainty, and Part (b) plots the Pearson correlation between the properties of the input graph and the performance of the pre-trained model on the training set (using this graph) when applied to different unseen test datasets (see Appendix F for other properties that exhibit little/correlation with performance).
3.1 Graph selector
In view of the curse of big data phenomenon, it is more appealing to carefully choose data well suited for graph pre-training rather than training on a massive amount of data. Conventionally, the criterion of suitable data, or the contribution of a data point to the model, is defined based on the output predictions on downstream tasks [20]. In graph pre-training where downstream information is absent, new selection criteria or guidelines are needed to provide effective instructions for the model. Here we introduce two kinds of selection criteria, originating from different points of view, to help select suitable data for pre-training. The predictive uncertainty measures the model's understanding of certain data, and thus helps select the least certain data points for the current model. In addition to the measure of model's ability, some inherent properties of graphs can also be used to assess the level of representativeness or informativeness of a given graph.
Predictive uncertainty. The notion of predictive uncertainty can be explained via an illustrative example, as shown in part (a) of the graph selector component in Figure 2. Consider a query subgraph instance  $\zeta_{i}$  (denote by  $(\widehat{\mathbb{Q}})$  in Figure 2) from the ego network  $\Gamma_{i}$  in a graph  $G$ . If the pre-training model cannot tell its similar instance  $\zeta_{i}^{+}$  (denoted by  $(\widehat{\mathbb{Q}})$ ) from its dissimilar instance  $\zeta_{i}^{-} \in \Omega_{i}^{-}$  (denoted by  $(\widehat{\mathbb{Q}})$ ), we say that the current model is uncertain about the query instance  $\zeta_{i}$ . Therefore, the contrastive loss function in Eq. (1) comes in handy as a natural measure for the predictive uncertainty of the instance  $\zeta_{i}$ :  $\phi_{uncertain}(\zeta_{i}) = \mathcal{L}_{i}$ . Accordingly, the predictive uncertainty of a graph  $G$  (i.e., the graph-level predictive uncertainty) is defined as  $\phi_{uncertain}(G) = (1 / M)\sum_{i=1}^{M}\mathcal{L}_{i}$ , where  $M$  is the number of subgraph instances queried in this graph.
We further establish a provable connection between the proposed predictive uncertainty and the conventional definition of uncertainty. In most existing work, model uncertainty is often defined in the label space, e.g., taking as the uncertainty the cross entropy loss  $\mathcal{L}_{\mathrm{CE}}(\zeta)$  of an instance  $\zeta$  on the downstream classifier [43, 54, 56, 58]. Comparatively, our definition of uncertainty,  $\phi_{\mathrm{uncertain}}(\zeta)$ , is in the representation space. The theoretical connection between these two losses is given in the following theorem.
Theorem 1 (Connection between uncertainties.). Let  $\mathcal{X}$ ,  $\mathcal{Z}$  and  $\mathcal{Y}$  be the input space, representation space and label set of downstream classifier. Denote a downstream classifier by  $h\colon \mathcal{Z}\to \mathcal{Y}$  and the
set of downstream classifiers by  $\mathcal{H}$ . Assume that the distribution of labels is a uniform distribution over  $\mathcal{V}$ . For any graph encoder  $f: \mathcal{X} \to \mathcal{Z}$ , one has
$$
\mathcal {L} _ {\mathrm {C E}} (\mathcal {X}) \geq \log \left(\frac {\log | \mathcal {Y} |}{\log 2 - \phi_ {\text {u n c e r t a i n}} (\mathcal {X})}\right),
$$
where  $\mathcal{L}_{\mathrm{CE}}$  denotes the conventional uncertainty, defined as cross entropy loss and estimated from the composition of graph encoder and downstream classifier  $h\circ f$ , and  $\phi_{\mathrm{uncertain}}$  is the proposed uncertainty estimated from graph encoder  $f$  (independent of the downstream classifier).
While the proof and additional discussion on the advantage of  $\phi_{\mathrm{uncertain}}$  are postponed to Appendix B, we emphasize here that, by Theorem 1, a smaller  $\mathcal{L}_{\mathrm{CE}}$  over all downstream classifiers cannot be achieved without a smaller  $\phi_{\mathrm{uncertain}}$ .
Although GCC is used as the backbone model in the presented framework, our data selection strategy can be easily adapted to other non-contrastive learning tasks. In that case, the InfoNCE loss used in  $\phi_{uncertain}$  should be replaced with another pre-training loss associated with the specific learning task. More details with the example of graph reconstruction is included in Appendix H.
Graph properties. As we see above, the predictive uncertainty measures the model's ability to distinguish (or identify) a given graph (or subgraph instance). However, predictive uncertainty is sometimes misleading, especially when the chosen graph (or subgraph) happens to be an outlier of the entire data collection. Hence, learning solely from the most uncertain data is not enough to boost the overall performance, or worse still, might lead to overfitting. The inherent properties of the graph turn out to be equivalently important as a selection criterion for graph pre-training. Intuitively, it is preferable to choose those graphs that are good by themselves, e.g., those with better structure, or those containing more information. To this end, we introduce five inherent properties of graphs, i.e., network entropy, density, average degree, degree variance and scale-free exponent, to help select better data points for pre-training. All these properties exhibit a strong correlation with downstream performance, which is empirically verified and presented in part (b) of the graph selector component in Figure 2. The choice of these properties also has an intuitive explanation, and here we discuss the intuition behind the network entropy as an example.
The use of network entropy is inspired from the sampling methods used in many graph pre-training models (see, e.g., [22, 50]). In those works, random walks are used to construct subgraph instances (used as model input). Random walks can also serve as a means to quantify the amount of information contained in a graph. In particular, the amount of information contained in a random walk from node  $v_{i}$  to node  $v_{j}$  is defined as  $-\log P_{ij}$  [6], where  $P$  is the transition matrix. Thus, the network entropy of a connected graph can be defined as the expected information of individual transitions over the random walk process [3]:
$$
\phi_ {\text {e n t r o p y}} = \left\langle - \log P \right\rangle_ {P} = - \sum_ {i, j} \pi_ {i} P _ {i j} \log P _ {i j}, \tag {2}
$$
where  $\pi$  is the stationary distribution of the random
walk and  $\langle \cdot \rangle_P$  denotes the expectation of a random variable according to  $P$ . Network entropy (2) is in general difficult to calculate, but is still tractable for special choices of the transition matrix  $P$ . For example, for a connected unweighted graph  $G = (V,E)$  with node degree vector  $\pmb{d} \in \mathbb{R}^{|V|}$ , if the transition matrix is defined by  $P_{ij} = 1 / d_i$ , then the stationary distribution is  $\pi = (1/2|E|)d$  and the network entropy (2) reduces to
$$
\phi_ {\text {e n t r o p y}} = \frac {1}{2 | E |} \sum_ {i = 1} ^ {| V |} d _ {i} \log d _ {i}. \tag {3}
$$
In this case, the network entropy of a graph depends solely on its degree distribution, which is straightforward and inexpensive to compute.
Although the definition of network entropy originates from random walks on graphs, it is still useful in graph pre-training even when the sampling of subgraph instances does not depend on random
Figure 3: Illustrative graphs with increasing network entropy (bottom left to top right), and the other four graph properties.
walks. Its usefulness can also be explained via the coding theory. Network entropy can be viewed as the entropy rate of a random walk, and it is known that the entropy rate is the expected number of bits per symbol required to describe a stochastic process [6]. Similarly, the network entropy can be interpreted as the expected number of "words" needed to describe the graph. Thus, intuitively, the larger the network entropy is, the more information the graph contains.
We also note that the connectivity assumption does not limit the usefulness of Eq. (3) in our case. For disconnected input graphs, we can simply compute the network entropy of the largest connected component, since for most real-world networks, the largest connected component contains most of the information [14]. Alternatively, we can also take some of the largest connected components from the graph and treat them separately as several connected graphs.
Furthermore, the other four graph properties, i.e., density, average degree, degree variance and scale-free exponent, are closely related to the network entropy. Figure 3 presents a clear correlation between the network entropy and the other four graph properties, as well as provides some illustrative graphs. (These example graphs are generated by the configuration model proposed in [47], and Appendix F contains more results on real-world networks.) Intuitively, graphs with higher network entropy contain a larger amount of information, and so are graphs with larger density, higher average degree, higher degree variance, or a smaller scale-free exponent. The connections between all five graph properties can also be theoretically justified and the motivations of choosing these properties can be found in Appendix A. The detailed empirical justification of these properties and the pre-training performance in included in Appendix F.
Time-adaptive selection strategy. The proposed predictive uncertainty and the five graph properties together act as a powerful indicator of a graph's suitability for a pre-training model. Then, we aim to select the graph with the highest score, where the score is defined as
$$
\mathcal {J} (G) = \left(1 - \gamma_ {t}\right) \hat {\phi} _ {\text {u n c e r t a i n}} + \gamma_ {t} \mathrm {M E A N} \left(\hat {\phi} _ {\text {e n t r o p y}}, \hat {\phi} _ {\text {d e n s i t y}}, \hat {\phi} _ {\text {a v g ＿ d e g}}, \hat {\phi} _ {\text {d e g ＿ v a r}}, - \hat {\phi} _ {\alpha}\right), \tag {4}
$$
where the optimization variable is the graph  $G$  to be selected,  $\gamma_t \in [0,1]$  is a trade-off parameter to balance the weight between predictive uncertainty and graph properties, and  $t$  is the iteration counter. The small hat on the terms  $\hat{\phi}_{\text{uncertain}}, \hat{\phi}_{\text{entropy}}, \hat{\phi}_{\text{density}}, \hat{\phi}_{\text{avg\_deg}}, \hat{\phi}_{\text{deg\_var}}$  and  $\hat{\phi}_{\alpha}$  indicates that all the values are already  $z$ -normalized, so the objective (especially the MEAN operator) is independent of their original scales.
Note that the pre-training model learns nothing at the beginning, so we initialize  $\gamma_0 = 1$ , and in later iterations, the balance between the predictive uncertainty and the inherent graph properties ensures that the selected graph is a good supplement to the current pre-training model as well as an effective representative for the entire data distribution. In particular, at the beginning of the pre-training, the outputs of the model are not accurate enough to guide data selection, so the parameter  $\gamma_t$  should be set larger so that the graph properties play a leading role. As the training phase proceeds, the graph selector gradually pays more attention to the feedback  $\phi_{uncertain}$  via a smaller value of  $\gamma_t$ . Therefore, in our framework, the parameter  $\gamma_t$  is called the time-adaptive parameter, and is set to be a random variable depending on time  $t$ . In this work, we take  $\gamma_t$  from a Beta distribution  $\gamma_t \sim \mathrm{Beta}(1, \beta_t)$ , where  $\beta_t$  decreases over time (training iterations).
3.2 Graph pre-training model
Instead of swallowing all the pre-training graphs as a whole, our graph pre-training model takes the input graphs and samples one by one in a sequential order and enhances itself in a progressive manner. However, such a straightforward sequential training does not guarantee that the model will remember all the contributions of previous input data. As shown in the orange curve in Figure 4, the previously learned graph exhibits a larger predictive uncertainty as the training phase proceeds.