Title: Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

pre-training
Abstract: Pre-training on graph neural networks (GNNs) aims to learn transferable knowledge for downstream tasks with unlabeled data, and it has recently become an active research area. The success of graph pre-training models is often attributed to the massive amount of input data. In this paper, however, we identify the curse of big data phenomenon in graph pre-training: more training data do not necessarily lead to better downstream performance. Motivated by this observation, we propose a better-with-less framework for graph pre-training: fewer, but carefully chosen data are fed into a GNN model to enhance pre-training. The proposed pre-training pipeline is called the data-active graph pre-training (APT) framework, and is composed of a graph selector and a pre-training model. The graph selector chooses the most representative and instructive data points based on the inherent properties of graphs as well as predictive uncertainty. The proposed predictive uncertainty, as feedback from the pre-training model, measures the confidence level of the model in the data. When fed with the chosen data, on the other hand, the pre-training model grasps an initial understanding of the new, unseen data, and at the same time attempts to remember the knowledge learned from previous data. Therefore, the integration and interaction between these two components form a unified framework (APT), in which graph pre-training is performed in a progressive and iterative way. Experiment results show that the proposed APT is able to obtain an efficient pre-training model with fewer training data and better downstream performance.

Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks

Abstract

Pre-training on graph neural networks (GNNs) aims to learn transferable knowledge for downstream tasks with unlabeled data, and it has recently become an active research area. The success of graph pre-training models is often attributed to the massive amount of input data. In this paper, however, we identify the curse of big data phenomenon in graph pre-training: more training data do not necessarily lead to better downstream performance. Motivated by this observation, we propose a better-with-less framework for graph pre-training: fewer, but carefully chosen data are fed into a GNN model to enhance pre-training. The proposed pre-training pipeline is called the data-active graph pre-training (APT) framework, and is composed of a graph selector and a pre-training model. The graph selector chooses the most representative and instructive data points based on the inherent properties of graphs as well as predictive uncertainty . The proposed predictive uncertainty, as feedback from the pre-training model, measures the confidence level of the model in the data. When fed with the chosen data, on the other hand, the pretraining model grasps an initial understanding of the new, unseen data, and at the same time attempts to remember the knowledge learned from previous data. Therefore, the integration and interaction between these two components form a unified framework (APT), in which graph pre-training is performed in a progressive and iterative way. Experiment results show that the proposed APT is able to obtain an efficient pre-training model with fewer training data and better downstream performance.

1 Introduction

Pre-training Graph neural networks (GNNs) shows great potential to be an attractive and competitive strategy for learning from graph data without costly labels [29, 50]. Recent advancements have been made in developing various graph pre-training strategies, which aim to capture transferable patterns from a diverse set of unlabeled graph data [22, 30, 31, 40, 44, 60, 76, 77]. Very often, the success of a graph pre-training model is attributed to the massive amount of unlabeled training data, a well-established consensus for pre-training in computer vision [12, 19, 26] and natural language processing [11,45].

In view of this, contemporary research almost has no controversy on the following issue: Is a massive amount of input data really necessary, or even beneficial, for pre-training GNNs? Yet, two simple experiments regarding the number of training samples (and graph datasets) seem to doubt the positive

msrc-21

wisconsin

imdb-binary

imdb-binary

msrc-21

wisconsin

brazil

brazil

# of graphs

# of graphs

# of graphs

# of graphs

# of samples

# of samples

# of samples

# of samples

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

(a) Node classification

(b) Graph classification

std : 3.1

std :1.5

std : 1.4

std :1.4

model config

.

5 layers

4 layers

3 layers

(all)

(all)

(all)

(all)

answer to this question. We observe that scaling pre-training samples does not result in a one-modelfits-all increase in downstream performance (top row in Figure 1), and that adding input graphs (while fixing sample size) does not improve and sometimes even deteriorates the generalization of the pre-trained model (bottom row in Figure 1). Moreover, even if the number of input graphs (the horizontal coordinate) is fixed, the performance of the model pre-trained on different combinations of inputs varies dramatically; see the standard deviation in blue in Figure 1. As the first contribution of this work, we identify the curse of big data phenomenon in graph pre-training: more training samples (and graph datasets) do not necessarily lead to better downstream performance.

Therefore, instead of training on a massive amount of data, it is more appealing to choose a few suitable samples and graphs for pre-training. However, without the knowledge of downstream tasks, it is difficult to design new data selection criteria for the pre-training model. To this end, we propose the graph selector which is able to provide the most instructive data for the model by incorporating two criteria: predictive uncertainty and graph properties . On one hand, predictive uncertainty is introduced to measure the level of confidence (or certainty) in the data. On the other hand, some graphs are inheritantly more informative and representative than others, and thus the fundamental properties should help in the selection process.

Apart from the graph selector, the pre-training model is also designed to co-evolve with the data. Instead of swallowing data as a whole, the pre-training model is encouraged to learn from the data in a progressive way. After learning a certain amount of training data, the model receives feedback (from predictive uncertainty) on what kind of data the model has least knowledge of. Then the pre-training model is able to reinforce itself on highly uncertain data in next training iterations.

Putting together, we integrate the graph selector and the pre-training model into a unified paradigm and propose a data-active graph pre-training (APT) framework. The term 'data-active ' is used to emphasize the co-evolution of data and model, rather than mere data selection before model training. The two components in the framework actively cooperate with each other. The graph selector recognizes the most instructive data for the model; equipped with this intelligent selector, the pre-training model is well-trained and in turn provides better guidance for the graph selector.

The rest of the paper is organized as follows. In §2 we present the basic graph pre-training framework and review existing work on this topic. Then in §3 we describe in detail the proposed data-active graph pre-training (APT) paradigm. §4 contains numerical experiments, demonstrating the superiority of APT in different downstream tasks, and also includes the applicable scope of our pre-trained model.

2 Basic Graph Pre-training Framework

This section reviews the basic framework of graph pre-training commonly used in the literature. The backbone of our graph pre-training model also follows this framework.

We start with a natural question: What does graph pre-training actually learn? On one hand, graph pre-training tries to learn transferable semantic meaning associated with structural patterns. For example, both in citation networks and social networks, the closed triangle structure ( ) is interpreted as a stable relationship, while the open triangle ( ) indicates an unstable relationship. In comparison, these semantic meanings can be quite different in other networks, e.g. , molecular networks. On the other hand, however, the distinction (or relationship) between different structural patterns is still transferable. Taking the same example, the closed and open triangles might yield different interpretations in molecular networks (stability of certain chemical property) from those in social networks (stability in social intimacy), but the distinction between these two structures remains the same: they indicate opposite (or contrastive) semantic meanings [32,41,57]. Therefore, the graph pre-training either learns representative structural patterns (when semantic meanings are present), or more importantly, obtains the capability of distinguishing these patterns. This observation in graph pre-training is not only different from that in other areas ( e.g. , computer vision and natural language processing), but may also explain why graph pre-training is effective, especially when some downstream information is absent.

With the hope to learn the transferable structural patterns or the ability to distinguish them, the graph pre-training model is fed with a diverse collection of input graphs, and the learned model, denoted by f θ (or simply f if the parameter θ is clear from context), maps a node to a low-dimensional representation. Unaware of the specific downstream task as well as task-specific labels, one typically designs a self-supervised task for the pre-training model. Such self-supervised information for a node is often hidden in its neighborhood pattern, and thus the structure of its ego network can be used as the transferable pattern. Naturally, subgraph instances sampled from the same ego network Γ i are considered similar while those sampled from different ego networks are rendered dissimilar . Therefore, the pre-training model attempts to capture the similarities (and dissimilarities) between subgraph instances, and such a self-supervised task is called the subgraph instance discrimination task . More specifically, given a subgraph instance ζ i from an ego network Γ i centered at node v i as well as its representation x i = f ( ζ i ) , the model f aims to encourage higher similarity between x i and the representation of another subgraph instance ζ + i sampled from the same ego network. This can be done by minimizing, e.g. , the InfoNCE loss [48],

̸

where Ω -i is a collection of subgraph instances sampled from different ego networks Γ j ( j = i ), and τ > 0 is a temperature hyper-parameter. Here the inner product is used as a similarity measure between two instances. One common strategy to sample these subgraph instances is via random walks on graphs, as used in GCC [50], and other sampling methods as well as loss functions are also valid.

3 Data-Active Graph Pre-training

In this section, we present the proposed APT framework for graph pre-training, and the overall pipeline is illustrated in Figure 2. The APT framework consists of two major components: a graph selector and a graph pre-training model. The technical core is the interaction between these two components: The graph selector feeds suitable data for pre-training, and the graph pre-training model learns from the carefully chosen data. The feedback of the pre-training model in turn helps select the needed data tailored to the model.

The rest of this section is organized as follows. We describe the graph selector in §3.1 and the graph pre-training model in §3.2. The overall pre-training and fine-tuning strategy is presented in §3.3.

<latexi sh

1_b

64="nKog

8C95y

F

qO

/Vj mu3

M

>A

c

ZDLS

N

IY

X

R

2

G

JUQd0

T

k

r+

f

v

w

z

B

p

W

7

E

P

H

<latexi sh

1_b

64="nKog

8C95y

F

qO

/Vj mu3

M

>A

c

ZDLS

N

IY

X

R

2

G

JUQd0

T

k

r+

f

v

w

z

B

p

W

7

E

P

H

<latexi sh

1_b

64="R

MEzc+g

y

C

X

SQ

w

5

>A

n

Z

L

N

FIY

9V

r

W

k

o

uH

U

K

0

8

T

fB

/

7

q

3

d

j

m

v

p

P

G

2

O

graph pre-training model

graph selector

pre-training

data

dim2

dim1

-

-

-

decision

boundary

similar/dissimilar instance of

+

-

/

(b)

Graph properties

?

?

query instance

with most

uncertainty

(a)

Predictive uncertainty

Combined for graph

selector

The graph and samples

friendly to pre-training

Pearson

correlation

íGLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=25,font=/AAAAAG+TimesNewRomanPSMT>

íGLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=23,font=/AAAAAG+TimesNewRomanPSMT>

íGLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=21,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=21,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=23,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=25,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=27,font=/AAAAAG+TimesNewRomanPSMT>

cora

pubmed

brazil

dd68

wisconsin

cornell

micro F1 on

?

+

graph model

input data

…

new

data

old

data

regularization

for old data

loss for

new data

forwardprop

backprop

∂

∂θ

íGLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=25,font=/AAAAAG+TimesNewRomanPSMT>

íGLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=23,font=/AAAAAG+TimesNewRomanPSMT>

íGLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=21,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=21,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=23,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=25,font=/AAAAAG+TimesNewRomanPSMT>

GLYPH<c=19,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=17,font=/AAAAAG+TimesNewRomanPSMT>GLYPH<c=27,font=/AAAAAG+TimesNewRomanPSMT>

network entropy

density

average degree

scale-free exponent

other properties

∂

∂θ

∂

∂θ

-1.00.0 1.0

dim3

select

added to

provided to

predictive

uncertainty

3.1 Graph selector

In view of the curse of big data phenomenon, it is more appealing to carefully choose data well suited for graph pre-training rather than training on a massive amount of data. Conventionally, the criterion of suitable data, or the contribution of a data point to the model, is defined based on the output predictions on downstream tasks [20]. In graph pre-training where downstream information is absent, new selection criteria or guidelines are needed to provide effective instructions for the model. Here we introduce two kinds of selection criteria, originating from different points of view, to help select suitable data for pre-training. The predictive uncertainty measures the model's understanding of certain data, and thus helps select the least certain data points for the current model. In addition to the measure of model's ability, some inherent properties of graphs can also be used to assess the level of representativeness or informativeness of a given graph.

Predictive uncertainty. The notion of predictive uncertainty can be explained via an illustrative example, as shown in part (a) of the graph selector component in Figure 2. Consider a query subgraph instance ζ i (denote by ? in Figure 2) from the ego network Γ i in a graph G . If the pre-training model cannot tell its similar instance ζ + i (denoted by + ) from its dissimilar instance ζ -i ∈ Ω -i (denoted by GLYPH<c=16,font=/DBNFMT+PingFangSC-Regular> ), we say that the current model is uncertain about the query instance ζ i . Therefore, the contrastive loss function in Eq. (1) comes in handy as a natural measure for the predictive uncertainty of the instance ζ i : ϕ uncertain ( ζ i ) = L i . Accordingly, the predictive uncertainty of a graph G ( i.e. , the graph-level predictive uncertainty) is defined as ϕ uncertain ( G ) = (1 /M ) ∑ M i =1 L i , where M is the number of subgraph instances queried in this graph.

We further establish a provable connection between the proposed predictive uncertainty and the conventional definition of uncertainty. In most existing work, model uncertainty is often defined in the label space, e.g. , taking as the uncertainty the cross entropy loss L CE ( ζ ) of an instance ζ on the downstream classifier [43,54,56,58]. Comparatively, our definition of uncertainty, ϕ uncertain ( ζ ) , is in the representation space. The theoretical connection between these two losses is given in the following theorem.

Theorem 1 (Connection between uncertainties.) . Let X , Z and Y be the input space, representation space and label set of downstream classifier. Denote a downstream classifier by h : Z → Y and the

set of downstream classifiers by H . Assume that the distribution of labels is a uniform distribution over Y . For any graph encoder f : X → Z , one has where L CE denotes the conventional uncertainty, defined as cross entropy loss and estimated from the composition of graph encoder and downstream classifier h ◦ f , and ϕ uncertain is the proposed uncertainty estimated from graph encoder f (independent of the downstream classifier).

While the proof and additional discussion on the advantage of ϕ uncertain are postponed to Appendix B, we emphasize here that, by Theorem 1, a smaller L CE over all downstream classifiers cannot be achieved without a smaller ϕ uncertain .

Although GCC is used as the backbone model in the presented framework, our data selection strategy can be easily adapted to other non-contrastive learning tasks. In that case, the InfoNCE loss used in ϕ uncertain should be replaced with another pre-training loss associated with the specific learning task. More details with the example of graph reconstruction is included in Appendix H.

Graph properties. As we see above, the predictive uncertainty measures the model's ability to distinguish (or identify) a given graph (or subgraph instance). However, predictive uncertainty is sometimes misleading, especially when the chosen graph (or subgraph) happens to be an outlier of the entire data collection. Hence, learning solely from the most uncertain data is not enough to boost the overall performance, or worse still, might lead to overfitting. The inherent properties of the graph turn out to be equivalently important as a selection criterion for graph pre-training. Intuitively, it is preferable to choose those graphs that are good by themselves , e.g. , those with better structure, or those containing more information. To this end, we introduce five inherent properties of graphs, i.e. , network entropy, density, average degree, degree variance and scale-free exponent, to help select better data points for pre-training. All these properties exhibit a strong correlation with downstream performance, which is empirically verified and presented in part (b) of the graph selector component in Figure 2. The choice of these properties also has an intuitive explanation, and here we discuss the intuition behind the network entropy as an example.

The use of network entropy is inspired from the sampling methods used in many graph pre-training models (see, e.g. , [22, 50]).In those works, random walks are used to construct subgraph instances (used as model input). Random walks can can also serve as a means to quantify the amount of information contained in a graph. In particular, the amount of information contained in a random walk from node v i to node v j is defined as -log P ij [6], where P is the transition matrix. Thus, the network entropy of a connected graph can be defined as the expected information of individual transitions over the random walk process [3]:

where π is the stationary distribution of the random walk and ⟨·⟩ P denotes the expectation of a random variable according to P . Network entropy (2) is in general difficult to calculate, but is still tractable for special choices of the transition matrix P . For example, for a connected unweighted graph G = ( V, E ) with node degree vector d ∈ R | V | , if the transition matrix is defined by P ij = 1 /d i , then the stationary distribution is π = (1 / 2 | E | ) d and the network entropy (2) reduces to

normalized value of structural properties

network entropy

density

negative scale-free exponent

average degree

common structural properties

(

i.e.

, average degree, density, degree variance or negative scale-free exponent)

less

more

Informativeness

informativeness

In this case, the network entropy of a graph depends solely on its degree distribution, which is straightforward and inexpensive to compute.

Although the definition of network entropy originates from random walks on graphs, it is still useful in graph pre-training even when the sampling of subgraph instances does not depend on random

walks. Its usefulness can also be explained via the coding theory. Network entropy can be viewed as the entropy rate of a random walk, and it is known that the entropy rate is the expected number of bits per symbol required to describe a stochastic process [6]. Similarly, the network entropy can be interpreted as the expected number of 'words' needed to describe the graph. Thus, intuitively, the larger the network entropy is, the more information the graph contains.

We also note that the connectivity assumption does not limit the usefulness of Eq. (3) in our case. For disconnected input graphs, we can simply compute the network entropy of the largest connected component, since for most real-world networks, the largest connected component contains most of the information [14]. Alternatively, we can also take some of the largest connected components from the graph and treat them separately as several connected graphs.

Furthermore, the other four graph properties, i.e. , density, average degree, degree variance and scalefree exponent, are closely related to the network entropy. Figure 3 presents a clear correlation between the network entropy and the other four graph properties, as well as provides some illustrative graphs. (These example graphs are generated by the configuration model proposed in [47], and Appendix F contains more results on real-world networks.) Intuitively, graphs with higher network entropy contain a larger amount of information, and so are graphs with larger density, higher average degree, higher degree variance, or a smaller scale-free exponent. The connections between all five graph properties can also be theoretically justified and the motivations of choosing these properties can be found in Appendix A. The detailed empirical justification of these properties and the pre-training performance in included in Appendix F.

Time-adaptive selection strategy. The proposed predictive uncertainty and the five graph properties together act as a powerful indicator of a graph's suitability for a pre-training model. Then, we aim to select the graph with the highest score, where the score is defined as

where the optimization variable is the graph G to be selected, γ t ∈ [0 , 1] is a trade-off parameter to balance the weight between predictive uncertainty and graph properties, and t is the iteration counter. The small hat on the terms ˆ ϕ uncertain , ˆ ϕ entropy , ˆ ϕ density , ˆ ϕ avg_deg , ˆ ϕ deg_var and ˆ ϕ α indicates that all the values are already z -normalized, so the objective (especially the MEAN operator) is independent of their original scales.

Note that the pre-training model learns nothing at the beginning, so we initialize γ 0 = 1 , and in later iterations, the balance between the predictive uncertainty and the inherent graph properties ensures that the selected graph is a good supplement to the current pre-training model as well as an effective representative for the entire data distribution. In particular, at the beginning of the pre-training, the outputs of the model are not accurate enough to guide data selection, so the parameter γ t should be set larger so that the graph properties play a leading role. As the training phase proceeds, the graph selector gradually pays more attention to the feedback ϕ uncertain via a smaller value of γ t . Therefore, in our framework, the parameter γ t is called the time-adaptive parameter , and is set to be a random variable depending on time t . In this work, we take γ t from a Beta distribution γ t ∼ Beta (1 , β t ) , where β t decreases over time (training iterations).

3.2 Graph pre-training model

Instead of swallowing all the pre-training graphs as a whole, our graph pre-training model takes the input graphs and samples one by one in a sequential order and enhances itself in a progressive manner. However, such a straightforward sequential training does not guarantee that the model will remember all the contributions of previous input data. As shown in the orange curve in Figure 4, the previously learned graph exhibits a larger predictive uncertainty as the training phase proceeds.

The empirical result indicates that the knowledge or information contained in previous input data will be forgotten or covered by newly incoming data. This phenomenon, called catastrophic forgetting , was first noticed in continual learning [35] and also

50

55

60

65

70

75

80

85

90

training epoch

4.5

5.0

5.5

6.0

6.5

predictive uncertainty

without proximal term

with proximal term (L2)

appears in our case. Intuitively, when the training data is taken in a progressive or iterative manner, the learned parameters will cater to the newly incoming data and forget the old, previous data points.

One remedy for this issue is adding a proximal term to the objective. The additional proximal term ( i.e. , the regularization) guarantees the proximity between the new parameters and the model parameters learned from previous graphs. Therefore, when the model is learning the k -th input graph, the loss function for our pre-training model in APT is

where L i is given in Eq. (1), the first summation is taken over the subgraph instances sampled from the latest input graph, θ ( k -1) is the model parameters after learning from the first k -1 graphs, and the parameter λ describes the trade-off between the knowledge learnt from new data and that from previous data. Here, F ( k -1) is the Fisher information matrix of θ ( k -1) , and F ( k ) jj is its j -th diagonal element. When F is set as an identity matrix, the second term degenerates to the L2 regularization (which serves as one of our variants). The proximal term in Eq. (5) is absent when the first input graph is introduced to the model, and the term is applied on the first three layers of the pre-training model. Finally, we note that the total number of parameters in the pre-training model is in the same order of magnitude as classical GNNs, so the memory cost would not be a bottleneck.

3.3 Training and fine-tuning

Integrating the graph selector and the pre-training model forms the entire APT framework, and the overall algorithm is presented in Appendix C. After the training phase, the APT framework returns a pre-trained GNN model, and then the pre-trained model is applied to various downstream tasks from a wide spectrum of domains. In the so-called freezing mode , the pre-trained model returned by APT is directly applied to downstream tasks, without any changes in parameters. Alternatively, the fine-tuning mode uses the pre-trained graph encoder as initialization, and offers the flexibility of training the graph encoder and the downstream classifier together in an end-to-end manner.

4 Experiments

In the experiments, we pre-train a graph representation model via the proposed APT framework, and then evaluate the transferability of our pre-trained model on multiple unseen graphs in the node classification and graph classification task. Lastly, we include the applicable scope of our pre-trained model. Additional experiments can be found in Appendix H, including our adaptation to backbone pre-training models GraphCL, JOAO and Mole-BERT, training time, impact of different graph properties, analysis of the ablation studies and selected pre-training graphs, hyper-parameter sensitivity, explorations of various combinations of graph properties.

4.1 Experimental setup

Datasets. The datasets for pre-training and testing, along with their statistics, are listed in Appendix D. Pre-training datasets are collected from different domains, including social, citation, and movie networks. Testing datasets comprise both in-domain ( e.g. , citation, movie) and crossdomain ( e.g. , image, web, protein, transportation and others) datasets to evaluate transferability comprehensively, also including large-scale datasets with millions of edges sourced from [28].

Baselines. We evaluate our model against the following baselines for node and graph classification tasks. For node classification, ProNE [80], DeepWalk [49], struc2vec [53], DGI [65], GAE [34], and GraphSAGE [23] are used as baselines, and then the learned representations are fed into the logistic regression, as most of baselines did. As for graph classification, we take graph2vec [46], InfoGraph [59], DGCNN [81] and GIN [71] as baselines, using SVM as the classifier, which aligns with the methodology of most baselines. For both tasks, we also compare our model with (1) Random: random vectors are generated as representations; (2) GraphCL [76]: a GNN pre-training scheme based on contrastive learning with augmentations; (3) JOAO [75]: a GNN pre-training scheme that automatically selects data augmentations; (4) GCC [50]: the state-of-the-art cross-domain graph pre-training model (which is our model's version without the data selector, trained on all pre-training

Method

Dataset

brazil

dd68

dd687

wisconsin

cornell

cora

pubmed

ogbarxiv

ogbproteins

A.R.

Random

32.16(13.65)

6.71(2.44)

8.29(4.35)

5.98(2.70)

26.79(8.59)

39.77(7.26)

26.80(1.62)

38.85(0.76)

11.89(4.66)

52.69(5.94)

13.2

ProNE

50.24(11.56)

10.04(2.56)

7.73(3.11)

3.88(1.66)

44.67(7.49)

47.32(12.14) 80.76(2.92)

∗

78.80(0.98)

∗

65.96(0.06)

∗

76.28(0.34)

∗

7.5

DeepWalk

43.16(16.78)

8.11(1.45)

6.72(3.04)

6.17(2.18)

39.61(9.11)

47.67(8.30)

49.85(9.26)

44.99(10.89)

16.04 (2.98)

64.74(0.49)

8.7

struc2vec

25.54(11.74)

13.71(2.66)

10.30(3.23)

7.98(2.74)

45.39(7.36)

38.39(9.18)

36.01(2.41)

44.45(0.78)

/

/

10.6

DGI

56.44(7.79)

14.35(0.44)

13.57(0.44)

11.04(1.93)

49.46(5.46)

49.85(9.26)

30.02(0.44)

42.39(0.84)

12.93(7.67)

55.98(0.33)

6.6

GAE

57.88(10.68)

14.09(1.52)

13.43(0.96)

10.25(2.63)

45.78(4.18)

49.26(5.24)

30.10(0.31)

40.14(0.68)

/

/

8.5

GraphSAGE

67.93(9.28)

14.33(0.37)

13.55(0.70)

10.39(0.78)

47.03(1.98)

49.20(1.46)

35.93(1.76)

39.94(0.02)

/

/

7.1

GraphCL (freeze)

50.71(5.00)

9.53(2.50)

9.36(3.63)

6.03(1.86)

38.85(10.80)

41.05(5.67)

16.95(2.39)

41.07(1.16)

/

/

12.2

JOAO (freeze)

71.22(7.21)

7.98(2.90)

12.36(2.59)

5.34(1.43)

42.69(8.15)

43.16(5.67)

18.13(2.82)

41.05(0.87)

/

/

10.9

GCC (freeze)

67.47(4.09)

15.83(0.80)

11.95(1.13)

9.61(0.94)

52.57(1.69)

46.87(1.73)

35.47(0.51)

46.40(0.18)

14.56(7.60)

59.15(0.35)

7.0

APT-G (freeze)

68.69(3.42)

17.21(1.13)

11.98(0.75)

9.54(1.29)

54.45(1.90)

46.53(1.59)

34.89(0.25)

46.49(0.22)

12.32(7.71)

60.38(0.41)

6.3

APT-P (freeze)

66.55(2.35)

16.58(0.97)

12.48(0.85)

10.33(0.83)

51.90(1.64)

47.33(2.31)

35.63(0.56)

46.16(0.12)

12.86(7.54)

60.32(0.32)

6.2

APT-R (freeze)

68.12(3.07)

16.72(0.72)

12.42(1.24)

11.05(0.88)

54.48(1.77)

46.80(1.08)

34.93(0.36)

46.02(0.11)

18.79(5.87)

62.18(0.46)

5.0

APT-L2 (freeze)

69.82(2.32)

16.79(0.88)

12.68(0.81)

10.34(1.12)

55.11(1.74)

48.76(2.20)

34.27(0.43)

46.21(0.15)

19.64(6.46)

60.23(0.37)

4.4

APT (freeze)

73.39(2.55)

16.57(0.94)

12.08(0.89)

10.35(1.24)

53.38(1.19)

47.37(1.29)

36.69(0.49)

46.88(0.21)

22.04(0.29)

62.29(0.55)

3.8

GraphCL (rand, fine-tune)

64.43(14.95)

15.04(0.85)

14.69(2.48)

10.99(0.58)

63.85(2.18)

44.21(10.58)

30.45(0.37)

40.73(0.66)

/

/

8.3

JOAO (rand, fine-tune)

72.14(6.74)

10.93(2.85)

8.08(2.15)

7.40(3.48)

45.38(13.30) 45.26(10.31)

29.93(2.84)

42.01(0.68)

/

/

9.6

GCC (rand, fine-tune)

58.51(3.07)

15.98(1.05)

13.16(1.06)

9.74(0.95)

53.85(2.58)

50.95(2.26)

43.70(0.52)

49.72(0.17)

18.61(1.88)

59.12(0.35)

7.6

GraphCL (fine-tune)

73.57(10.33)

15.35(0.99)

13.51(2.57)

10.66(1.04)

63.85(4.42)

51.05(2.41)

30.81(0.36)

42.91(0.91)

/

/

7.5

JOAO (fine-tune)

75.00(5.76)

10.54(3.07)

7.56(1.94)

8.77(2.39)

50.0(12.28)

42.11(10.26)

29.34(3.04)

42.21(0.88)

/

/

9.5

GCC (fine-tune)

74.46(3.05)

19.32(0.80)

13.87(1.13)

10.37(1.06)

59.47(1.49)

48.32(2.42)

43.34(0.38)

50.87(0.19)

18.62(1.92)

60.08(0.56)

6.4

APT-G (fine-tune)

77.60(1.48)

25.45(0.60)

∗

17.78(1.14)

11.27(0.76)

66.09(2.28)

53.02(1.51)

45.63(0.66)

50.81(0.18)

27.33(4.80)

60.02(0.32)

3.8

APT-P (fine-tune)

78.99(2.44)

25.19(0.87)

16.40(1.22)

11.69(1.19)

64.24(1.90)

50.05(1.39)

45.53(0.30)

50.66(0.18)

27.20(4.80)

59.86(0.32)

4.8

APT-R (fine-tune)

79.14(1.97)

24.96(0.57)

17.43(1.05)

11.29(1.04)

66.28(1.94)

53.56(2.28)

∗

46.02(0.83)

51.00(0.21)

18.41(1.84)

60.10(0.38)

3.4

APT-L2 (fine-tune)

78.75(1.63)

24.62(0.90)

17.83(1.35)

∗

12.26(0.78)

∗

67.04(1.50)

∗

52.94(1.95)

47.48(0.46)

51.25(0.21)

27.40(4.97)

60.85(0.46)

2.6

APT (fine-tune)

79.67(2.30)

∗

28.62(0.55)

∗

20.30(1.13)

∗

12.80(1.54)

∗

67.08(1.75)

∗

52.15(2.25)

47.51(0.62)

51.30(0.16)

27.40(4.87)

61.64(0.35)

1.3

data). GCC, GraphCL and JOAO are trained on the entire collected input data, and the suffix (rand, fine-tune) indicates whether the model is trained from scratch. We also include 9 variants of our model: (1) APT-G: removes the criteria of graph properties in the graph selector; (7) APT-P: removes the criteria of predictive uncertainty in the graph selector; (8) APT-R: removes the regularization w.r.t old knowledge in Eq. (5); (8) APT-L2: degenerates the second term in Eq. (5) with L2 regularization.

Experimental settings. In the training phase, we aim to utilize data from different domains to pre-train one graph model. We iteratively select graphs for pre-training until the predictive uncertainty of any candidate graph is below 3.5. For each selected graph, we choose samples with predictive uncertainty higher than 3. We set the number of subgraph instances queried in the graph for uncertainty estimation M as 500 . The time-adaptive parameter γ t in Eq. (4) follows a γ t ∼ Beta (1 , β t ) , where β t = 3 -0 . 995 t . We set the trade-off parameter λ = 10 for APT-L2, and λ = 500 for APT. The total iteration number is 100. We adopt GCC as the backbone pre-training model with its default hyper-parameters, including their subgraph instance definition. In the fine-tuning phase, we select logistic regression or SVM as the downstream classifier and adopt the same setting as GCC. Due to space limit, results of our adaption on other backbones like GraphCL, JOAO and Mole-BERT, as well as details can be found in Appendices H and G. See Appendix G for details.

We provide an open-source implementation of our model APT at https://github.com/ galina0217/APT .

4.2 Experimental results

Node classification. Table 1 presents the micro F1 score of different methods over 10 unseen graphs from a wide spectrum of domain for node classification task. We observe that our model beats the graph pre-training competitor by an average of 9 . 94% and 17 . 83% under freezing and fine-tuning mode respectively. This suggests that instead of pre-training on all the collected graphs (like GCC), it is better to choose a part of graphs better suited for pre-training (like our model APT).

Moreover, compared with the traditional models without pre-training, the performance gain of our model is attributed to the transferable knowledge learned by pre-training strategies. We also find that some proximity-based models, like ProNE, enforce neighboring nodes to share similar representations, leading to superior performance on graphs with strong homophily rather than weak homophily. Nonetheless, when applied to different graphs, these models require re-training, which makes them non-transferable. In contrast, we target at a general, transferable model free from specific assumptions on graph types, and thus applicable across wide scenarios (including both homophilic and heterophilic graphs). Ablation study reveals the necessity of all the components in APT ( e.g.,

the proximity regularization with respect to old knowledge, proximity term, graph properties and predictive uncertainty). We also explore the impacts of the five graph properties used in our model, and demonstrate their indispensability by experiments. Thus, combining all graph properties is essential to boost the performance of APT. Details in ablation study and more experimental results can be found in Appendix H.

Graph classification. The micro F1 score on unseen test data in the graph classification task is summarized in Table 2. Especially, our model is 7 . 2% and 1 . 3% on average better than the graph pre-training backbone model under freezing and fine-tuning mode, respectively. Interestingly, we observe that all variants of APT perform quite well, and thus, a simpler yet well-performed version of APT could be used in practice. This phenomenon could happen since "graph pre-train and fine-tune" is an extremely complicated non-convex optimization problem. Another observation is that in specific cases, such as dd dataset, a decrease in downstream performance after fine-tuning is observed, as compared to the freezing mode. This could happen since 'graph pre-train and fine-tune' is an extremely complicated non-convex optimization problem. A similar observation has been made in previous work [37] as well.

Finally, it is worth noting that our APT model achieves a training time 2.2 × faster than the competitive model GCC, achieved through a reduced number of carefully selected training graphs and samples. More specifically, we carefully selected only 7 datasets out of the available 11 and performed pre-training using at most 24.92% of the samples in each selected dataset. Moreover, for each newly added dataset, our model only needs a few more training iterations to convergence, rather than being trained from scratch.

4.3 Discussion: scope of application

The transferability of the pre-trained model comes from the learned representative structural patterns and the ability to distinguish these patterns (as discussed in §2). Therefore, our pretraining model is more suitable for the datasets where the target ( e.g. , labels) is correlated with

subgraph patterns or structural properties ( e.g. , motifs, triangles, betweenness, stars). For example, for node classification on heterophilous graphs ( e.g. , winconsin, cornell), our model performs very well because in these graphs, nodes with the same label are not directly connected, but share similar structural properties and behavior (or role, position). On the contrary, graphs with strong homophily (like cora, pubmed, ogbarxiv and ogbproteins) may not benefit too much from our models. Similar observation can also be made on graph classification: our model could also benefit the graphs whose label has a strong relationship with their structure, like molecular, chemical, and protein networks ( e.g. , dd in our experiments) [17, 66].

5 Related Work

Graph pre-training. Inspired by pre-training in CV/NLP, recent efforts have shed light on pre-training on GNNs. Initially, unsupervised graph representation learning is used for graph pretraining [13, 21, 23, 46, 53, 63, 80]. The design of unsupervised models is largely based on the neighborhood similarity assumption, and thus cannot generalize to unseen graphs. More recently, self-supervised graph learning emerges as another line of research, including graph generative and contrastive models. Graph generative models aim to capture the universal graph patterns by recovering certain parts of input graphs [7, 27, 30, 34, 67], but they rely heavily on domain-specific knowledge. In comparison, contrastive models maximize the agreement between positive pairs and minimize that between negative pairs [24,25,29,39,40,44,59,60,65,69,70,76,78,83,85-87]. Some work in

this direction takes subgraph sampling as augmentation, in the hope that the transferable subgraph patterns can be captured during pre-training. However, all the aforementioned studies only focus on the design of pre-training models, rather than suitable selection of data for pre-training.

Uncertainty-based sample selection. The terminology uncertainty is widely used in machine learning, without a universally-accepted definition. In general, this term refers to the lack of confidence of an ML model in certain model parameters. The majority of existing works define uncertainty in the label space, such as taking the uncertainty as the confidence level about the prediction [16, 18, 42, 61]. Only a few works define uncertainty in the representation space [52, 68]. In [52], uncertainty is measured based on the representations of an instance's nearest neighbors with the same label. However, this approach requires access to the label information of the neighbors, and thus cannot be adapted in pre-training with unlabeled data. [68] introduces a pretext task for training a model of uncertainty over the learned representations, but this method assumes a well-pre-trained model is already available. Such a post processing manner is not applicable to our scenario, because we need an uncertainty that can guide the selection of data during pre-training rather than after pre-training.

Some works on active learning and hard example mining (HSM) have also introduced concept similar to uncertainty. In active learning, uncertainty is measured via classification prediction, and the active learning model focuses on those samples which the model is least certain about [4,74,82,84]. However, these techniques all rely on the labels and cannot be adapted in pre-training with unlabeled data. As another line of work, HSM introduces similar strategies and works on those samples with the greatest loss, which can also be regarded as a kind of uncertainty [36,43,55,56]. Nevertheless, existing HSM approaches do not meet the following two requirements needed in our setting. (1) The chosen instances should follow a joint distribution that reflects the topological structures of real-world graphs. This is satisfied by our use of graph-level predictive uncertainty and graph properties, but is not met in HSM. (2) The chosen set of graphs should include informative and sufficiently diverse instances. This is only achieved by the proposed APT framework while existing HSM methods fail to consider this requirement.

Pre-training in CV and NLP. For pre-training in CV and NLP, scaling up the pre-training data size often results in a better or saturating performance in the downstream [1, 15, 33, 51, 62]. In view of this, data selection is not an active research direction for CV and NLP. Existing studies mainly focus on selecting pre-training data that closely matches the downstream domain [2,5,8-10,38,72]. The assumption on downstream domain knowledge differs from our graph pre-training setting, making such data selection less relevant to our work.

Data-centric AI. This recently introduced concept emphasizes the enhancement of data quality and quantity, rather than model design [73,79]. Following-up works in graph pre-training [24,64] exploits the data-centric idea to design data augmentation. For example, [24] introduces a graph data augmentation method by interpolating the generator of different classes of graphs. [64] mainly focuses on the theoretical analysis of data-centric properties of data augmentation. While many of these works advocate for shifting the focus to data, they do not consider the co-evolution of data and model, as is the case in our work.

6 Conclusion

In this paper, we identify the curse of big data phenomenon for pre-training graph neural networks (GNNs). This observation then motivates us to choose a few suitable graphs and samples for GNN pre-training rather than training on a massive amount of unselected data. Without any knowledge of the downstream tasks, we propose a novel graph selector to provide the most instructive data for pre-training. The pre-training model is then encouraged to learn from the data in a progressive and iterative way, reinforce itself on newly selected data, and provide instructive feedback to the graph selector for further data selection. The integration of the graph selector and the pre-training model into a unified framework forms a data-active graph pre-training (APT) paradigm, in which the two components are able to mutually boost the capability of each other. Extensive experimental results verify that the proposed APT framework indeed enhances model capability with fewer input data.

Acknowledgments

This work was partially supported by NSFC (62206056, 62322606). Xin Jiang and Carl Yang were not supported by any fund from China.

References

Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In ICLR , 2022.

Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. In EMNLP-IJCNLP , pages 3615-3620, 2019.

Zdzislaw Burda, Jarek Duda, Jean-Marc Luck, and Bartek Waclaw. Localization of the maximal entropy random walk. Physical review letters , 102(16):160602, 2009.

Hongyun Cai, Vincent W. Zheng, and Kevin Chen-Chuan Chang. Active learning for graph embedding. arXiv preprint arXiv:1705.05085 , 2017.

Shuvam Chakraborty, Burak Uzkent, Kumar Ayush, Kumar Tanmay, Evan Sheehan, and Stefano Ermon. Efficient conditional pre-training for transfer learning. In CVPR , pages 4241-4250, 2022.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory . John Wiley & Sons, 1999.

Ganqu Cui, Jie Zhou, Cheng Yang, and Zhiyuan Liu. Adaptive graph encoder for attributed graph embedding. In SIGKDD , pages 976-985, 2020.

Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge J. Belongie. Large scale finegrained categorization and domain-specific transfer learning. In CVPR , pages 4109-4118, 2018.

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. Using similarity measures to select pretraining data for ner. In Proceedings of NAACL-HLT , pages 1460-1470, 2019.

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. Cost-effective selection of pretraining data: A case study of pretraining bert on social media. In EMNLP (Findings) , pages 1675-1681, 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , page 4171-4186, 2019.

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML , pages 647-655, 2014.

Claire Donnat, Marinka Zitnik, David Hallac, and Jure Leskovec. Learning structural node embeddings via diffusion wavelets. In SIGKDD , pages 1320-1329, 2018.

David Easley and Jon Kleinberg. Networks, crowds, and markets: Reasoning about a highly connected world . Cambridge university press, 2010.

Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740 , 2021.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML , pages 1050-1059. PMLR, 2016.

Eleanor J. Gardiner, Peter Willett, and Peter J. Artymiuk. Graph-theoretic techniques for macromolecular docking. Journal of Chemical Information and Computer Sciences , 40(2):273279, 2000.

Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review , pages 1-77, 2023.

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , pages 580-587, 2014.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT press, 2016.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In SIGKDD , pages 855-864, 2016.

Hakim Hafidi, Mounir Ghogho, Philippe Ciblat, and Ananthram Swami. Graphcl: Contrastive self-supervised learning of graph representations. arXiv preprint arXiv:2007.08025 , 2020.

William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS , pages 1025-1035, 2017.

Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In ICML , pages 8230-8248, 2022.

Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In ICML , pages 4116-4126, 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , pages 9729-9738, 2020.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In SIGKDD , page 594-604, 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR , 2020.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In SIGKDD , pages 1857-1867, 2020.

Ziniu Hu, Changjun Fan, Ting Chen, Kai-Wei Chang, and Yizhou Sun. Pre-training graph neural networks for generic structural feature extraction. arXiv preprint arXiv:1905.13728 , 2019.

Hong Huang, Jie Tang, Sen Wu, Lu Liu, and Xiaoming Fu. Mining triadic closure patterns in social networks. In WWW , pages 499-504, 2014.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Thomas N Kipf et al. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning , 2016.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13):3521-3526, 2017.

Adit Krishnan, Mahashweta Das, Mangesh Bendre, Hao Yang, and Hari Sundaram. Transfer learning via contextual invariants for one-to-many cross-domain recommendation. In SIGIR , pages 1081-1090, 2020.

Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Finetuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054 , 2022.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics , 36(4):1234-1240, 2020.

Namkyeong Lee, Junseok Lee, and Chanyoung Park. Augmentation-free self-supervised learning on graphs. In AAAI , pages 7372-7380, 2022.

Pengyong Li, Jun Wang, Ziliang Li, Yixuan Qiao, Xianggen Liu, Fei Ma, Peng Gao, Seng Song, and Guotong Xie. Pairwise half-graph discrimination: A simple graph-level self-supervised strategy for pre-training graph neural networks. In IJCAI , pages 2694-2700, 2021.

Bin Liu, Yulin Zhu, and Ke Yan. Fold-LTR-TCP: protein fold recognition based on triadic closure principle. Briefings in Bioinformatics , 21(6):2185-2193, 2020.

Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. NeurIPS , 33:7498-7512, 2020.

Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343 , 2015.

Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi. Learning to pre-train graph neural networks. In AAAI , pages 4276-4284, 2021.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NeurIPS , pages 3111-3119, 2013.

Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv:1707.05005 , 2017.

Mark E. J. Newman. The structure and function of complex networks. SIAM Reviews , 45:167256, 2003.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In SIGKDD , pages 701-710, 2014.

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: Graph contrastive coding for graph neural network pre-training. In SIGKDD , pages 1150-1160, 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020.

Tiago Ramalho and Miguel Miranda. Density estimation in representation space to predict model uncertainty. In Engineering Dependable and Secure Machine Learning Systems: Third International Workshop, EDSMLS 2020, New York City, NY, USA, February 7, 2020, Revised Selected Papers 3 , pages 84-96. Springer, 2020.

Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In SIGKDD , pages 385-394, 2017.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR , pages 815-823, 2015.

Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In CVPR , pages 761-769, 2016.

Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, and Francesc Moreno-Noguer. Fracking deep convolutional image descriptors. arXiv preprint arXiv:1412.6537 , 2014.

Stavros Sintos and Panayiotis Tsaparas. Using strong triadic closure to characterize ties in social networks. In SIGKDD , pages 1466-1475, 2014.

Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-based hard example mining for deep metric learning. In CVPR , pages 7244-7252, 2019.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR , 2020.

Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. MoCL: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In SIGKDD , pages 3585-3594, 2021.

Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. NeurIPS , 32, 2019.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML , pages 6105-6114, 2019.

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Largescale information network embedding. In WWW , page 1067-1077, 2015.

Puja Trivedi, Ekdeep S Lubana, Mark Heimann, Danai Koutra, and Jayaraman Thiagarajan. Analyzing data-centric properties for graph contrastive learning. NeurIPS , 35:14030-14043, 2022.

Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In ICLR , 2019.

S. V. N. Vishwanathan, Nicol N. Schraudolph, Risi Kondor, and Karsten M. Borgwardt. Graph kernels. Journal of Machine Learning Research , 11:1201-1242, 2010.

Chun Wang, Shirui Pan, Guodong Long, Xingquan Zhu, and Jing Jiang. MGAE: Marginalized graph autoencoder for graph clustering. In CIKM , pages 889-898, 2017.

Mike Wu and Noah Goodman. A simple framework for uncertainty in contrastive learning. arXiv preprint arXiv:2010.02038 , 2020.

Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogcl: Information-aware graph contrastive learning. In NeurIPS , pages 30414-30425, 2021.

Jiarong Xu, Yang Yang, Junru Chen, Xin Jiang, Chunping Wang, Jiangang Lu, and Yizhou Sun. Unsupervised adversarially robust representation learning on graphs. In AAAI , pages 4290-4298, 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks. In ICLR , 2019.

Xi Yan, David Acuna, and Sanja Fidler. Neural data server: A large-scale search engine for transfer learning data. In CVPR , pages 3893-3902, 2020.

Cheng Yang, Deyu Bo, Jixi Liu, Yufei Peng, Boyu Chen, Haoran Dai, Ao Sun, Yue Yu, Yixin Xiao, Qi Zhang, et al. Data-centric graph learning: A survey. arXiv preprint arXiv:2310.04987 , 2023.

Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. IJCV , 113:113-127, 2015.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML , pages 12121-12132, 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS , pages 5812-5823, 2020.

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks. In ICML , pages 10871-10880, 2020.

Jiaqi Zeng and Pengtao Xie. Contrastive self-supervised learning for graph classification. In AAAI , pages 10824-10832, 2021.

Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, and Xia Hu. Data-centric ai: Perspectives and challenges. In SDM , pages 945-948. SIAM, 2023.

Jie Zhang, Yuxiao Dong, Yan Wang, Jie Tang, and Ming Ding. Prone: Fast and scalable network representation learning. In IJCAI , pages 4278-4284, 2019.

Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In AAAI , page 4438-4445, 2018.

Wentao Zhang, Yu Shen, Yang Li, Lei Chen, Zhi Yang, and Bin Cui. ALG: fast and accurate active learning framework for graph convolutional networks. In SIGMOD , pages 2366-2374, 2021.

Zaixin Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self-supervised learning for molecular property prediction. In NeurIPS , pages 15870-15882, 2021.

Jingbo Zhu, Huizhen Wang, Tianshun Yao, and Benjamin K. Tsou. Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In COLING , pages 1137-1144, 2008.

Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. An empirical study of graph contrastive learning. In NeurIPS D&B , 2021.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 , 2020.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW , pages 2069-2080, 2021.

A Theoretical Connection Between Network Entropy and Typical Graph Properties

Many interesting graph structural properties from basic graph theory give rise to a graph with high network entropy [19]. We here theoretically show some connections between the proposed network entropy and typical structural properties.

To make theoretical analysis, we consider connected, unweighted and undirected graph, whose network entropy depends solely on its degree distribution (see Eq. (3)). Considering a random graph G with a fixed node set, we suppose that the degree of any node v i independently follows distribution p , which is a common setting in random graph theory [8]. Then the expected network entropy of G is

where every d i (and d ) is an independent random variable follows the distribution p .

Now we are ready to discuss the connection between network entropy ⟨H ( G ) ⟩ and some typical graph properties ( i.e. , average degree ⟨ d ⟩ , degree variance Var( d ) and the scale-free exponent α ).

Average degree. Given that the function x log x is convex in x , we have

It is clear that average degree is the lower bound of network entropy. Based on our discussion on §3.1, we conclude that when used for pre-training, an input graph with higher average degree would in general result in better performance of the pre-trained model.

Degree variance. The Taylor expansion of ⟨ d log d ⟩ in Eq. (6) at ⟨ d ⟩ gives

where Var( d ) is the variance of d . We find that log ⟨ d ⟩ is exactly the zeroth-order term in the expansion. When average degree is fixed, the network entropy and the degree variance Var( d ) are positively correlated. This in turn implies a positive correlation between degree variance and the test performance of the model.

Scale-free exponent. Most real-world networks exhibit an interesting scale-free property ( i.e. , only a few nodes have high degrees), and thus the degree distribution often follows a power-law distribution. That is, we can just write the degree distribution as p ( x ) ∼ x -α , where α is called the scale-free exponent . For a real-world network, the scale-free exponent α is usually larger than 2 [4]. Suppose the degrees of a random graph G with N nodes follows a power-law distribution p ( x ) = Cx -α where C is a normalization constant. When α > 2 , we could approximately have [8]

Clearly, a smaller scale-free exponent α results in a higher network entropy.

Remark 1 (Connection between network entropy and typical structural properties) . A graph with high network entropy arises from graphs with typical structural characteristics like large average degree, large degree variance, and scale-free networks with low scale-free exponent.

Besides the above theoretical analysis. The motivation of choosing density, average degree, degree variance and scale-free exponent is similar to that of network entropy. Intuitively, graphs with larger average degree and higher density have more interactions among the nodes, thus providing more topological information to graph pre-training. Also, the larger the diversity of node degrees, the more diverse the subgraph samples. The diversity of node degrees can be measured by degree variance and scale-free exponent . (A smaller scale-free exponent indicates the length of the tail of degree distribution is relatively longer, i.e. , the degree distribution spreads out wider.)

B Theoretical Analysis and Advantages of Predictive Uncertainty

In this section, we provide the proofs of Theorem 1 and demonstrate the advantages of our proposed uncertainty.

Proofs of Theorem 1 Before we start the proof for Theorem 1, we introduce three useful lemmas that are adopted to prove the theorem as follows.

Lemma 1 (Fano's Inequality.) . Let X be a random variable uniformly distributed over a finite set of outcomes X . For any estimator ˆ X such that X → Y → ˆ X forms a Markov chain, we have

̸

Lemma 2 (Data-Processing Inequality.) . For any Markov chain X → Y → Z , we have

The proof of lemma 1 and lemma 2 can be found under Chapter 2 in [5]. The above lemmas establish a connection between mutual information and the accuracy of downstream task. Next, we further establish the connection between mutual information and ours uncertainty measure.

Lemma 3 (Connection between InfoNCE and mutual information) . Assume that graph encoder g is a scaling function, namely g ( x ) = k x . Hence, the mutual information I( X ; Z ) can be expressed by the InfoNCE loss ϕ uncertain for node in Eq.(1) as follows:

Proof: We follow [20] to prove the theorem. The mutual information I( X ; Z ) can be expressed as:

where f ( x ; y ) is any similarity scoring function like f ( x ; y ) = cos( x ) cos( y ) /τ .

Therefore, we can now prove that lower existing uncertainty L CE ( ζ ) over all downstream classifiers cannot be achieved without lower uncertainty ϕ uncertain ( ζ ) .

Theorem 1 (Theoretical connection between uncertainty.) . Let X , Z and Y be the input space, representation space and label set of downstream classifier. Assume that the distribution of labels is a uniform distribution over Y . Consider the set of downstream classifiers H = { h : Z → Y} . For any graph encoder f : X → Z , where L CE denotes the conventional uncertainty (which is the cross entropy loss here) estimated from the composition of graph encoder and downstream classifier h ◦ f , and ϕ uncertain is our proposed uncertainty estimated from graph encoder f .

Proof. For h ∈ H , we have the Markov chain where X,Y are random variables for input and label distributions respectively. The first Markov chain Y → X can be understood as a generative model for generating inputs according to the conditional probability distribution µ X | Y . Therefore, applying Lemmas 1 and 2, we obtain the inequality,

̸

̸

Pr[ h ( f ( X )) = Y ] represents the error rate. According to the definition of cross entropy, the cross entropy for node can be denoted as follows:

̸

By combining Eq. (13), Eq. (10) with Eq. (14), we have

̸

Thus, we completed the proof.

Advantages of our proposed uncertainty Except for the theoretical advantages, we further discuss two advantages of using the model loss ( i.e. , InfoNCE loss) as predictive uncertainty for data selection. First, InfoNCE loss is exactly the objective function of our model, so what we do is actually to select the samples with the greatest contributions to the objective function. Such strategy has been justified to accelerate convergence and enhance the discriminative power of the learned representations [18, 28, 29, 31]. Second, as the loss function of our model, InfoNCE is already computed during the training, and thus no additional computation expense is needed in the data selection phase.

C Algorithm

The overall algorithm for APT is given in Algorithm 1. Given a collection of graphs G = { G 1 , . . . , G N } from various domains, APT aims to pre-train a better generalist GNN ( i.e. , pretraining model) on wisely chosen graphs and samples. Our APT pipeline involves the following three steps. (i) At the beginning, the graph selector chooses a graph for pre-training according to the graph properties (line 1). (ii) Given the chosen graph, the graph selector chooses the subgraph samples whose predictive uncertainty is higher than T s in this graph (line 3). (iii) The selected samples are then fed into the model for pre-training until the predictive uncertainty of the chosen graph is below T g or the number of training iterations on this chosen graph reaches F (line 4-5). (iv) The model's feedback in turn helps select the most needed graph based on predictive uncertainty and graph properties until the predictive uncertainty of any candidate graph is low enough (line 6-7). The last three steps are repeated until the iteration number reaches a pre-set maximum value T (which can be considered as the total iteration number required to train on all selected graphs).

Algorithm 1 Overall algorithm for APT.

Input: A collection of graphs G = { G 1 , . . . , G N } , maximal period F of training one graph, trade-off parameter γ t = 0 , hyperparameter { β t } , the learning rate µ , the predictive uncertainty threshold of moving to a new graph T g , the predictive uncertainty threshold of choosing training samples T s , and the maximum iteration number T .

Output: Model parameter θ of the pre-trained graph model.

1: Choose a graph G ∗ from G via the graph selector, and G ← G\{ G ∗ } .

2: while The iteration number reaches T do

3: Sample instances with predictive uncertainty higher than T s from G ∗ via the graph selector.

4: Update model parameters θ ← θ -µ ∇ θ L ( θ ) .

5: if ϕ uncertain ( G ∗ ) < T g or the model has been trained on G ∗ by F iterations then

6: Update the trade-off parameter γ t ∼ Beta(1 , β t ) .

8: end if

7: Choose a graph G ∗ from G , and G ← G\{ G ∗ } .

9: end while

The time complexity of our model mainly consists of five components: data augmentation, GNN encoder propagation, contrastive loss, sample selection and graph selection. Suppose the maximal number of nodes of subgraph instances is | V | , the batch size is B , and D is the representation dimension. (1) As for the data augmentation, the time complexity of random walk with restart is at least O ( B | V | 3 ) [34]. (2) The time complexity of GNN encoder propagation depends on the architectures of the backbone GNN. We denote it as X here. (3) The time complexity of the contrastive loss is O ( B 2 D ) [17]. (4) Sample selection is conducted by choosing the samples with high contrastive loss (the loss is computed before), which costs O ( B ) . (5) Graph selection costs

O ( |G| M 2 D ) (where M the number of samples needed to compute the predictive uncertainty of a graph, and |G| is the number of graphs that have not been selected). This step is executed only in a few epochs (around 6% in our current model), so we ignore its time overhead in graph selection. Therefore, the overall time complexity of APT in each batch is O ( B | V | 3 + X + B 2 D + B ) .

D Dataset Details

The graph datasets for pre-training and testing in this paper are collected from a wide spectrum of domains (see Table 3 for an overview). The consideration of the graphs for pre-training and test is as follows. When selecting pre-training data, we hope that the graph size is at least hundreds of thousands to contain enough information for pre-training. When selecting test data, we hope that: (1) some test data is in the same domain as the pre-training data, and some is cross-domain, so as to comprehensively evaluate our model's in and across-domain transferability. Accordingly, the in-domain test data is selected from the type of movie and citations, and the others test data are across-domain; (2) the size of test graphs can scale from hundreds to millions, including large-scale datasets with millions of edges from Open Graph Benchmark [10].

Regarding the pre-training datasets, arxiv, dblp and patents-main are citation networks collected from [2], [37] and [9], respectively. Imdb is the collection of movie from [27]. As for the social networks, soc-sign0902 and soc-sign0811 are collected from [16], wiki-vote is from [15], academia is from [7], and michigan, msu and uillions are from [32]. Regarding the test datasets, we collect the protein network dd and ogbproteins from [6] and [10]. The image network msrc-21 is from [23]. The movie network imdb-binary is from [36]. The citation networks, cora, pubmed and ogbarxiv, are from [21], [22] and [10]. The web networks cornell and wisconsin are collected from [24]. The transportation network brazil is form [26], and dd242, dd68 and dd687 are from [27].

The detailed graph properties of the pre-training data and test data are presented in Table 4 and Table 5, respectively.

Dataset

Properties

|

V

|

|

E

|

avg properties avg degree degree var

density

entropy

α

2

(avg, std)

avg clustering coef

soc-sign0902

81867

497672

-0.32

13.16

1643.20

1.49e-04

3.91

1.51 1192.33, 2305.99

0.06

soc-sign0811

77350

468554

-0.32

13.12

1631.77

1.57e-04

3.93

1.52 1226.93, 2312.03

0.05

imdb

896305 3782447

-0.66

9.44

298.27

9.42e-06

3.07

1.53

316.16, 614.03

5e-05

patent

240547

560943

-0.96

5.66

34.95

1.94e-05

2.04

1.57

117.23, 172.40

0.08

academia

137969

369692

-0.89

6.36

102.14

3.88e-05

2.38

1.57

101.40, 225.96

0.14

wiki-Vote

7115

103689

0.74

29.32

3314.79

4.10e-03

4.46

1.40

972.03, 1045.13

0.14

dblp

93156

178145

-1.12

4.82

58.05

4.11e-05

2.16

1.72

58.85, 90.46

0.27

arxiv

86376

517563

-0.43

12.98

382.12

1.39e-04

3.22

1.41

145.21, 309.12

0.68

michigan

30147

1176516

1.40

79.05

6369.17

2.59e-03

4.78

1.23 4683.90, 3655.25

0.21

msu

32375

1118774

1.13

70.11

5087.53

2.13e-03

4.62

1.23 4567.63, 3465.16

0.21

uillinois

30809

1264428

1.44

83.08

6306.02

2.66e-03

4.78

1.22 5267.10, 3831.37

0.21

E Additional Observations of Curse of Big Data Phenomenon

This section provides more comprehensive observations to support the curse of big data phenomenon in graph pre-training, i.e. , more training samples and graph datasets do not necessarily lead to better downstream performance.

We investigate 3630 experiments with GCC [25] and GraphCL [39] model with different model configurations ( i.e. , the number of GNN layers is set to be 3, 4 and 5 respectively), when pre-trained on all training graphs listed in Table 3 and evaluated on different test graphs (annotated in the upper left corner of each figure) under freezing setting. For each experiment, we calculate the mean and standard deviation over 10 evaluation results of the downstream task with random training/testing splits.

The observations of GCC and GraphCL model can be found in Figure 6 and Figure 7 respectively. The downstream results of different test data are presented in separate rows. The figures in left three columns present the effect of scaling up the number of graphs on the downstream performance under different model configurations ( i.e. , the number of GNN layers) respectively. We first pre-train the model with only two input graphs, and the result is plotted in a dotted line. The largest standard deviation among the results w.r.t different graph last is also marked by the blue arrow. The figures in the last column illustrate the effect of scaling up sample size (log scale) on the performance.

Curse of big data phenomenon in molecular pre-training. We also conduct observations of scaling up the number of graphs on the downstream performance in molecular pre-training. We take AttrMasking [13], ContextPred [13], and GraphCL [39] as backbone models. Following their settings, the pre-training data is sampled from the molecular dataset ZINC15 and we take the molecular dataset BBBP as downstream data. We directly adopt the default settings of these pre-train models except for the number of molecules for pre-training. The number of molecular graphs are selected as {1,000,

5,000, 10,000, 50,000, 100,000}. The results are shown in Figure 5, which suggest that 'curse of big data' phenomenon is still found even when pre-training and downstream data are all molecules.

10

3

10

4

10

5

# of graphs

66

68

70

72

74

micro F1

(a) AttrMasking

10

3

10

4

10

5

# of graphs

66

68

70

72

74

micro F1

(b) ContextPred

10

3

10

4

10

5

# of graphs

66

68

70

72

74

micro F1

(c) GraphCL

The explanation of convex hull fit. In order to better show the changing trend, the blue curve in the last column in Figure 6 and Figure 7 is fitted to the convex hull of the points. The convex hull is proposed to capture the performance of a randomized classifier made by choosing pre-training models with different probabilities [1].

We first introduce the concept of randomized classifier . Given two classifiers with training sample size and downstream performance c 1 = ( c sz 1 , c ds 1 ) and c 2 = ( c sz 2 , c ds 2 ) , a randomized classifier can be made to choose the first classifier with probability p and the second classifier with probability 1 -p . Then the output of the randomized classifier is pc 1 +(1 -p ) c 2 , which is the convex combination of c 1 and c 2 . All the points on this convex combination can be obtained by choosing different p . Extend the notion to the case of multiple classifiers, we can consider the output of such a randomized classifier to be a convex combination of the outputs of its endpoints [1]. All the points on the convex hull are achievable. Therefore, the output of the randomized classifier is equivalent to the convex hull of our trained classifiers' performance.

In our experiments, we include the upper hull of the convex hull of the model performances, i.e. , the highest downstream performance for every given sample size. Such convex hull fit is proved to be robust to the density of the points in each figure [1].

A final remark is that our observations on different downstream datasets do not result in a onemodel-fits-all trend. So we propose to fit a complicated curve whose function has form f ( x ) = a 1 ln x/x a 2 + a 3 ( a 1 , a 2 , a 3 > 0) to the best performing models ( i.e. , the convex hull fit as discussed above). The fitted parameters a 1 , a 2 and a 3 in this function of each curve are given in Table 6.

dd687

dd687

dd687

micro F1

micro F1

micro F1

# of samples

micro F1

dd687

# of graphs

# of graphs

# of graphs

wisconsin

wisconsin

brazil

brazil

# of graphs

# of samples

# of samples

micro F1

micro F1

micro F1

micro F1

std : 3.1

std :1.5

micro F1

# of graphs

# of graphs

# of graphs

# of graphs

# of graphs

wisconsin

wisconsin

micro F1

micr oF1

micro F1

brazil

brazil

model config

.

5 layers

4 layers

3 layers

std : 1.1

std :2.1

std : 0.8

std :5.0

std : 4.1

std : 1.1

std : 1.3

# of graphs

# of samples

# of samples

micro F1

micro F1

micro F1

dd68

dd68

# of graphs

# of graphs

# of graphs

micro F1

micro F1

# of graphs

micro F1

micro F1

micro F1

dd68

dd68

# of graphs

micro F1

micro F1

micro F1

micro F1

# of graphs

# of graphs

# of graphs

# of samples

pubmed

pubmed

pubmed

pubmed

# of samples

# of graphs

# of graphs

# of graphs

micro F1

micro F1

micro F1

micro F1

cornell

cornell

cornell

cornell

cora

cora

micro F1

micro F1

micro F1

micro F1

# of graphs

# of graphs

# of graphs

# of samples

cora

cora

std : 1.8

std : 1.0

std : 0.6

std : 0.3

std : 0.7

std : 2.0

std : 0.8

std : 0.5

std : 0.5

std : 0.6

std : 0.8

std : 0

.

3

std : 0.5

std : 0.5

std : 1

.

3

# of graphs

# of samples

micro F1

micro F1

dd

dd

# of graphs

micro F1

micro F1

# of graphs

dd

dd

msrc-21

imdb-binary

# of samples

# of samples

micro F1

micro F1

msrc-21

std : 1.4

# of graphs

# of graphs

# of graphs

# of graphs

# of graphs

# of graphs

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

msrc-21

msrc-21

imdb-binary

std :1.4

imdb-binary

imdb-binary

std : 1.4

std : 0.6

std :1.4

std : 1.2

std : 0.7

std : 1.3

std : 0.5

GCC (5 layers)

GCC (5 layers)

GCC (4 layers)

GCC (3 layers)

GCC (5 layers)

GCC (4 layers)

GCC (3 layers)

GCC (4 layers)

GCC (3 layers)

GCC (5 layers)

GCC (5 layers)

GCC (5 layers)

GCC (5 layers)

GCC (5 layers)

GCC (5 layers)

GCC (5 layers)

GCC (5 layers)

GCC (4 layers)

GCC (4 layers)

GCC (4 layers)

GCC (4 layers)

GCC (4 layers)

GCC (4 layers)

GCC (4 layers)

GCC (4 layers)

GCC (3 layers)

GCC (3 layers)

GCC (3 layers)

GCC (3 layers)

GCC (3 layers)

GCC (3 layers)

GCC (3 layers)

GCC (3 layers)

(b)  Graph classification

(a)  Node classification

micro F1

micro F1

# of samples

micro F1

# of graphs

# of graphs

# of graphs

# of graphs

# of samples

# of samples

micro F1

micro F1

micro F1

micro F1

micro F1

# of graphs

# of graphs

# of graphs

# of graphs

# of graphs

wisconsin

model config

.

5 layers

4 layers

3 layers

std :3.0

# of graphs

# of samples

# of samples

micro F1

micro F1

micro F1

# of graphs

# of graphs

# of graphs

# of graphs

micro F1

micro F1

dd68

# of graphs

micro F1

micro F1

# of graphs

# of graphs

# of graphs

# of samples

pubmed

# of graphs

# of graphs

# of graphs

micro F1

micro F1

cornell

micro F1

micro F1

micro F1

# of graphs

# of graphs

# of graphs

# of samples

std : 2.5

std : 1.0

std : 1.0

std : 0.3

# of graphs

# of samples

micro F1

# of graphs

micro F1

micro F1

# of graphs

imdb-binary

# of samples

# of samples

micro F1

micro F1

# of graphs

# of graphs

# of graphs

# of graphs

# of graphs

# of graphs

micro F1

micro F1

micro F1

micro F1

std : 1.4

std : 1.4

msrc-21

dd

# of samples

cornell

pubmed

cora

dd687

dd68

wisconsin

brazil

cora

std : 0.8

GraphCL (5 layers)

brazil

std :3.3

dd687

std : 0.8

GraphCL (5 layers)

GraphCL (5 layers)

GraphCL (5 layers)

GraphCL (5 layers)

GraphCL (5 layers)

GraphCL (5 layers)

GraphCL (5 layers)

imdb-binary

GraphCL (5 layers)

GraphCL (5 layers)

msrc-21

std : 1.4

dd

GraphCL (5 layers)

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

micro F1

(b)

Graph classification

(a)

Node classification

imdb-binary

GraphC

L (4 layers)

msrc-21

GraphC

L (4 layers)

micro F1

pubmed

GraphCL (4 layers)

wisconsin

GraphCL (4 layers)

dd687

dd

brazil

cora

cornell

dd68

GraphCL (4 layers)

GraphCL (4 layers)

GraphCL (4 layers)

GraphCL (4 layers)

GraphCL (4 layers)

GraphCL (4 layers)

GraphCL (4 layers)

std : 2.0

std : 5.7

std : 0.7

std : 1.1

std : 0.6

std : 0.7

std : 4.7

std : 1.4

std : 1.7

std : 1.0

std : 1.0

micro F1

micro F1

micro F1

pubmed

msrc-21

imdb-binary

dd

cornell

cora

dd687

dd68

wisconsin

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

GraphCL (3 layers)

brazil

GraphCL (3 layers)

std : 3.2

std : 3.1

std : 0.8

std : 1.2

std : 1.3

std : 1.0

std : 1.7

std : 5.1

std : 6.1

std : 3.4

std : 0.7

F Empirical Study of Graph Properties

Additional properties for part (b) in Figure 2. In Figure 8, we plot the Pearson correlation between the graph properties of the graph used in pre-training (shown in the y -axis) and the performance of the pre-trained model using this graph on different unseen test datasets (shown in the x -axis). Note that the pre-training is performed on each of the input training graphs (in Table 3) via GCC. The results indicate that network entropy, density, average degree and degree variance exhibit a clear positive correlation with the performance, while the scale-free exponent presents an obviously negative relation with the performance. On the contrary, some other properties of graphs, including clique number, transitivity, degree assortativity and average clustering coefficient, do not seem to have connections with downstream performance, and also exhibit little or no correlation with the performance. Therefore, the favorable properties of network entropy, density, average degree, degree variance and the scale-free exponent of a real graph are able to characterize the contribution of a graph to pre-training.

-

0.6

-

0.4

-

0.2

0.0

0.2

0.4

0.6

0.8

no

correlation

positive

correlation

negative

correlation

cora

pubmed

brazil

dd68

wisconsin

cornell

Density

Average degree

Degree variance

Clique number

Transivity

Degree assortativity

Clustering coefficient

Scale-free exponent

Entropy

Detailed illustrations of Figure 3. In Figure 3, the illustrative graphs are generated by the configuration model with 15-18 nodes. The shaded area groups the illustrative graphs whose network entropy and graph properties are similar. Each four points on the same horizontal coordinate represent four graph properties of an illustrating graph. Each curve is fitted by least squares and represents the relation between entropy and other graph properties.

Additional real-world example for Figure 3. In Figure 9, we provide a real-world example of how network entropy correlates with four typical structural properties (in red), as well as the performance of the pre-trained model on test graphs (in blue). Numerical experiments again support our explanation (or intuition) of their strong correlation.

1e-3

2e-3

3e-3

1

2

3

4

5

46

48

50

52

54

56

Network entropy

Performance (%)

(a) Density

20

40

60

1

2

3

4

5

46

48

50

52

54

56

Network entropy

Performance (%)

(b) Average degree

20

40

60

1

2

3

4

5

46

48

50

52

54

56

Network entropy

Performance (%)

(c) Degree variance

1.35

1.50

1.65

1

2

3

4

5

46

48

50

52

54

56

Network entropy

Performance (%)

(d) Scale-free exponent

G Implementation Details

The number reported in all the experiments are the mean and standard deviation over 10 evaluation results of the downstream task with random training/testing splits. When conducting the downstream task, For each dataset, we consistently use 90% of the data as the training set, and 10% as the testing set. We conduct all experiments on a single machine of Linux system with an Intel Xeon Gold 5118 (128G memory) and a GeForce GTX Tesla P4 (8GB memory).

Implementations of our model. The regularization for weights of the model in Eq. (5) is applied to first 2 layers of GIN. The maximal period of training one graph F is 6, the maximum iteration number T is 100, and the predictive uncertainty thresholds T s and T g are set to be 3 and 2 respectively. The selected instances are sampled from 20,000 instances each epoch. Since the pre-training model is unable to provide precise predictive uncertainty in the initial training stage, the model is warmed up over the first 20 iterations. Since we adopt GCC as the backbone pre-training model, the other settings are the same as GCC.

Our model is implemented under the following software settings: Pytorch version 1.4.0+cu100, CUDA version 10.0, networkx version 2.3, DGL version 0.4.3post2, sklearn version 0.20.3, numpy version 1.19.4, Python version 3.7.1.

Implementations of baselines. We compare against several graph representation learning methods. For implementation, we directly adopt their public source codes and most of their default hyperparameters. The key parameter settings and code links can be found in Table 7.

H Additional Experimental Results and Analysis

Analysis of ablation studies of APT. Table 1 presents the node classification results of APT and its 4 variants, i.e. , APT-G, APT-P, APT-R, and APT-L2. We here analyze the performance of these ablation studies from the following three points of view: (1) APT-L2 and APT, the variants with the proximity regularization w.r.t old knowledge, perform the best in most cases. This suggests that catastrophic forgetting of previous trained graphs could occur during pre-training, and it is necessary to add the proximity regularization to prevent this; (2) Sometimes APT-R (the variant without proximity term) achieves the best performance on the test data dd687 and cornell. One potential reason is that the last data to be selected for pre-training is dblp, while the test data dd687 and cornell exhibit some properties similar to dblp. Therefore, even excluding the proximity term, the pre-training model can still remember knowledge from the last pre-trained data, and achieve good performance on the test data that is similar to the last pre-trained data; (3) APT-G and APT-P present suboptimal performance among all variants, which suggests the utility of graph properties and predictive uncertainty.

Impact of five graph properties combination. We study the effect of the strategy of utilizing only one graph property in Table 8. We find that the five properties used in our model are all indispensable, and the most important one probably varies for different datasets. That's why we choose to combine all graph properties.

Moreover, these case studies may provide some clues of how to select pre-training graphs when some knowledge of the downstream tasks is known. For example, if the downstream dataset is extremely dense (like imdb-binary), the density property dominates among the selection criteria (such that the probability of encountering very dense out-of-distribution samples during testing can be reduced). If the entropy of downstream dataset is very high (like brazil), it is perhaps better to choose graphs with high entropy for pre-training. But still, when the downstream task is unknown, using the combination of five metrics often leads to the most satisfactory and robust results.

node classification

graph classification

Method

Dataset

brazil

dd68

dd687

wisconsin

cornell

cora

pubmed

imdb-binary

dd

msrc-21

Entropy

80.04(2.15)

25.79(0.94) 16.31(0.81) 11.08(0.82) 67.01(2.00) 52.80(2.46) 45.41(0.85) 50.85(0.19)

76.78(0.84)

75.56(0.84) 24.34(1.50)

Density

79.23(1.92)

27.29(0.62

)

19.89(0.95)

12.22(1.06) 65.58(1.87) 51.15(1.59) 46.18(0.71) 50.74(0.15)

77.20(0.66)

75.29(0.54) 24.20(1.31)

Average degree

79.22(1.65) 24.99(0.67) 16.56(1.01) 11.67(1.18) 67.02(1.86) 51.43(4.16) 46.38(0.48) 50.99(0.31)

76.87(0.93)

75.46(0.53) 25.14(1.54)

Degree variance

78.44(2.24) 24.94(0.61) 16.62(1.04) 11.51(1.17) 65.65(1.28) 50.45(2.14) 45.76(0.65) 50.70(0.21)

76.39(1.04)

75.47(0.67) 25.22(1.51)

Scale-free exponent

79.70(2.71) 24.94(0.68) 17.26(0.63) 12.03(1.41) 64.77(2.31) 51.37(2.70) 45.18(0.52) 50.84(0.26)

75.24(0.62)

75.52(1.24) 23.19(1.39)

Our combination

78.75(1.63) 24.62(0.90) 17.83(1.35)

12.26(0.78) 67.04(1.50) 52.94(1.95) 47.48(0.46) 51.25(0.21)

75.93(0.84)

75.58(1.06) 25.58(1.57)

Analysis of the selected graphs. The data sequentially selected via our graph selector are uillinois, soc-sign0811, msu, michigan, wiki-vote, soc-sign0902 and dblp. To further analyze why these graphs are chosen, we present their detailed structural properties in Table 4 in the Appendix D. We first observe that uillinois, michigan and msu have the largest value of the term related to graph properties ( i.e. , MEAN ( ˆ ϕ entropy , ˆ ϕ density , ˆ ϕ avg_deg , ˆ ϕ deg_var , -ˆ ϕ α ) ), while dblp has the smallest. This indicates that the selection of data is influenced by a combination of factors, including graph properties and potentially other factors such as predictive uncertainty. Moreover, it is also interesting to see that the selected graph wiki-vote is the smallest graph among all the pre-training graphs, but it still contributes to the performance. This observation again verifies the curse of big data phenomenon in graph pre-training, which suggests that having more training samples does not necessarily result in improved downstream performance.

Results of APT under different backbone models. We here include GraphCL [39] and JOAO [38] as our backbone models. Table 9 shows our superiority under most cases. The datasets used for pre-training and testing are the same as those employed in the experiments where GCC was the backbone model (refer to Table 7). Since the downstream tasks of GraphCL and JOAO are limited to graph classification, we can directly adapt them to our setting with graph classification as downstream task. For the node classification task, we simply take the RWR subgraphs as the input of GraphCL and JOAO, and treat the learned subgraph representation as node representation. We directly adopt GraphCL and JOAO with their default hyper-parameters. The backbone GNN model is GIN with 5 layers.

Results of APT under molecular pre-training. We also evaluate APT under molecular pretraining setting. We adopt the state-of-art model, Mole-BERT [35] and a commonly used strategy, EdgePred [12] as backbone molecular pre-training models.

The predictive uncertainty mentioned before is based on contrastive loss, however, the loss function of Mole-BERT and EdgePred are different from that used in GCC. To adapt these two models, we need to make corresponding changes to our predictive uncertainty. More specifically, for Mole-BERT, we take triplet masked contrastive learning L TMCL , which is the combination of triplet loss L tri with commonly-used contrastive loss L con in the original paper, as the predictive uncertainty. For EdgePred, we take the loss function, i.e. , negative log likelihood in the original paper as the predictive uncertainty

Besides, we adjust the data selection process in molecular pre-training. This is because each molecular graph is taken as a sample in molecular pre-training, so we can directly select graphs. The overall algorithm for APT under molecular setting is given in Algorithm 2.

Algorithm 2 Overall algorithm for APT (Mole).

Input: A collection of graphs G = { G 1 , . . . , G N } , the number of graphs in the pre-training dataset |G 0 | , the maximal period F of training a set of graphs, trade-off parameter γ t = 0 , ratio of the number of selected graphs each time to the number of graphs in the pre-training dataset η , hyperparameter { β t } , the learning rate µ , the predictive uncertainty threshold of moving to a new set of graphs T g , and the maximum iteration number T .

Output: Model parameter θ of the pre-trained graph model.

1: Choose a set of graphs G ∗ ( |G ∗ | = η ∗ |G 0 | ) from G via the graph selector, and G ← G\{G ∗ } .

2: while The iteration number reaches T do

3: Update model parameters θ ← θ -µ ∇ θ L ( θ ) .

5: Update the trade-off parameter γ t ∼ Beta(1 , β t ) .

4: if ϕ uncertain ( G ∗ ) < T g or the model has been trained on G ∗ by F iterations then

6: Choose a set of graphs G ∗ ( |G ∗ | = η ∗ |G 0 | ) from G , and G ← G\{G ∗ } .

7: end if

8: end while

For pre-training datsets, we use 2 million molecules sampled from the ZINC15 database following [12, 35]. For downstream datasets, we adopt the widely-used 8 binary classification datasets contained in MoleculeNet [33] following [12,35]. For implementation, T g is set as -2.5 for Mole-BERT and 0.5 for EdgePred. For both Mole-BERT and EdgePred, the ratio of the number of selected graphs each time to the number of the graphs in original pre-training dataset η is 0.2. Other parameters related to APT are set the same as those in APT with GCC as pre-training backbone model. The other parameters related to molecular pre-training are the same as those employed in Mole-BERT and EdgePred, respectively.

The results of APT under molecular pre-training with Mole-BERT and EdgePred as backbone models are reported in Table 10. We can see that our APT adaptions show significant superiority compared to the backbones. Moreover, APT (Mole-BERT) achieves the best performance on 6 out of 8 downstream datasets among all the baseline models, which shows the effectiveness of our pipeline.

Training time. As empirically noted in Table 11, the total training time of APT-L2 and APT is 18321.39 seconds and 18592.01 seconds respectively (including the time consumed in graph selection

and regularization term), while the competitive graph pre-training model GCC takes 40161.68 seconds for the same number of training epochs on the same datasets.

The time spent on the inference on all graphs during graph selection (which is the main time spent for graph selection) only accounts for 3.95% and 3.87% of the total time under APT-L2 and APT respectively. Note that this step is executed only in a few epochs (around 6% in our current model).

The time cost of the L2 regularization term only accounts for 0.08% of the total time and the EWC regularization term only accounts for 0.45% of the total time, which is calculated by the runtime gap between the models with and without the regularization term. Note that the regularization term is imposed on the first two layers of the GNN encoder, which only accounts for 12.4% of the total number of parameters.

The efficiency of our model is due to a much smaller number of carefully selected training graphs and samples at each epoch. In addition, the number of parameters in our model is 190,544, which is the same order of magnitude as classical GNNs like GraphSAGE, GraphSAINT, etc . and is relatively small among models in open graph benchmark [11].

Effects of hyper-parameters. Here we show that the effect of the hyper-parameter { λ } in the proximity term, the maximal number of training iterations on one graph F , the predictive uncertainty threshold of moving to a new graph T g , and the predictive uncertainty threshold of choosing training samples T s on performance in Figure 10 and Figure 11.

The hyper-parameter λ is the trade-off parameters between the knowledge learnt from new data and that from previous data in Eq. (5). We use the dataset dd242 as an example to find the suitable values of the hyper-parameter under the L2 and EWC regularization setting respectively, and present here for reference (see Figure 10). Clearly, a too small or too large λ would deteriorate the performance. Thus, an appropriate value of λ is preferred to ensure that the graph pre-training model can learn from new data as well as remember previous knowledge. We leave changing λ as the future work.

1

10

50

100

16.5

16.6

16.7

16.8

micro F1

(a) Ablation study on APT-

L2 (freeze)

100

300

500

700

16.3

16.4

16.5

micro F1

(b) Ablation study on APT

(freeze)

Our model training involves three hyper-parameter F, T g , T s , where F controls the largest number of epochs training on each graph, T g is the predictive uncertainty threshold of moving to a new graph, T s is the predictive uncertainty threshold of choosing training samples. We use grid search to show F ∈ { 4 , 5 , 6 } 's, T g ∈ { 3 , 3 . 5 , 4 } 's and T s ∈ { 1 , 2 , 3 } 's role in the pre-training. F remains at 5 while studying T g and T s , T g remains at 3.5 while studying F and T s , and T s remains at 2 while studying F and T g . Figure 11 presents the effect of these parameters, We find that if the value of F is set too small or that of T g is too large, the model cannot learn sufficient knowledge from each graph, leading to suboptimal results. Too large F or small T g also lead to poor performance. This indicates that instead of training on a graph for a large period, it would be better to switch to training on various graphs in different domains to gain diverse and comprehensive knowledge. Regarding the hyper-parameter T s , we observe that large T s would make the model having too few training samples to learn knowledge, and small T s could not select the most uncertain and representative samples, thus both cases achieve suboptimal performance.

5

6

7

15.0

16.0

17.0

micro F1

(a)

Ablation study on

F

1

2

3

15.0

16.0

17.0

micro F1

(b)

Ablation study on

T

g

1

2

3

15.0

16.0

17.0

micro F1

(c)

Ablation study on

T

s

The choice of β t , its alternatives, and ablation study. At the beginning of the pre-training, the model is less accurate and needs more guidance from graph properties. We therefore set γ t as larger at the beginning and gradually decrease it. To simplify this process, we follow [3] to use the exponential formula of β t = c 1 -c t 2 to set the expectation of γ t to be strictly decreasing (where γ t ∼ Beta(1 , β t ) ).

We then illustrate that the choice of the decay function of β t is robust. Table 12 and Table 13 below show the effect of linear decay, step decay and exponential decay on β t . (The function for linear decay and step decay are designed as β t = 2 . 001 + 0 . 004 t , β t = 2 . 005 + floor( t/ 20) , respectively. The initial value β 1 is set the same as ours.) While there is no universally better decay function, the performance of our method is not significantly impacted by the choice of different decay functions, and our performance is better than the baselines in most cases regardless of the choices of specific decay functions.

The parameters c 1 and c 2 in the exponential formula of β t = c 1 -c t 2 are suggested as 1.005 and 0.995 in [3]. We simply perform grid search on c 1 in { 1 . 005 , 3 , 5 } ; see the effect of c 1 in the Figure 12.

1.005

3

5

53

54

55

micro F1

(a) Wisconsin on node clas-

sification

1.005

3

5

72

73

74

micro F1

(b) Imdb-binary on graph

classification

Results under GCC's original experimental setting. For a more comprehensive comparison, we also adopt the same pretraining datasets and downstream datasets as GCC. Table 14 and table 15 shows the performance of our model and the strongest competitor GCC, under the experiment setting of GCC in node classification and graph classification task, repectively. (The performance of GCC is directly taken from its paper.) The results indicate that our model still outperforms GCC under the experimental setting of GCC. The results still suggest ours superiority in most cases.

The justification of input graphs' learning order. Table 16 reveals the downstream performance can be affected by the learning order of input training graphs. With the guidance of graph selector, the pre-training model is encouraged to first learn the graphs and samples with higher predictive uncertainty and graph properties. Such learning order accomplishes better downstream performance compared to the reverse or random one.

The choice of the 'difficult' data. Among all the data, 'difficult' samples contribute the most to the loss function, and thus they can yield gradients with large magnitude. Comparatively, training with easy samples may suffer from inefficiency and poor performance as these data points produce gradients with magnitudes close to zero [14, 30]. In addition, learning from difficult samples has proven to be able to accelerate convergence and enhance the expressive power of the learnt representations [28, 31]. For our model, the importance of learning from difficult samples is also justified empirically, as shown in Table 17.

Time comparison: pre-training vs. training from scratch. Using a pre-trained model can significantly reduce the time required for training from scratch. The reason is that the weights of the pre-trained model have already been put close to appropriate and reasonable values; thus the model converges faster during fine-tuning on a test data. As shown in Figure 13, compared to regular GNN model ( e.g. GIN), our model yields a speedup of 4.7× on average (which is measured by the ratio of the training time of GIN to the fine-tuning time of APT). Based on above analysis, we can draw a conclusion that pre-training is beneficial both in effectiveness and efficiency.

msrc-21

dd

imdb-binary

running time (sec)

APT

GIN

200

400

600

5.2

3.9

4.9

×

×

×

<latexi sh

1_b

64="

OT

KgBv

/Y3

pP

W7uqU

+A

>

X

c

V

NS8

EJ

r

9L

0

Q

F

2

o

z

D

Hm

G

Z

j

w

y

I

C

n

f

M

d

5

R

k

<latexi sh

1_b

64="

OT

KgBv

/Y3

pP

W7uqU

+A

>

X

c

V

NS8

EJ

r

9L

0

Q

F

2

o

z

D

Hm

G

Z

j

w

y

I

C

n

f

M

d

5

R

k

<latexi sh

1_b

64="

OT

KgBv

/Y3

pP

W7uqU

+A

>

X

c

V

NS8

EJ

r

9L

0

Q

F

2

o

z

D

Hm

G

Z

j

w

y

I

C

n

f

M

d

5

R

k

References

Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In ICLR , 2022.

Francesco Bonchi, Pooya Esfandiar, David F Gleich, Chen Greif, and Laks VS Lakshmanan. Fast matrix computations for pairwise and columnwise commute times and katz scores. Internet Mathematics , 8(1-2):73-112, 2012.

Hongyun Cai, Vincent W. Zheng, and Kevin Chen-Chuan Chang. Active learning for graph embedding. arXiv preprint arXiv:1705.05085 , 2017.

Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. Power-law distributions in empirical data. SIAM review , 51(4):661-703, 2009.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory . John Wiley & Sons, 1999.

Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology , 330(4):771-783, 2003.

M. Fire, L. Tenenboim, O. Lesser, R. Puzis, L. Rokach, and Y. Elovici. Link prediction in social networks using computationally efficient topological features. In IEEE third international conference on social computing , pages 73-80, 2011.

Jesús Gómez-Gardenes and Vito Latora. Entropy rate of diffusion processes on complex networks. Physical Review E , 78(6):065102, 2008.

Bronwyn H Hall, Adam B Jaffe, and Manuel Trajtenberg. The nber patent citation data file: Lessons, insights and methodological tools, 2001.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. NeurIPS , 33:22118-22133, 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR , 2020.

Ziniu Hu, Changjun Fan, Ting Chen, Kai-Wei Chang, and Yizhou Sun. Pre-training graph neural networks for generic structural feature extraction. arXiv preprint arXiv:1905.13728 , 2019.

Chen Huang, Chen Change Loy, and Xiaoou Tang. Local similarity-aware deep feature embedding. In NeurIPS , page 1262-1270, 2016.

Jure Leskovec, Daniel Huttenlocher, and Jon Kleinberg. Predicting positive and negative links in online social networks. In WWW , pages 641-650, 2010.

Jure Leskovec, Kevin J Lang, Anirban Dasgupta, and Michael W Mahoney. Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters. Internet Mathematics , 6(1):29-123, 2009.

Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML , pages 13052-13065, 2022.

Ilya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint arXiv:1511.06343 , 2015.

Christopher W Lynn, Lia Papadopoulos, Ari E Kahn, and Danielle S Bassett. Human information processing in complex networks. Nature Physics , 16(9):965-973, 2020.

Martin Q Ma, Yao-Hung Hubert Tsai, Paul Pu Liang, Han Zhao, Kun Zhang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Conditional contrastive learning for improving fairness in self-supervised learning. arXiv preprint arXiv:2106.02866 , 2021.

Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval , 2(3):127-163, 2000.

Galileo Namata, Ben London, Lise Getoor, and Bert Huang. Query-driven active surveying for collective classification. In MLG , page 1, 2012.

Marion Neumann, Roman Garnett, Christian Bauckhage, and Kristian Kersting. Propagation kernels: efficient graph kernels from propagated information. Machine Learning , 102(2):209245, 2016.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In ICLR , 2019.

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: Graph contrastive coding for graph neural network pre-training. In SIGKDD , pages 1150-1160, 2020.

Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In SIGKDD , pages 385-394, 2017.

Ryan A. Rossi and Nesreen K. Ahmed. The network data repository with interactive graph analytics and visualization. In AAAI , page 4292-4293, 2015.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR , pages 815-823, 2015.

Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, and Francesc Moreno-Noguer. Fracking deep convolutional image descriptors. arXiv preprint arXiv:1412.6537 , 2014.

Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS , page 1849-1857, 2016.

Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-based hard example mining for deep metric learning. In CVPR , pages 7244-7252, 2019.

Amanda L Traud, Peter J Mucha, and Mason A Porter. Social structure of Facebook networks. Physica A: Statistical Mechanics and its Applications , 391(16):4165-4180, 2012.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science , 9(2):513-530, 2018.

Feng Xia, Jiaying Liu, Hansong Nie, Yonghao Fu, Liangtian Wan, and Xiangjie Kong. Random walks: A review of algorithms and applications. IEEE Transactions on Emerging Topics in Computational Intelligence , 4(2):95-107, 2019.

Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In ICLR , 2023.

Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In SIGKDD , pages 1365-1374, 2015.

Jaewon Yang and Jure Leskovec. Defining and evaluating network communities based on ground-truth. In SIGKDD , pages 1-8, 2012.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML , pages 12121-12132, 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS , pages 5812-5823, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work found that more data does not necessarily improve the pre-training of GNNs and then propose a method to adaptively select data for pre-training GNNs. Experimental results show the proposed method is effective.\n\nSTRENGTHS:\n1. this work is well motivated by the observation that more data does not necessarily lead to better pretrained graph encoder\n\n2. that the experiments seems to be convincing since APT generally outperforms other baselines\n\nWEAKNESSES:\nThere are several weaknesses mainly about the method:\n\n1. The optimization problem in equation 5 is confusing, what is the variable you are optimizing, a series of binary variable indicating whether to select a graph or not? Based on description in subsequent part, you are basically selecting graphs with the highest score, in that case it is unnecessary to formulate an optimization problem, just state your scoring function.\n\n2. The description is confusing, eg, what is M in line 296? The number of subgraphs being selected?\n\n3. Efficiency is important in data selection, but the authors do not discuss it in the main body of the paper. I assume the method is slower than baseline pretraining method but how slow is it?\n\n4. The motivation of the other four property criteria is unclear, especially when they are highly correlated with the network entropy. Can we just drop the other four criteria?\n\n5. The authors claim in the conclusion that APT can enhance the model with a fewer number of input data, but I can't find evidence for that. Is APT only trained on fewer data than baselines?\n\nOverall, the description of the proposed method could be improved. In addition, the authors claimed they are not curriculum learning but I don't think so. Time-adaptive selection strategy is just curriculum learning in a broader sense, I think easy-first is just one type of curriculum.\n\nQUESTIONS:\nSee weakness above\n\nLIMITATIONS:\nThe authors do not state limitations in either the appendix or the main body of the paper, please add them.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduced an novel graph pre-training pipeline from a data-centric view by proposing a graph data selector. The proposed approach involved sequentially feeding representative data into the model for pre-training. These representative data were determined by a data selector, which utilized prediction uncertainty and graph statistics to suggest the most relevant examples. This novel methodology offers a fresh perspective on graph pre-training and demonstrates the importance of data selection in the process.\n\nSTRENGTHS:\n1. The idea is novel and interesting.\n2. The proposed training pipeline is tested on many tasks from different domains.\n\nWEAKNESSES:\n1. The utilization of predictive uncertainty based on pre-training loss lacks convincing evidence.\n\n2. The proposed training pipeline requires input graphs to be processed order-by-order, which may introduce unnecessary bias and is not scalable for large graph datasets.\n\nQUESTIONS:\nFirst, there are several questions about the predictive uncertainty:\n\n1. The loss used for graph pre-training may not effectively align with the actual loss of the downstream task [1,2]. Consequently, the predictive uncertainty based on the current definition may lack meaningful interpretation. A more promising approach would be to ensure that the uncertainty of the pretraining loss accurately reflects the difference between the pre-training loss and the downstream task loss.\n\n2. Prediction uncertainty has been extensively explored in previous research [3,4]. While many existing methods have been proposed with ground-truth labels, it would be beneficial to have discussions (and even experiments) that shed light on the connection and distinctions between different approches of uncertainty modeling.\n\nFor experimentds\n\n3. The experimental setups lack clarity. It is unclear whether data from all domains are used to pre-train a single model or if separate models are pre-trained for different domains. \n\nOthers\n\n4. Please check descriptions in lines 215~217. Smaller gamma seems leading to important roles of the predictive uncertainty term.\n\n5. Choosing subgraphs seems to be a crucial component of the pre-training algorithm. However, the paper lacks a thorough explanation or detailed descriptions of this process, which would greatly improve the understanding of this important aspect.\n\n6. Recently, there are many related work focusing on the data-centric approaches [2,5,6], espeically for graph pre-training [2,5]. It would be great and helpful to readers if the authors could provide a brief discussion on the connections and differences between their work and these existing approaches.\n\nRef.\n\n[1] Does gnn pretraining help molecular representation? NeurIPS 2022.\n\n[2] Data-Centric Learning from Unlabeled Graphs with Diffusion Model. Arxiv.\n\n[3] Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ICML 2016.\n\n[4] Single-model uncertainties for deep learning. NeurIPS 2019.\n\n[5] Analyzing data-centric properties for graph contrastive learning. NeurIPS 2022.\n\n[6] Data-centric ai: Perspectives and challenges. Arxiv.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a data-centric framework, abbreviated as APT, for cross-domain graph pre-training. APT is composed of a graph selector and a graph pre-training model. The core idea is to select the most representative and informative data points based on the inherent properties of graphs, as well as predictive uncertainty. For the pre-training stage, when fed with the carefully selected data points, a proximal term is added to prevent catastrophic forgetting and remember all the contributions of previous input data.\n\nSTRENGTHS:\n1.This paper proposes that big data is not a necessity for pre-training GNNs. Instead of training on a massive amount of data, it is more reasonable to select a few suitable samples for pre-training. This approach can also reduce the amount of data and computational costs. Compared to pre-training on the entire dataset, selecting a more carefully selected subset of data for pre-training can indeed achieve better results.\n2.This paper provides theoretical justification for the connection between uncertainties by establishing a provable connection between the proposed predictive uncertainty and the conventional definition of uncertainty. The predictive uncertainty is defined in the representation space, which enables the identification of more challenging samples that can benefit the model training more significantly.\n3.The entire framework seems very reasonable, and the process is described clearly. From the experiments, it appears that good results have been achieved.\n\nWEAKNESSES:\n1.This paper mentions both \"data-active\" and \"data-centric\" concepts. It may be helpful to clarify the relationship between these two to avoid confusion. Maybe only using \"data-active\" in the paper.\n2.There is no individual ablation experiment about the graph selector to demonstrate the effectiveness for each of the ﬁve graph properties.\n3.There is a spelling error in the title, 'Prespective' should be corrected to 'Perspective'.\n\nQUESTIONS:\nQ1 Are all the Graph properties used in this paper useful when dealing with graph-inherent features? Also, what is the computational complexity since multiple properties need to be calculated?\nQ2 Why did the performance of the \"dd\" dataset decrease after fine-tuning compared to the frozen parameters of APT in table 2?\nQ3 I am slightly concerned that if all the five graph properties are useful for pre-training.\nQ4 In existing works, is there any works that define model uncertainty in the representation space? And what is the difference between theirs and this work?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach to pre-training graph neural networks (GNNs) called the data-active graph pre-training (APT) framework. This framework introduces a unique method that involves a graph selector and a pre-training model. These two components work together in a progressive and iterative way to choose the most representative and instructive data points for pre-training.\n\nSTRENGTHS:\n1. The key insight of the paper is that using fewer, but carefully selected data can lead to better downstream performance compared to using a massive amount of input data. This approach challenges the common practice in machine learning of using large datasets for pre-training, a phenomenon the authors refer to as the \"curse of big data\" in graph pre-training.\n\n2. The paper is well-motivated, well-written and easy to understand.\n\n3. The experimental results demonstrate that the proposed method outperformed GCC, which is the previous SOTA and a direct ablation to the proposed APT framework.\n\nWEAKNESSES:\n1. The proposed method is incremental. The proposed “predictive uncertainty” is adapted from curriculum learning, and “graph properties” are common statistics. \n\n2. The proposed methods are not able to handle node features. It only provides structural information which does not align with most real-world scenarios. As a result, performance is not comparable with a supervised model trained on node features.\n\n3. The node classification performance on homophily graphs (which is an important family of graphs) are much lower than ProNE. The improvement in graph classification tasks seems limited.\n\n4. Selection based on graph properties might cause potential test data leakage because the graph properties are selected according to test performance (as shown in Figure 2).\n\nQUESTIONS:\nIn line 67, section 2, the authors present an example of a triangle to show what kind of knowledge the pre-trained model can learn. However, the proposed subgraph instance discrimination task can break down that knowledge. For example, ego-graph sampling can break down the triangle. Can the authors provide some insights or justification for this issue?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}