SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} - \phi_ {\text {u n c e r t a i n}} := \sup  _ {f} \mathbb {E} _ {\left(x _ {i}, g \left(x _ {i}\right)\right) \sim P _ {X, g (X)}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \log \frac {e ^ {f \left(x _ {i} , g \left(x _ {i}\right)\right)}}{\sum_ {j = 1} ^ {n} e ^ {f \left(x _ {i} , g \left(x _ {j}\right)\right)}} \right] \tag {11} \\ \leq D _ {\mathrm {K L}} \left(P _ {X, g (X)} \| P _ {X} P _ {g (X)}\right) = \mathrm {I} (X; g (X)) = \mathrm {I} (X; Z), \\ \end{array}
$$
where  $f(x; y)$  is any similarity scoring function like  $f(x; y) = \cos(x) \cos(y) / \tau$ .
Therefore, we can now prove that lower existing uncertainty  $\mathcal{L}_{\mathrm{CE}}(\zeta)$  over all downstream classifiers cannot be achieved without lower uncertainty  $\phi_{uncertain}(\zeta)$ .
Theorem 1 (Theoretical connection between uncertainty.). Let  $\mathcal{X}$ ,  $\mathcal{Z}$  and  $\mathcal{Y}$  be the input space, representation space and label set of downstream classifier. Assume that the distribution of labels is a uniform distribution over  $\mathcal{Y}$ . Consider the set of downstream classifiers  $\mathcal{H} = \{h : \mathcal{Z} \to \mathcal{Y}\}$ . For any graph encoder  $f : \mathcal{X} \to \mathcal{Z}$ ,
$$
\mathcal {L} _ {\mathrm {C E}} (\zeta) \geq \log \left(\frac {\log | \mathcal {Y} |}{\log 2 - \phi_ {\text {u n c e r t a i n}} (\zeta)}\right), \tag {12}
$$
where  $\mathcal{L}_{\mathrm{CE}}$  denotes the conventional uncertainty (which is the cross entropy loss here) estimated from the composition of graph encoder and downstream classifier  $h \circ f$ , and  $\phi_{\mathrm{uncertain}}$  is our proposed uncertainty estimated from graph encoder  $f$ .
Proof. For  $h \in \mathcal{H}$ , we have the Markov chain
$$
Y \to X \stackrel {f} {\to} f (X) \stackrel {h} {\to} (h \circ f) (X),
$$
where  $X, Y$  are random variables for input and label distributions respectively. The first Markov chain  $Y \rightarrow X$  can be understood as a generative model for generating inputs according to the conditional probability distribution  $\mu_{X|Y}$ . Therefore, applying Lemmas 1 and 2, we obtain the inequality,
$$
\Pr \left[ h (f (X)) \neq Y \right] \geq 1 - \frac {\operatorname {I} (Y ; (h \circ f) (X)) + \log 2}{\log | \mathcal {Y} |} \geq 1 - \frac {\operatorname {I} (X ; f (X)) + \log 2}{\log | \mathcal {Y} |}. \tag {13}
$$
$\operatorname*{Pr}\left[h\left(f(X)\right) \neq Y\right]$  represents the error rate. According to the definition of cross entropy, the cross entropy for node can be denoted as follows:
$$
\mathcal {L} _ {\mathrm {C E}} = - \log (1 - \Pr [ h (f (X)) \neq Y ]). \tag {14}
$$
By combining Eq. (13), Eq. (10) with Eq. (14), we have
$$
\begin{array}{l} \Pr \left[ h (f (X)) \neq Y \right] = 1 - e ^ {- \mathcal {L} _ {\mathrm {C E}} (\zeta)} \geq 1 - \frac {\operatorname {I} (X ; Z) + \log 2}{\log | \mathcal {Y} |} \geq 1 + \frac {\phi_ {\text {u n c e r t a i n}} (\zeta) - \log 2}{\log | \mathcal {Y} |}, \tag {15} \\ \mathcal {L} _ {\mathrm {C E}} (\zeta) \geq \log \left(\frac {\log | \mathcal {Y} |}{\log 2 - \phi_ {\text {u n c e r t a i n}} (\zeta)}\right). \\ \end{array}
$$
Thus, we completed the proof.
Advantages of our proposed uncertainty Except for the theoretical advantages, we further discuss two advantages of using the model loss (i.e., InfoNCE loss) as predictive uncertainty for data selection. First, InfoNCE loss is exactly the objective function of our model, so what we do is actually to select the samples with the greatest contributions to the objective function. Such strategy has been justified to accelerate convergence and enhance the discriminative power of the learned representations [18, 28, 29, 31]. Second, as the loss function of our model, InfoNCE is already computed during the training, and thus no additional computation expense is needed in the data selection phase.
The overall algorithm for APT is given in Algorithm 1. Given a collection of graphs  $\mathcal{G} = \{G_1, \ldots, G_N\}$  from various domains, APT aims to pre-train a better generalist GNN (i.e., pre-training model) on wisely chosen graphs and samples. Our APT pipeline involves the following three steps. (i) At the beginning, the graph selector chooses a graph for pre-training according to the graph properties (line 1). (ii) Given the chosen graph, the graph selector chooses the subgraph samples whose predictive uncertainty is higher than  $T_s$  in this graph (line 3). (iii) The selected samples are then fed into the model for pre-training until the predictive uncertainty of the chosen graph is below  $T_g$  or the number of training iterations on this chosen graph reaches  $F$  (line 4-5). (iv) The model's feedback in turn helps select the most needed graph based on predictive uncertainty and graph properties until the predictive uncertainty of any candidate graph is low enough (line 6-7). The last three steps are repeated until the iteration number reaches a pre-set maximum value  $T$  (which can be considered as the total iteration number required to train on all selected graphs).
Algorithm 1 Overall algorithm for APT.
Input: A collection of graphs  $\mathcal{G} = \{G_1, \dots, G_N\}$ , maximal period  $F$  of training one graph, trade-off parameter  $\gamma_t = 0$ , hyperparameter  $\{\beta_t\}$ , the learning rate  $\mu$ , the predictive uncertainty threshold of moving to a new graph  $T_g$ , the predictive uncertainty threshold of choosing training samples  $T_s$ , and the maximum iteration number  $T$ .
Output: Model parameter  $\theta$  of the pre-trained graph model.
1: Choose a graph  $G^{*}$  from  $\mathcal{G}$  via the graph selector, and  $\mathcal{G} \gets \mathcal{G} \backslash \{G^{*}\}$ .
2: while The iteration number reaches  $T$  do
3: Sample instances with predictive uncertainty higher than  $T_{s}$  from  $G^{*}$  via the graph selector.
4: Update model parameters  $\theta \gets \theta -\mu \nabla_{\theta}\mathcal{L}(\theta)$
5: if  $\phi_{\mathrm{uncertain}}(G^{*}) < T_{g}$  or the model has been trained on  $G^{*}$  by  $F$  iterations then
6: Update the trade-off parameter  $\gamma_{t} \sim \mathrm{Beta}(1, \beta_{t})$ .
7: Choose a graph  $G^{*}$  from  $\mathcal{G}$ , and  $\mathcal{G} \gets \mathcal{G} \backslash \{G^{*}\}$ .
8: end if
9: end while
The time complexity of our model mainly consists of five components: data augmentation, GNN encoder propagation, contrastive loss, sample selection and graph selection. Suppose the maximal number of nodes of subgraph instances is  $|V|$ , the batch size is  $B$ , and  $D$  is the representation dimension. (1) As for the data augmentation, the time complexity of random walk with restart is at least  $O(B|V|^3)$  [34]. (2) The time complexity of GNN encoder propagation depends on the architectures of the backbone GNN. We denote it as  $X$  here. (3) The time complexity of the contrastive loss is  $O(B^2D)$  [17]. (4) Sample selection is conducted by choosing the samples with high contrastive loss (the loss is computed before), which costs  $O(B)$ . (5) Graph selection costs
Table 3: Datasets for pre-training and testing, where * denotes the average statistic of multiple graphs under graph classification setting.  $|V|$  and  $|E|$  denote the number of nodes and the number of edges in a graph, respectively.
Type	Name	|V|	|E|	Description
pre-training data	citations	arxiv	86,376	517,563	citations between papers on the arxiv
dblp	93,156	178,145	same as above (dblp)
patents-main	240,547	560,943	citations between US patents
social	soc-sign0902	81,867	497,672	friend/foe links between the users ofSlashdot in Feb. 2009
soc-sign0811	77,350	468,554	same as above (Nov. 2008)
wiki-vote	7,115	100,762	voting relationships between wikipedia users
academia	137,969	369,692	friendships between academics on Academia.edu
michigan	30,147	1,176,516	friendships between Facebook users in University of Michigan
msu	32,375	1,118,774	same as above (Michigan State University)
uillinois	30,809	1,264,428	same as above (University of Ullinois)
movie	imdb	896,305	3,782,447	relationships between actors and movies
test data	protein	dd	284.32*	715.66*	molecular interactions between amino acids
ogbproteins	132,534	39,561,252	biologically associations between proteins
image	msrc-21	77.52*	198.32*	adjacency between superpixels of the image segmentations
movie	imdb-binary	19.77*	96.53*	collaboration relationships between actors/actresses
citations	cora	2,708	5,278	citations between Machine Learning papers
pubmed	19,717	88,648	citations between scientific papers
ogbarxiv	169,343	1,166,243	citation network between computer science arxiv papers
web	cornell	183	280	hyperlinks between webpages collected from Cornell University
wisconsin	251	466	same as above (Wisconsin University)
transportation	brazil	131	2,077	commercial flights between airports in Brazil
others	dd242	1,284	3,303	this network dataset is in the category of labeled networks
dd68	775	2,093	same as above
dd687	725	2,600	same as above
$O(|\mathcal{G}|M^2 D)$  (where  $M$  the number of samples needed to compute the predictive uncertainty of a graph, and  $|\mathcal{G}|$  is the number of graphs that have not been selected). This step is executed only in a few epochs (around  $6\%$  in our current model), so we ignore its time overhead in graph selection. Therefore, the overall time complexity of APT in each batch is  $O(B|V|^3 + X + B^2 D + B)$ .
D Dataset Details
The graph datasets for pre-training and testing in this paper are collected from a wide spectrum of domains (see Table 3 for an overview). The consideration of the graphs for pre-training and test is as follows. When selecting pre-training data, we hope that the graph size is at least hundreds of thousands to contain enough information for pre-training. When selecting test data, we hope that: (1) some test data is in the same domain as the pre-training data, and some is cross-domain, so as to comprehensively evaluate our model's in and across-domain transferability. Accordingly, the in-domain test data is selected from the type of movie and citations, and the others test data are across-domain; (2) the size of test graphs can scale from hundreds to millions, including large-scale datasets with millions of edges from Open Graph Benchmark [10].
Regarding the pre-training datasets, arxiv, dblp and patents-main are citation networks collected from [2], [37] and [9], respectively. Imdb is the collection of movie from [27]. As for the social networks, soc-sign0902 and soc-sign0811 are collected from [16], wiki-vote is from [15], academia is from [7], and michigan, msu and uillions are from [32]. Regarding the test datasets, we collect the protein network dd and ogbproteins from [6] and [10]. The image network msrc-21 is from [23]. The movie network imdb-binary is from [36]. The citation networks, cora, pubmed and ogbarxiv, are from [21], [22] and [10]. The web networks cornell and wisconsin are collected from [24]. The transportation network brazil is form [26], and dd242, dd68 and dd687 are from [27].
The detailed graph properties of the pre-training data and test data are presented in Table 4 and Table 5, respectively.
Table 4: Detailed structural properties of pre-training datasets, where avg properties equals to  $\mathrm{MEAN}(\hat{\phi}_{\mathrm{entropy}},\hat{\phi}_{\mathrm{density}},\hat{\phi}_{\mathrm{avg\_deg}},\hat{\phi}_{\mathrm{deg\_var}}, - \hat{\phi}_{\alpha})$  in Eq. (4), and  $nei_{2}$  denotes the average number and standard deviation of 2-hop neighbors,  $|V|$  and  $|E|$  denote the number of nodes and the number of edges in a graph, respectively.
Properties
Dataset	|V|	|E|	avg properties	avg degree	degree var	density	entropy	α	nei2(avg, std)	avg clustering coef
soc-sign0902	81867	497672	-0.32	13.16	1643.20	1.49e-04	3.91	1.51	1192.33, 2305.99	0.06
soc-sign0811	77350	468554	-0.32	13.12	1631.77	1.57e-04	3.93	1.52	1226.93, 2312.03	0.05
imdb	896305	3782447	-0.66	9.44	298.27	9.42e-06	3.07	1.53	316.16, 614.03	5e-05
patent	240547	560943	-0.96	5.66	34.95	1.94e-05	2.04	1.57	117.23, 172.40	0.08
academia	137969	369692	-0.89	6.36	102.14	3.88e-05	2.38	1.57	101.40, 225.96	0.14
wiki-Vote	7115	103689	0.74	29.32	3314.79	4.10e-03	4.46	1.40	972.03, 1045.13	0.14
dblp	93156	178145	-1.12	4.82	58.05	4.11e-05	2.16	1.72	58.85, 90.46	0.27
arxiv	86376	517563	-0.43	12.98	382.12	1.39e-04	3.22	1.41	145.21, 309.12	0.68
michigan	30147	1176516	1.40	79.05	6369.17	2.59e-03	4.78	1.23	4683.90, 3655.25	0.21
msu	32375	1118774	1.13	70.11	5087.53	2.13e-03	4.62	1.23	4567.63, 3465.16	0.21
uillinois	30809	1264428	1.44	83.08	6306.02	2.66e-03	4.78	1.22	5267.10, 3831.37	0.21
Table 5: Detailed structural properties of test datasets, where  $nei_2$  denotes the average number and standard deviation of 2-hop neighbors, and the numbers with  $*$  denote the average statistics of multiple graphs under graph classification setting.  $|V|$ ,  $|E|$  and  $|G|$  denote the number of nodes in a graph, the number of edges in a graph and the number of graphs in graph classification datasets, respectively.
Properties Dataset	|V|	|E|	|G|	avg degree	degree var	density	entropy	nei2(avg, std)	avg clustering coef # of classes
imdb-binary	19.77*	193.06*	1000	9.89*	116.01*	1.04*	1.07*	24.89*,15.91*	0.95*
msrc-21	77.52*	198.32*	563	6.10*	30.26*	6.81e-02*	1.71*	17.00*,5.81*	0.51*
dd	284.32*	715.66*	1178	6.00*	27.60*	2.78e-02*	1.65*	14.30*,5.68*	0.48*
cora	2708	5278	/	4.90	42.53	1.44e-03	1.71	34.98,47.70	0.24
pubmed	19717	44327	/	5.50	75.44	2.28e-04	2.23	57.10,82.72	0.06
brazil	131	1074	/	16.85	539.18	1.26e-01	3.14	92.27,28.50	0.66
dd242	1284	3303	/	6.14	28.80	4.01e-03	1.68	14.57,4.30	0.47
dd68	775	2093	/	6.40	33.42	6.98e-03	1.76	17.40,8.93	0.44
dd687	725	2600	/	8.17	55.78	9.91e-03	2.01	25.45,9.96	0.48
wiscosin	251	466	/	4.65	76.26	1.49e-02	1.84	68.04,58.22	0.23
cornell	183	280	/	4.04	58.48	1.68e-02	1.74	54.09,44.30	0.18
ogbarxiv	16343	1157799	/	14.67	4898.17	8.07e-05	3.63	3483.08,6711.40	0.23
ogbproteins	132534	39561252	/	598.00	742637.58	4.50e-03	6.84	32265.17,19401.46	0.28
E Additional Observations of Curse of Big Data Phenomenon
This section provides more comprehensive observations to support the curse of big data phenomenon in graph pre-training, i.e., more training samples and graph datasets do not necessarily lead to better downstream performance.
We investigate 3630 experiments with GCC [25] and GraphCL [39] model with different model configurations (i.e., the number of GNN layers is set to be 3, 4 and 5 respectively), when pre-trained on all training graphs listed in Table 3 and evaluated on different test graphs (annotated in the upper left corner of each figure) under freezing setting. For each experiment, we calculate the mean and standard deviation over 10 evaluation results of the downstream task with random training/testing splits.
The observations of GCC and GraphCL model can be found in Figure 6 and Figure 7 respectively. The downstream results of different test data are presented in separate rows. The figures in left three columns present the effect of scaling up the number of graphs on the downstream performance under different model configurations (i.e., the number of GNN layers) respectively. We first pre-train the model with only two input graphs, and the result is plotted in a dotted line. The largest standard deviation among the results w.r.t different graph last is also marked by the blue arrow. The figures in the last column illustrate the effect of scaling up sample size (log scale) on the performance.
**Curse of big data phenomenon in molecular pre-training.** We also conduct observations of scaling up the number of graphs on the downstream performance in molecular pre-training. We take AttrMasking [13], ContextPred [13], and GraphCL [39] as backbone models. Following their settings, the pre-training data is sampled from the molecular dataset ZINC15 and we take the molecular dataset BBBP as downstream data. We directly adopt the default settings of these pre-train models except for the number of molecules for pre-training. The number of molecular graphs are selected as {1,000,
5,000, 10,000, 50,000, 100,000}. The results are shown in Figure 5, which suggest that "curse of big data" phenomenon is still found even when pre-training and downstream data are all molecules.
Table 6: The value of parameters for fitting the curve according to the function  $f\left( x\right)  = {a}_{1}\ln x/{x}^{{a}_{2}} +$ ${a}_{3}\left( {{a}_{1},{a}_{2},{a}_{3} > 0}\right)$  ,based on the points in the last column in Figure 6 and Figure 7.
Parameter Dataset	GCC	GraphCL
a1	a2	a3	a1	a2	a3
cora	0.45	0	33.43	1038.19	1.24	17.15
pubmed	4.74	0.11	39.63	3.50	0.12	36.83
brazil	39.10	0.09	0	29.59	0.19	41.83
dd242	6.60	0.08	3.82	7.84	0.12	0
dd68	6.44	0.11	3.83	2.08e+17	6.55	10.37
dd687	968.01	1.37	10.31	5.21	0.12	1.35
wisconsin	26.78	0.11	16.59	11.42	0.21	38.24
cornell	15.46	0.41	45.45	7.84	4.83	51.79
imdb-binary	7.43	0.13	64.97	1.06	0	51.69
dd	0.81	0.31	75.53	13.76	0.10	36.33
msrc	5.56	0.11	4.85	4.71	0.13	0
(a) AttrMasking
(b) ContextPred
Figure 5: The effect of scaling up the number of graph datasets on the downstream performance under molecular pre-training on different pre-training models.
(c) GraphCL
The explanation of convex hull fit. In order to better show the changing trend, the blue curve in the last column in Figure 6 and Figure 7 is fitted to the convex hull of the points. The convex hull is proposed to capture the performance of a randomized classifier made by choosing pre-training models with different probabilities [1].
We first introduce the concept of randomized classifier. Given two classifiers with training sample size and downstream performance  $c_{1} = (c_{1}^{\mathrm{sz}}, c_{1}^{\mathrm{ds}})$  and  $c_{2} = (c_{2}^{\mathrm{sz}}, c_{2}^{\mathrm{ds}})$ , a randomized classifier can be made to choose the first classifier with probability  $p$  and the second classifier with probability  $1 - p$ . Then the output of the randomized classifier is  $pc_{1} + (1 - p)c_{2}$ , which is the convex combination of  $c_{1}$  and  $c_{2}$ . All the points on this convex combination can be obtained by choosing different  $p$ . Extend the notion to the case of multiple classifiers, we can consider the output of such a randomized classifier to be a convex combination of the outputs of its endpoints [1]. All the points on the convex hull are achievable. Therefore, the output of the randomized classifier is equivalent to the convex hull of our trained classifiers' performance.
In our experiments, we include the upper hull of the convex hull of the model performances, i.e., the highest downstream performance for every given sample size. Such convex hull fit is proved to be robust to the density of the points in each figure [1].
A final remark is that our observations on different downstream datasets do not result in a one-model-fits-all trend. So we propose to fit a complicated curve whose function has form  $f(x) = a_{1}\ln x / x^{a_{2}} + a_{3}$  ( $a_{1}, a_{2}, a_{3} > 0$ ) to the best performing models (i.e., the convex hull fit as discussed above). The fitted parameters  $a_{1}, a_{2}$  and  $a_{3}$  in this function of each curve are given in Table 6.
(a) Node classification
Figure 6: The additional observations of curse of big data phenomenon, performed on different GCC pre-training models. Left three columns present the effect of scaling up the number of graphs on the downstream performance under different model configurations (i.e., the number of GNN layers) respectively. Last column illustrates the effect of scaling up sample size (log scale) on the performance.
Figure 7: The additional observations of curse of big data phenomenon, performed on different GraphCL pre-training models.
F Empirical Study of Graph Properties
Additional properties for part (b) in Figure 2. In Figure 8, we plot the Pearson correlation between the graph properties of the graph used in pre-training (shown in the  $y$ -axis) and the performance of the pre-trained model using this graph on different unseen test datasets (shown in the  $x$ -axis). Note that the pre-training is performed on each of the input training graphs (in Table 3) via GCC. The results indicate that network entropy, density, average degree and degree variance exhibit a clear positive correlation with the performance, while the scale-free exponent presents an obviously negative relation with the performance. On the contrary, some other properties of graphs, including clique number, transitivity, degree assortativity and average clustering coefficient, do not seem to have connections with downstream performance, and also exhibit little or no correlation with the performance. Therefore, the favorable properties of network entropy, density, average degree, degree variance and the scale-free exponent of a real graph are able to characterize the contribution of a graph to pre-training.
Figure 8: Pearson correlation between the structural features of the graph used in pre-training and the performance of the pre-trained model (using this graph) on different unseen test datasets.
Detailed illustrations of Figure 3. In Figure 3, the illustrative graphs are generated by the configuration model with 15-18 nodes. The shaded area groups the illustrative graphs whose network entropy and graph properties are similar. Each four points on the same horizontal coordinate represent four graph properties of an illustrating graph. Each curve is fitted by least squares and represents the relation between entropy and other graph properties.
Additional real-world example for Figure 3. In Figure 9, we provide a real-world example of how network entropy correlates with four typical structural properties (in red), as well as the performance of the pre-trained model on test graphs (in blue). Numerical experiments again support our explanation (or intuition) of their strong correlation.
(a) Density
(b) Average degree
(c) Degree variance
Figure 9: The red plot shows the network entropy (left  $y$ -axis) versus typical structural properties in a graph (i.e., density, average degree, degree variance, and the parameter  $\alpha$  in a scale-free network), and the blue one shows the pre-training performances on wisconsin dataset (right  $y$ -axis) versus structural features.
(d) Scale-free exponent
The number reported in all the experiments are the mean and standard deviation over 10 evaluation results of the downstream task with random training/testing splits. When conducting the downstream task, For each dataset, we consistently use  $90\%$  of the data as the training set, and  $10\%$  as the testing set. We conduct all experiments on a single machine of Linux system with an Intel Xeon Gold 5118 (128G memory) and a GeForce GTX Tesla P4 (8GB memory).
Implementations of our model. The regularization for weights of the model in Eq. (5) is applied to first 2 layers of GIN. The maximal period of training one graph  $F$  is 6, the maximum iteration number  $T$  is 100, and the predictive uncertainty thresholds  $T_{s}$  and  $T_{g}$  are set to be 3 and 2 respectively. The selected instances are sampled from 20,000 instances each epoch. Since the pre-training model is unable to provide precise predictive uncertainty in the initial training stage, the model is warmed up over the first 20 iterations. Since we adopt GCC as the backbone pre-training model, the other settings are the same as GCC.
Our model is implemented under the following software settings: Pytorch version 1.4.0+cu100, CUDA version 10.0, networkx version 2.3, DGL version 0.4.3post2, sklearn version 0.20.3, numpy version 1.19.4, Python version 3.7.1.
Implementations of baselines. We compare against several graph representation learning methods. For implementation, we directly adopt their public source codes and most of their default hyperparameters. The key parameter settings and code links can be found in Table 7.