SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
h _ {\mathcal {G} _ {i}} ^ {\mathrm {G P F}} = \operatorname {S u m} \left(\mathbf {H} _ {i} ^ {\mathrm {G P F}}\right) \tag {74}
$$
where  $[1]^{\mathrm{T}}$  is a column vector with all 1's, and  $p\in \mathbb{R}^{1\times F}$  is the extra learnable vector of GPF. The squared regression loss  $l$  can be expressed as:
$$
l = \sum_ {i} ^ {m} \left(h _ {\mathcal {G} _ {i}} \cdot \theta - y _ {i}\right) ^ {2} \tag {75}
$$
where  $\theta \in \mathbb{R}^{F' \times 1}$  is a linear projection head. In such case, the optimal tuning loss of fine-tuning  $l_{\mathrm{FT}}$  and GPF  $l_{\mathrm{GPF}}$  can be expressed as:
$$
l _ {\mathrm {F T}} = \min  _ {\mathbf {W}, \theta} \sum_ {i} ^ {m} \left(\operatorname {S u m} \left(\mathbf {S} _ {i} \cdot \mathbf {X} _ {i} \cdot \mathbf {W}\right) \cdot \theta - y _ {i}\right) ^ {2} \tag {76}
$$
$$
l _ {\mathrm {G P F}} = \min  _ {p, \theta} \sum_ {i} ^ {m} \left(\operatorname {S u m} \left(\mathbf {S} _ {i} \cdot \left(\mathbf {X} _ {i} + [ 1 ] ^ {\mathrm {T}} \cdot p\right) \cdot \mathbf {W}\right) \cdot \theta - y _ {i}\right) ^ {2} \tag {77}
$$
Before we prove Theorem 2, we first illustrate the following proposition.
Proposition 7. Given a series of graphs  $\mathcal{D} = \{\mathcal{G}_1\colon (\mathbf{A}_1,\mathbf{X}_1),\ldots ,\mathcal{G}_m\colon (\mathbf{A}_m,\mathbf{X}_m)\}$  and a linear projection  $\hat{\mathbf{W}}$  there exist  $\hat{p}\in \mathbb{R}^{1\times F}$ $\hat{\theta}\in \mathbb{R}^{F^{\prime}\times 1}$  , and  $\delta >0$  that satisfy:
$$
\sum_ {i} ^ {m} \left\| \operatorname {S u m} \left(\mathbf {S} _ {i} \cdot \left(\mathbf {X} _ {i} + [ 1 ] ^ {\mathrm {T}} \cdot \hat {p}\right) \cdot \hat {\mathbf {W}}\right) \cdot \hat {\theta} - \operatorname {S u m} \left(\mathbf {S} _ {i} \cdot \mathbf {X} _ {i} \cdot \mathbf {W} ^ {\prime}\right) \cdot \theta^ {\prime} \right\| _ {2} > \delta \tag {78}
$$
for any  $\mathbf{W}' \in \mathbb{R}^{F \times F'}$ ,  $\theta' \in \mathbb{R}^{F' \times 1}$ .
For analytical simplification, we gather all the unique node features in  $\mathcal{D}$  into a set  $S_{\mathbf{X}} = \{x_1,\dots ,x_l\}$ . Consequently, the node feature matrix  $\mathbf{X}_i$  of any graph  $\mathcal{G}i$  can be constructed from elements in the set  $\mathcal{S}\mathbf{X}$ . Next, we proceed to expand the function  $\mathrm{Sum}(\cdot)$  as:
$$
h _ {\mathcal {G} _ {i}} = \operatorname {S u m} \left(\mathbf {S} _ {i} \cdot \mathbf {X} _ {i} \cdot \mathbf {W}\right) = \sum_ {k} ^ {N _ {i}} \sum_ {j} ^ {N _ {i}} \mathbf {S} _ {i, (j, k)} \cdot \mathbf {X} _ {i, (k,:)} \cdot \mathbf {W} \tag {79}
$$
$$
\begin{array}{l} h _ {\mathcal {G} _ {i}} ^ {\mathrm {G P F}} = \operatorname {S u m} \left(\mathbf {S} _ {i} \cdot \left(\mathbf {X} _ {i} + \left[ 1 \right] ^ {\mathrm {T}} \cdot \hat {p}\right) \cdot \mathbf {W}\right) \\ = \sum_ {k} ^ {N _ {i}} \sum_ {j} ^ {N _ {i}} \mathbf {S} _ {i, (j, k)} \cdot \mathbf {X} _ {i, (k,:)} \cdot \mathbf {W} + \sum_ {k} ^ {N _ {i}} \sum_ {j} ^ {N _ {i}} \mathbf {S} _ {i, (j, k)} \cdot p \cdot \mathbf {W} \tag {80} \\ \end{array}
$$
where  $\mathbf{S}_{i,(j,k)}$  denotes the element in the  $j$ -th row and  $k$ -th column of  $\mathbf{S}_i$ , and  $\mathbf{X}_{i,(k,:)}$  denotes the  $k$ -th row vector of  $\mathbf{X}_i$ . In order to express the representations of different graphs in a unified form, we rewrite the above formulas to calculate the graph representations  $h_{\mathcal{G}_i}$  and  $h_{\mathcal{G}_i}^{\mathrm{GPF}}$  from the node representations in  $\mathcal{S}_{\mathbf{X}}$ :
$$
h _ {\mathcal {G} _ {i}} = \sum_ {k} ^ {N _ {i}} \sum_ {j} ^ {N _ {i}} \mathbf {S} _ {i, (j, k)} \cdot \mathbf {X} _ {i, (k,:)} \cdot \mathbf {W} = \sum_ {j} ^ {l} c _ {i, j} \cdot x _ {j} \cdot \mathbf {W} \tag {81}
$$
$$
\begin{array}{l} h _ {\mathcal {G} _ {i}} ^ {\mathrm {G P F}} = \sum_ {k} ^ {N _ {i}} \sum_ {j} ^ {N _ {i}} \mathbf {S} _ {i, (j, k)} \cdot \mathbf {X} _ {i, (k,:)} \cdot \mathbf {W} + \sum_ {k} ^ {N _ {i}} \sum_ {j} ^ {N _ {i}} \mathbf {S} _ {i, (j, k)} \cdot p \cdot \mathbf {W} \\ = \sum_ {j} ^ {l} c _ {i, j} \cdot x _ {j} \cdot \mathbf {W} + \sum_ {j} ^ {l} c _ {i, j} \cdot p \cdot \mathbf {W} \tag {82} \\ \end{array}
$$
where  $c_{i,j}$  is the coefficient of  $x_j$  for the graph  $\mathcal{G}_i$ , which is calculated as  $c_{i,j} = \sum_k \sum_{j'}^{N_i} \mathbf{S}_{i,(j',k)}$  for all  $k$ 's that satisfy  $\mathbf{X}_{i,(k,:)} = x_j$ . We can also get that  $\sum_j^l c_{i,j} = \sum_k^{N_i} \sum_{j'}^{N_i} \mathbf{S}_{i,(j',k)}$ . Then, we rewrite Formula 78 into a matrix form as follows:
$$
\left\| \left(\mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} + \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {\mathrm {T}} \cdot \hat {p}\right) \cdot \hat {\mathbf {W}} \cdot \hat {\theta} - \mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} \cdot \mathbf {W} ^ {\prime} \cdot \theta^ {\prime} \| _ {2} > \delta \right. \tag {83}
$$
where  $\mathbf{X}_{S_{\mathbf{X}}}\in \mathbb{R}^{l\times F}$  denotes the feature matrix for  $S_{\mathbf{X}}$  that satisfies  $\mathbf{X}_{S_{\mathbf{X}},(j,:)} = x_j$ ,  $\mathbf{C}\in \mathbb{R}^{m\times l}$  denotes the coefficient matrix and the element in the  $i$ -th row and  $k$ -th column  $\mathbf{C}_{(i,j)}$  is equal to  $c_{i,j}$ ,  $[\sum_j\mathbf{C}_{(i,j)}]^{\mathrm{T}}$  denotes a column vector with the value of  $i$ -th row is  $\sum_j\mathbf{C}_{(i,j)}$ . We represent  $\mathbf{W}'\cdot \theta' = \hat{\mathbf{W}}\cdot \hat{\boldsymbol{\theta}} + \Delta \mathbf{W}\cdot \boldsymbol{\theta}$ , then we have:
$$
\begin{array}{l} \left\| \left(\mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} + \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {\mathrm {T}} \cdot \hat {p}\right) \cdot \hat {\mathbf {W}} \cdot \hat {\theta} - \mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} \cdot \left(\hat {\mathbf {W}} \cdot \hat {\theta} + \Delta \mathbf {W} \cdot \theta\right) \| _ {2} \right. (84) \\ = \left\| \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {\mathrm {T}} \cdot \hat {p} \cdot \hat {\mathbf {W}} \cdot \hat {\theta} - \mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} \cdot \Delta \mathbf {W} \cdot \theta \right\| _ {2} (85) \\ \end{array}
$$
As described by the condition of Proposition 7,  $\mathbf{W}'$  and  $\theta'$  can be chosen arbitrarily, which means  $\Delta \mathbf{W} \cdot \theta$  can be equal to any  $v \in \mathbb{R}^{F \times 1}$ . Therefore, Proposition 7 can be reformed as below.
Given a series of graphs  $\mathcal{D} = \{\mathcal{G}_1\colon (\mathbf{A}_1,\mathbf{X}_1),\ldots ,\mathcal{G}_m\colon (\mathbf{A}_m,\mathbf{X}_m)\}$  and a linear projection  $\hat{\mathbf{W}}$  there exist  $\hat{p}\in \mathbb{R}^{1\times F}$ $\hat{\theta}\in \mathbb{R}^{F^{\prime}\times 1}$  , and  $\delta >0$  that satisfy:
$$
\left\| \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {T} \cdot \hat {p} \cdot \hat {\mathbf {W}} \cdot \hat {\theta} - \mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} \cdot v \right\| _ {2} > \delta \tag {86}
$$
for any  $v\in \mathbb{R}^{F\times 1}$
We make the assumption that  $\mathbf{C}$  is a column full-rank matrix, which implies that there is no uniform feature distribution shared among different graphs, aligning with real-world scenarios. It is important to note that  $\mathbf{W}$  is pre-trained beforehand, and for any  $\hat{p}$  and  $\hat{\theta}$  satisfying  $\hat{p} \cdot \hat{\mathbf{W}} \cdot \hat{\theta} \neq 0$ , the nondegeneracy condition for Formula 86 is as follows:
$$
\begin{array}{l} \left\| \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {\mathrm {T}} \cdot \hat {p} \cdot \hat {\mathbf {W}} \cdot \hat {\theta} - \mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} \mathbf {X}} \cdot v \right\| _ {2} \neq 0 (87) \\ \rightarrow \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} \cdot v ^ {\prime} = [ 1 ] ^ {\mathrm {T}} \text {h a s n o s o l u t i o n} v ^ {\prime} (88) \\ \end{array}
$$
where  $[1]^{\mathrm{T}}\in \mathbb{R}^{m\times 1}$  is a column vector with all 1's. Therefore, Formula 86 and Proposition 7 hold for all  $\mathbf{X}_{\mathcal{S}_{\mathbf{X}}}$  for which  $\mathbf{X}_{\mathcal{S}_{\mathbf{X}}}\cdot v^{\prime} = [1]^{\mathrm{T}}$  has no solution  $v^{\prime}$ .
Finally, we revisit Theorem 2. Given a series of graphs under the non-degeneracy condition, a pretrained linear projection  $\hat{\mathbf{W}}$ ,  $\hat{p}$  and  $\hat{\theta}$  that satisfy  $\hat{p} \cdot \hat{\mathbf{W}} \cdot \hat{\theta} \neq 0$ , we can construct  $\mathcal{Y}' = \{y_1', \dots, y_m'\}$  as:
$$
y _ {i} ^ {\prime} = \mathbf {C} _ {(i,;)} \cdot \mathbf {X} _ {S _ {\mathbf {X}}} \cdot \hat {\mathbf {W}} \cdot \hat {\theta} + \sum_ {j} \mathbf {C} _ {(i, j)} \cdot \hat {p} \cdot \hat {\mathbf {W}} \cdot \hat {\theta} \tag {89}
$$
Under these conditions, the optimal theoretical tuning results for GPF and fine-tuning can be expressed as:
$$
l _ {\mathrm {G P F}} = \min  _ {p, \theta} \| \left(\mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} + \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {\mathrm {T}} \cdot p\right) \cdot \hat {\mathbf {W}} \cdot \theta - \left(\mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} + \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {\mathrm {T}} \cdot \hat {p}\right) \cdot \hat {\mathbf {W}} \cdot \hat {\theta} \| _ {2} = 0 \tag {90}
$$
$$
l _ {\mathrm {F T}} = \min  _ {\mathbf {W}, \theta} \| \mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} \cdot \mathbf {W} \cdot \theta - (\mathbf {C} \cdot \mathbf {X} _ {\mathcal {S} _ {\mathbf {X}}} + \left[ \sum_ {j} \mathbf {C} _ {(i, j)} \right] ^ {\mathrm {T}} \cdot \hat {p}) \cdot \hat {\mathbf {W}} \cdot \hat {\theta} \| _ {2} > 0 \tag {91}
$$
Here,  $l_{\mathrm{GPF}}$  can be achieved with  $p = \hat{p}$  and  $\theta = \hat{\theta}$ , while  $l_{\mathrm{FT}}$  consistently remains greater than 0, as stated in Proposition 7. Therefore, we can conclude that  $l_{\mathrm{GPF}} < l_{\mathrm{FT}}$ , thus establishing the proof of Theorem 2.
B More Information on Experiments
B.1 Details of the datasets
Dataset overview We utilize the datasets provided by Hu et al. [2020a] for our pre-training datasets. These datasets consist of two domains: chemistry and biology. The chemistry domain dataset comprises 2 million unlabeled molecules sampled from the ZINC15 database [Sterling and Irwin, 2015], along with 256K labeled molecules obtained from the preprocessed ChEMBL dataset [Mayr et al., 2018, Gaulton et al., 2012]. On the other hand, the biology domain dataset includes 395K unlabeled protein ego-networks and 88K labeled protein ego-networks extracted from PPI networks. For the models pre-trained on the chemistry dataset, we employ eight binary graph classification datasets available in MoleculeNet [Wu et al., 2017] as downstream tasks. As for the models pretrained on the biology dataset, we apply the pre-trained models to 40 binary classification tasks, with each task involving the prediction of a specific fine-grained biological function.
Pre-training datasets. The datasets provided by Hu et al. [2020a] consist of two distinct datasets: Biology and Chemistry, corresponding to the biology domain and chemistry domain, respectively. The Biology dataset contains 395K unlabeled protein ego-networks obtained from PPI networks of 50 species. These networks are used for node-level self-supervised pre-training. Additionally, 88K labeled protein ego networks serve as the training data for predicting 5000 coarse-grained biological functions. This graph-level multi-task supervised pre-training aims to predict these functions jointly. Regarding the Chemistry dataset, it comprises 2 million unlabeled molecules sampled from the ZINC15 database [Sterling and Irwin, 2015]. These molecules are utilized for node-level self-supervised pre-training. For graph-level multi-task supervised pre-training, a preprocessed ChEMBL dataset [Mayr et al., 2018, Gaulton et al., 2012] is employed. This dataset contains 456K molecules and covers 1310 different biochemical assays.
Downstream datasets. The statistics of the downstream datasets utilized for the models pre-trained on Biology and Chemistry are presented in Table 3.
Table 3: Statistics of datasets for downstream tasks.
Dataset	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	PPI
# Proteins / Molecules	2039	7831	8575	1427	1478	93087	41127	1513	88K
# Binary prediction tasks	1	12	617	27	2	17	1	1	40
B.2 Details of pre-training strategies
We adopt five widely used strategies (tasks) to pre-train the GNN models, which are listed as below:
- Deep Graph Infomax (denoted by Infomax). It is first proposed by Velickovic et al. [2019a]. Deep graph infomax obtains expressive representations for graphs or nodes via maximizing the mutual information between graph-level representations and substructure-level representations of different granularity.
- Edge Prediction (denoted by EdgePred). It is a regular graph reconstruction task used by many models, such as GAE [Kipf and Welling, 2016a]. The prediction target is the existence of edge between a pair of nodes.
- Attribute Masking (denoted by AttrMasking). It is proposed by Hu et al. [2020a]. It masks node/edge attributes and then let GNNs predict those attributes based on neighboring structure.
- Context Prediction (denoted by ContextPred). It is also proposed by Hu et al. [2020a]. Context prediction uses subgraphs to predict their surrounding graph structures, and aims to mapping nodes appearing in similar structural contexts to nearby embeddings.
- Graph Contrastive Learning (denoted by GCL). It embeds augmented versions of the anchor close to each other (positive samples) and pushes the embeddings of other samples (negatives) apart. We use the augmentation strategies proposed in You et al. [2020] for generating the positive and negative samples.
To pre-train our models, we follow the training steps outlined in Hu et al. [2020a] for Infomax, EdgePred, AttrMasking, and ContextPred tasks. We then perform supervised graph-level property prediction to enhance the performance of the pre-trained models further. For models pre-trained using GCL, we follow the training steps detailed in You et al. [2020].
B.3 Results of few-shot graph classification
The results for 50-shot scenarios. Table 4 summarizes the results for 50-shot graph classification.
Table 4: 50-shot test ROC-AUC (%) performance on molecular prediction benchmarks and protein function prediction benchmarks.
Pre-training Strategy	Tuning Strategy	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	PPI	Avg.
Infomax	FT	53.81±3.35	61.42±1.19	53.93±0.59	50.77±2.27	58.6±3.48	66.12±0.63	65.09±1.17	52.64±2.64	48.79±1.32	56.79
GPF	55.52±1.84	65.56±0.64	56.76±0.54	50.29±1.61	62.44±4.11	68.00±0.61	67.68±1.09	54.49±2.54	54.03±0.34	59.41
GPF-plus	58.09±2.12	65.71±0.37	57.13±0.48	51.33±1.14	62.96±3.27	67.88±0.42	66.80±1.43	56.56±6.81	53.78±0.45	60.02
EdgePred	FT	48.88±0.68	60.95±1.46	55.73±0.43	51.30±2.21	57.78±4.03	66.88±0.53	64.22±1.57	61.27±6.10	47.62±1.50	57.18
GPF	50.53±1.35	64.46±0.93	57.33±0.65	51.35±0.76	68.74±6.03	68.08±0.39	66.22±1.90	62.85±5.91	52.81±0.38	60.26
GPF-plus	54.49±4.60	64.99±0.53	57.69±0.61	51.30±1.18	66.64±2.40	68.16±0.48	62.05±3.39	62.60±2.48	53.30±0.34	60.13
AttrMasking	FT	51.26±2.33	60.28±1.73	53.47±0.46	50.11±1.63	61.51±1.45	59.35±1.31	67.18±1.59	55.62±5.04	48.17±2.45	56.32
GPF	54.24±0.74	64.24±0.40	56.84±0.28	50.62±0.88	65.34±1.93	61.34±0.60	67.94±0.48	57.31±6.71	51.26±0.32	58.79
GPF-plus	58.10±1.92	64.39±0.30	56.78±0.25	50.30±0.78	63.34±0.85	63.84±1.13	68.05±0.97	57.29±4.46	51.35±0.32	59.27
ContextPred	FT	49.45±5.74	58.77±0.70	54.46±0.54	49.89±1.16	48.60±3.40	56.14±4.82	60.91±1.84	56.37±1.90	46.33±1.76	53.43
GPF	52.55±1.24	59.73±0.86	55.70±0.29	50.54±0.91	53.03±5.98	61.93±5.84	60.59±1.28	59.91±6.31	50.14±0.33	56.01
GPF-plus	53.76±4.47	60.59±0.51	55.91±0.22	51.44±1.70	52.37±4.30	64.51±4.48	60.84±1.11	64.21±7.30	50.52±0.41	57.12
GCL	FT	54.40±2.87	48.35±1.67	50.29±0.19	53.23±0.87	54.05±4.16	46.73±1.88	60.05±3.80	49.87±1.78	49.94±1.77	51.62
GPF	53.87±2.17	50.58±0.49	52.64±0.50	53.86±0.45	64.44±4.64	47.22±3.55	64.86±1.29	67.56±2.29	50.40±1.17	55.62
GPF-plus	55.89±1.58	50.14±1.09	53.25±0.95	55.46±0.96	65.22±4.51	47.88±1.77	63.99±1.60	64.10±1.85	51.19±1.53	55.89
The results for 100-shot scenarios. We also conducted experiments where the number of training samples in the downstream task was limited to 100 for both the chemistry and biology datasets. The summary of the overall results can be found in Table 5. The experimental findings align with those observed in the 50-shot scenarios. Our proposed graph prompt tuning method achieves the best results in 42 out of 45 cases (14 out of 45 for GPF and 28 out of 45 for GPF-plus). Additionally, the average results of both GPF and GPF-plus surpass the average results of fine-tuning across all pre-training strategies, showcasing the superiority of our proposed graph prompt tuning approach.
B.4 Parameter efficiency analysis
We have computed the number of tunable parameters for full fine-tuning, GPF, and GPF-plus (excluding the task-specific projection head  $\theta$ ) on the chemistry and biology datasets. The statistics are presented in Table 6. The results indicate that the number of tunable parameters in GPF and GPF-plus is several orders of magnitude smaller than that of fine-tuning. Specifically, GPF utilizes no more than  $0.02\%$  of the tunable parameters used in fine-tuning, while GPF-plus utilizes no more than  $0.7\%$  of the tunable parameters used in fine-tuning. Our proposed graph prompt tuning methods
Table 5: 100-shot test ROC-AUC (%) performance on molecular prediction benchmarks and protein function prediction benchmarks.
Pre-training Strategy	Tuning Strategy	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	PPI	Avg.
Infomax	FT	56.29	60.46	55.34	50.49	50.90	65.88	65.81	57.35	49.74	56.91
±1.65	±0.66	±0.20	±1.29	±4.92	±1.76	±1.43	±2.67	±0.72
GPF	56.38	61.54	57.31	54.49	56.49	66.52	68.02	61.67	54.57	59.66
±2.84	±0.28	±0.35	±0.72	±1.98	±0.67	±1.22	±2.76	±0.51
GPF-plus	56.97	62.48	57.64	54.86	57.68	67.00	67.66	61.76	54.66	60.07
±3.46	±0.80	±0.30	±0.49	±0.89	±0.47	±1.13	±2.80	±0.52
EdgePred	FT	51.27	61.48	58.28	52.23	58.50	64.32	59.82	50.86	48.06	56.09
±3.89	±1.21	±0.81	±1.27	±2.54	±2.48	±1.47	±0.85	±2.00
GPF	55.13	63.35	59.09	52.30	65.02	65.47	63.19	48.64	52.52	58.30
±1.27	±0.94	±0.55	±0.54	±4.13	±0.31	±1.49	±1.70	±0.46
GPF-plus	54.20	64.80	59.42	52.47	62.73	65.37	63.18	50.02	53.00	58.35
±5.05	±0.95	±0.23	±0.64	±3.54	±0.37	±1.28	±5.65	±0.44
AttrMasking	FT	54.56	60.95	55.84	50.64	61.16	64.90	61.65	59.03	47.29	57.33
±4.82	±1.28	±0.40	±1.16	±1.19	±1.43	±3.31	±2.89	±1.43
GPF	55.23	63.36	57.66	50.08	63.05	65.58	69.79	59.37	52.31	59.60
±3.14	±0.61	±0.34	±0.59	±4.41	±0.69	±1.78	±3.90	±0.41
GPF-plus	53.58	63.89	57.72	51.70	62.68	66.47	69.35	58.50	52.28	59.57
±2.19	±0.58	±0.37	±0.61	±2.50	±0.43	±1.58	±2.36	±0.89
ContextPred	FT	50.42	60.74	56.00	51.81	51.48	64.87	59.82	50.43	45.39	54.55
±0.57	±0.88	±0.29	±1.77	±2.86	±2.30	±2.00	±3.74	±0.42
GPF	52.33	63.91	57.32	53.55	54.31	65.80	68.51	54.70	50.44	57.87
±5.07	±0.82	±0.30	±0.88	±2.58	±0.45	±2.23	±5.89	±0.64
GPF-plus	53.62	64.89	58.02	54.13	54.02	65.89	68.75	54.41	50.79	58.28
±6.59	±0.89	±0.52	±1.38	±2.38	±0.54	±3.80	±5.85	±0.50
GCL	FT	44.06	48.47	51.91	56.10	48.13	53.93	32.63	55.41	49.44	48.89
±2.55	±1.63	±0.33	±0.45	±3.23	±0.91	±0.93	±1.28	±1.53
GPF	51.34	55.46	53.78	53.37	60.44	54.06	44.23	49.20	54.35	52.91
±1.01	±1.54	±0.58	±0.99	±4.10	±2.56	±0.75	±2.94	±0.65
GPF-plus	52.47	57.42	53.07	52.90	60.22	55.68	46.24	51.64	54.47	53.79
±1.19	±1.36	±0.81	±1.05	±3.81	±1.25	±3.35	±4.25	±1.25
exhibit significant advantages in terms of parameter efficiency compared to fine-tuning. It leads to reduced training time and storage space required for downstream adaptations.
Table 6: The number of tunable parameters for different tuning strategies.
Dataset	Tuning Strategy	Tunable Parameters	Relative Ratio (%)
Chemistry	FT	~ 1.8M	100
GPF	~ 0.3K	0.02
GPF-plus	~ 3-12K	0.17-0.68
Biology	FT	~ 2.7M	100
GPF	~ 0.3K	0.01
GPF-plus	~ 3-12K	0.11-0.44
B.5 Comparison with linear probing
In the field of Natural Language Processing and Computer Vision, linear probing [Kumar et al., 2022, Wu et al., 2020, Tripuraneni et al., 2020, Du et al., 2020] is a widely employed method for adapting pre-trained models to downstream tasks. This approach involves freezing the parameters of the pre-trained model  $f$  and solely optimizing the linear projection head  $\theta$ . To evaluate the effectiveness of linear probing, we conducted experiments on the chemistry datasets Toxcast and SIDER, and the results are summarized in Table 7. It is evident from the results that linear probing exhibits a significant performance decline compared to fine-tuning and our proposed graph prompt tuning. The primary distinction between linear probing and our proposed graph prompt tuning lies in the incorporation of an additional learnable graph prompt  $g_{\phi}(\cdot)$  in the input space. The substantial
Table 7: Test ROC-AUC (%) performance on Toxcast and SIDER with the linear probing.
Pre-training Strategy	Tuning Strategy	ToxCast	SIDER	Avg.
Infomax	FT	65.16 ±0.53	63.34 ±0.45	64.25
Linear Probing	63.84 ±0.10	59.62 ±0.73	61.73
GPF	66.10 ±0.53	66.17 ±0.81	66.13
GPF-plus	66.35 ±0.37	65.62 ±0.74	65.98
EdgePred	FT	66.29 ±0.45	64.35 ±0.78	65.32
Linear Probing	65.25 ±0.09	61.47 ±0.03	63.36
GPF	65.65 ±0.30	67.20 ±0.99	66.42
GPF-plus	65.94 ±0.31	67.51 ±0.59	66.72
AttrMasking	FT	65.34 ±0.30	66.77 ±0.13	66.05
Linear Probing	64.75 ±0.07	62.60 ±0.57	63.67
GPF	66.32 ±0.42	69.13 ±1.16	67.72
GPF-plus	66.58 ±0.13	68.65 ±0.72	67.61
ContextPred	FT	66.39 ±0.57	64.45 ±0.60	65.42
Linear Probing	65.35 ±0.09	61.28 ±0.39	63.31
GPF	67.92 ±0.35	66.18 ±0.46	67.05
GPF-plus	67.58 ±0.54	66.94 ±0.95	67.26
GCL	FT	62.54 ±0.26	60.63 ±1.26	61.58
Linear Probing	50.92 ±0.22	52.91 ±0.62	51.91
GPF	62.70 ±0.46	61.26 ±0.53	61.98
GPF-plus	62.76 ±0.75	62.37 ±0.38	62.56
performance gap observed between these approaches underscores the importance of integrating a graph prompt for the effective adaptation of pre-trained models.
B.6 Comparison with other tuning methods
We also compare our proposed graph prompt tuning with other tuning methods described as follows:
- PARTIAL- $k$ : We tune the last  $k$  layers of the pre-trained model  $f$  with a projection head  $\theta$  and freeze other parts, which is utilized in Zhang et al. [2016], He et al. [2021], Jia et al. [2022].
- MLP- $k$ : We freeze the pre-trained model  $f$  and utilize a multi-layer perceptron (MLP) with  $k$  layers as the projection head to perform the classification.
We conduct the experiments on the biology datasets (PPI), and Table 8 summarizes the results. The experimental results indicate that our methods outperform other tuning methods in all cases.
B.7 Extra results on GCC
Another popular pre-training strategy for graph contrastive learning involves following the training steps outlined in Qiu et al. [2020a]. First, we introduce the datasets utilized for pre-training GCC. The self-supervised pre-training task of GCC is conducted on six graph datasets, and Table 9 provides detailed statistics for each dataset. For the downstream tasks of the pre-trained GCC, we employ
Table 8: Test ROC-AUC (%) performance on protein function prediction benchmarks with different tuning methods.
Pre-training Strategy	FT	MLP-3	Partial-1	Partial-3	GPF	GPF-plus
Infomax	71.29 ±1.79	74.68 ±0.56	74.36 ±0.92	73.28 ±0.18	77.02 ±0.42	77.03 ±0.32
EdgePred	71.54 ±0.85	74.60 ±0.88	73.24 ±0.68	73.35 ±0.77	76.98 ±0.20	77.00 ±0.12
AttrMasking	73.93 ±1.17	77.99 ±0.42	75.91 ±0.10	74.02 ±0.37	78.91 ±0.25	78.90 ±0.11
ContextPred	72.10 ±1.94	76.01 ±0.68	76.62 ±0.92	74.86 ±0.79	77.42 ±0.07	77.71 ±0.21
IMDB-BINARY and IMDB-MULTI datasets [Yanardag and Vishwanathan, 2015]. Each dataset consists of a collection of graphs associated with specific target labels. We evaluate GPF on these datasets, and the results are presented in Table 10. The experimental findings consistently demonstrate that GPF outperforms fine-tuning when adapting models pre-trained using the GCC strategy.