Universal Prompt Tuning for Graph Neural Networks
Authors: Taoran Fang
Yunchao Mercer Zhang
Yang Yang
Chunping Wang
Lei CHEN
Venue: NeurIPS 2023 poster (2023)
Keywords: graph neural networks
prompt tuning
TL;DR: We propose a universal prompt-based tuning method that can be applied to any pre-trained GNN model.

Abstract: In recent years, prompt tuning has sparked a research surge in adapting pre-trained models. Unlike the unified pre-training strategy employed in the language field, the graph field exhibits diverse pre-training strategies, posing challenges in designing appropriate prompt-based tuning methods for graph neural networks. While some pioneering work has devised specialized prompting functions for models that employ edge prediction as their pre-training tasks, these methods are limited to specific pre-trained GNN models and lack broader applicability. In this paper, we introduce a universal prompt-based tuning method called Graph Prompt Feature (GPF) for pre-trained GNN models under any pre-training strategy. GPF operates on the input graph's feature space and can theoretically achieve an equivalent effect to any form of prompting function. Consequently, we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we employ GPF to obtain the prompted graph for the downstream task in an adaptive manner. We provide rigorous derivations to demonstrate the universality of GPF and make guarantee of its effectiveness. The experimental results under various pre-training strategies indicate that our method performs better than fine-tuning, with an average improvement of about 1.4% in full-shot scenarios and about 3.2% in few-shot scenarios. Moreover, our method significantly outperforms existing specialized prompt-based tuning methods when applied to models utilizing the pre-training strategy they specialize in. These numerous advantages position our method as a compelling alternative to fine-tuning for downstream adaptations.

Universal Prompt Tuning for Graph Neural Networks
Taoran Fang', Yunchao Zhang', Yang C Wang", Lei Chen?
'Zhejiang University, "Fin Volution Group {fangtr, 3190105622,, yangya)@zju........ edu. en, (wangchunping02, 040 com
Abstract
In recent years, prompt tuning has sparked research surge in adapting pre-trained models. Unlike the unified pre-training strategy employed in the language field, the graph field exhibits diverse pre-training strategies, posing challenges in designing appropriate prompt-base tuning methods for graph neural networks. While some pioncering work has devised specialized prompting functions for models that employ edge prediction as their vee-trainng tasks, these methods are limited to specific pre-trained GNN models and lack broader applicability. In this paper, we introduce a universal prompt-based tuning method called Graph Prompt Feature (GPF) for pre-trate pre-trained GNN models under any pre- training strategy. GPF operates on the input graph's feature space and can theoretically achieve an equivalent effect any form of prompting function. Consequently, we longer need illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we employ GPF obtain the prompted graph for the downstream task in an adaptive manner. We provide rigorous derivations to demonstrate the universality GPF and make guarantee of its effectiveness. The experimental results under various pre-training strategies indicate that our method performs better than fine- tuning. with an average improvement of about 1.4% in full-shot scenarios and about 3.2% in few-shot scenarios. Moreover, our method significantly outperforms existing specialized prompt- based tuning methods when applied models utilizing the pre-training strategy they specialize in. These numerous advantages position our method as a compelling alternative to fine-tuning for downstream adaptations. Our code is available at: https:: / //g thub. com/ st/zjjjuua unntt/CPF
Introduction
Graph neural networks (GNNs) have garnered significant attention from researchers due to their remarkable success in graph representation learning (Kipf and Welling 2017 Hamilton et al.) 2017 Xu et al. 2019]. However, two fundamental challenges hinder the large-scale practical applications of GNNs. One is the scarcity of labeled data in the real world [Zitnik et al. 2018 and the other is the low out-o--distribution generalization ability of the trained models Hu et al. 2020a Knyazev et al. 2019 Yehudai al. 2021 Morris et al.) 2019] To overcome these challenges, researchers have made substantial efforts in designing pre-tr GNN models (Xia et al.] 20226 Hu et al.) 2020a b. Lu al., [021] in recent years. Similar to the pre-trained models in the language field, pre-trained GNN models undergo training on extensive pre-training datasets and are subsequently adapted downstream tasks. Most existing pre-trained GNN models obey the "pre-train, fine-tune" learning strategy (Xu et al.) 2021a]. Specifically, we train a GNN model with a massive corpus of pre-training graphs, then we utilize the pre-trained GNN model as initialization and fine-tune the model parameters based on the specific downstream task.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Tuned
Downstream Tasks
Downstream Tasks
Downstream Tasks
Fruezn
Pretrained
Pretrained
Pretrained
GNN
GNN
GNN
Graph
Input
Specialized Gruph Template
Gruph Promut
I'euture
+
(a) Fine-tuning
(b) Specialized Graph Prompt Tuning
(c) Universal Graph Prompt Tuning
Figure I: Comparison of universal graph prompt tuning and existing approaches. (a) Fine-tuning updates the parameters of the pre-trained GNN model. (b) Existing specialized prompt- based tuning methods generate manual graph templates to adapt the models under certain pre-training strategies. (c) Our universal graph prompt tuning works on the feature space of the input graph. It can achieve an equivalent effect to any form of prompting function and be applied to any pre-trained GNN model.
However, the "pre-train, fine-tune" framework of pre-trained GNN models also presents several critical issues (Jin al., 2020]. First, there is a misalignment between the objectives of pre-training tasks and downstream tasks Liu et al. 2022a]. Most existing pre-trained models employ self- supervised tasks [Liu et al., 2021e such as edge prediction and attribute masking as the training targets during pre-training, while the downstream tasks involve graph or node classification. This disparity in objectives leads to sub-optimal performance [Liu et al. 2023]. Additionally, ensuring that the model retains its generalization ability is challenging. Pre-trained models may suffer from catastrophic forgetting (Zhou and Caol 202T] Liu et al.) [020] during downstream adaptation. This issue becomes particularly acute when the downstream data is small in scale, approaching the few- shot scenarios (Zhang et al., 2022]. The pre-trained model tends to over-fit the downstream data in such cases, rendering the pre-training process ineffective.
your question isn't getting the desired response, try rephrasing it." In recent years, a novel approach called prompt tuning has emerged as powerful method for downstream adaptation, addressing the aforementioned challenges. This technique has achieved significant success Natural Language Processing [i and Liang, [022 Lester et al. 2021a Liu et al. 2022a b) and Computer Vision [Bahng] al. 2022 Jia et al.) 2022]. Prompt tuning provides an alternative method for adapting pre-trained models to specific downstream tasks: it freezes the parameters of the pre- pre-trained model and modifies the input data. Unlike fine-tuning, prompt tuning diverges from tuning the parameters of the pre- trained model and instead focuses on adapting the data space by transforming the input.
Despite that, applying prompt tuning on pre-trained GNN models poses significant challenges and is far from straightforward. First, the diverse pre-training strategies employed on graphs make it difficult to design suitable prompting functions. Previous research [Liu et al., [022a] suggests that the prompting function should be closely aligned with the pre- pre-training strategy. For pre-trained language models, the typical pre-training tasks involve masked sentence completion [Brown et al. 2020]. In order to align with this task, we may modify a sentence like "I received gift" to " received a gift, and feel [Mask]" to make it closer to the task of sentence completion. However, in the case of graph pre-training, there is no unified pre-training task, making it challenging to design feasible prompting functions. Some pioneering studies (Sun et al., 2022]] Liu et al. [023] have applied prompt-based tuning methods to models pre-trained by edge prediction [Kipf' and Welling 2016b). They introduce virtual glass-prototype nodes/graphs with learnable links into the original graph, making the adaptation process more akin to edge prediction. However, these methods have limited applicability and are only compatible with specific models. When comes to more intricate pre-training strategies, it becomes challenging to design manual prompting functions in the same manner as employed for link prediction. Consequently, no prompt-based tuning method is available for models pre- pre-trained using alternative strategies, such attribute masking (Hu et al.) 2020a). Furthermore, existing prompt-based tuning methods for GNN models are predominantly designed based on intuition, lacking theoretical guarantees for their effectiveness.
In this paper, we address the aforementioned issues for graph prompt tuning. To deal with the diversity of graph pre-training strategies, we propose universal brsmdt- tuning method that can be

---

applied to the pre-trained GNN models that employ any pre-training strategy. Figure iillusrrate the distinction between our universal prompt-based tuning method and existing upproaches. Our solution, called Graph Prompt Feature (GPF), operates on the input graph's ss feature space and involves adding a shared learnable vector to node features in the graph. This approach is easily applicable to any GNN architecture. We rigorously demonstrate that GPF can achieve comparable results to any form of prompting function when applied to arbitrary pre-trained GNN models. Consequently, instead of explicitly illustrating the prompting function corresponding to each pre-training strategy, we adopt GPF to dynamically obtain the prompted graph for downstream tasks. We also introduce a theoretically stronger variant of GPF, named GPF-plus, for practical application, which incorporates different prompted features for different nodes in the graph. To guarantee the effectiveness of our proposed GPF and GPF-plus, we provide theoretical analyses to prove that GPF and GPF-plut are not weaker than full fine-tuning and can obtain better theoretical tuning results in some cases. Furthermore, we conduct extensive experiments validate the efficacy of our methods. Despite using a significantly smaller number of tunable parameters than fine-tuning, GPF and GPF-plus achieve better results across all pre-training strategies. For models pre-trained using edge prediction, GPF and GPF-plus exhibit a substantial performance advantage over existing specialized prompt-based tuning methods. Overall, the contributions of our work can be summarized as follows:
To the best of our knowledge, we present the first investigation of universal prompt- tuning methods for existing pre-trained GNN models. We propose GPF and its variant, GPF-plus, as novel approaches for universal graph prompt tuning. Our methods can be applied to the pre-trained GNN models that employ any pre-training strategy.
We provide theoretical guarantees for the effectiveness of GPF and GPF-plus. We demon- strate that GPF and GPF-plus can achieve an equivalent effect any prompting function and can obtain better tuning results in some cases compared to fine-tuning.
We conduct extensive experiments (both full-shot and few-shot scenarios) to validate the effectiveness of GPF and GPF-plus. The experimental results indicate that GPF and GPF- plus can perform better than fine-tuning, with an average improvement of about 1.4% in full-shot scenarios and about 3.2% in few-shot scenarios. Furthermore, GPF and GPF-plus significantly outperform existing prompt-based based tuning methods when applied to models that utilize the pre- training strategy they specialize in.
2 Related work
Pre-trained GNN Models Inspired by the remarkable achievements of pre-trained models in Natural
Language Processing (Qiu et al./ 2020b) and Computer Vision [Long et al. 2022). substantial efforts have been dedicated to pre-trained GNN models (PGMs) [Xia et al. 2022b in recent years. These methods utilize self-supervised strategies [Jin et al., 2020 to acquire meaningful representations from extensive pre-training graphs. GAE Kipf and Welling 2016a] first uses edge prediction as the objective task to train graph representations. Deep Graph Infomax (DGI) (Velickovic et al. 2019b] and InfoGraph [Sun et al. 2019] are proposed to garner nodes or graph representations by maximizing the mutual information between graph-level and substructure-
representations of different granularity. Hu et al. [2020a] employ attribute masking and context prediction pre-training tasks to predict molecular properties and protein functions. Both GROVER (Rong et al. 2020) and MGSSL. (Zhang al. 2021a] propose to predict the presence of the motifs or generate them with the consideration that rich domain knowledge of molecules hides in the motifs, Graph Contrastive Learning (GCL) is another widely adopted pre-training strategy for GNN models. GraphCL (You ct al. [020] and JOAO (You et al.) 2021] propose various augmentation strategies to generate different augmented views for contrastive learning. summary, there exists a diverse range of pre-training strategies for GNN models, each characterized by unique objectives.
Prompt-based Tuning Methods Prompt-based tuning methods, originating from Natural Language Processing, have been widely used to facilitate the adaptation of pre-trained language models to various downstream tasks (Liu et (0.122 Research has also explored the design of soft prompts to achieve optimal performance [Lester et al. 2021b Liu et al. [021c]. These methods freeze the parameters of the pre-train models and introduce additional learnable components in the input space,
thereby enhancing the compatibility between inputs and pre-trained models. Aside from the success of prompts the language field, the prompting methods are utilized other areas. Dia et al. 2022) and Bahng et al. 2022 investigate the efficacy of adapting large-scale models in the vision field by
3

---

modifying input images at the pixel level. In the realm of graph neural networks, the exploration of
prompt-based tuning methods still limited. Some pioneering work (Sun al. 2022 Liu et al. [203]] applies prompt-based tuning methods on the models pre-trained by edge prediction [Kipf and Welling 2016b]. These methods introduce virtual plass-prototype nodes/graphs with learnable links into the input graph, making the downstream adaptation more closely resemble edge prediction. However, these methods are specialized for models pre- pre-traine using edge prediction and cannot be applied to models trained with other strategies. We are the first to investigate the universal prompt-based tuning methods that can be applied to the GNN models under any pre-training strategy.
3 Methodology
We introduce graph prompt tuning for adapting pre-trained GNN models to downstream tasks. It is important to note that there are several types of downstream tasks in graph analysis, including node classification, link prediction, and graph classification. We first concentrate on the graph classification task and then extend our method to node-wise tasks. We define the notations in Section [.1]] then universal graph prompt tuning method in Section 3.3 and provide theoretical analyses in Section 3.4 Finally, we present the extension of our method to node-wise tasks (node classification and link prediction) in the appendix.
illustrate the process of graph prompt tuning in Section 3.2 We introduce
3.1 Preliminaries
Let G = (V,E) E represents graph, where V = {U1, V2..... u,,,..,,uu,... & C V V denote the node set and edge set respectively. The node features can be denoted as a matrix X = f ,,,..., E N where r; E R is the feature of the node 2/ and F is the dimensionality of node features. A E 10, 1 denotes the adjacency matrix, where Aij 1 if (v;, U3) E
.N)
Fine- Line-Tuning Pre-trained Models. Given a pre-trained GNN model f. learnable projection head I and downstream task dataset D = {G9, (1), v1),... .... (9m }m)}. }. we adjust the parameters of the pre-trained model / and the projection head to maximize the likelihood of predicting the correct labels of the downstream graph G:
max (v)((
1.0
3.2 Graph Prompt Tuning
Overall Process. Our proposed graph prompt tuning works on the input space by drawing on the design of the prompt tuning in the language field [Liu et al.) 2022a]. Given frozen pre-trained GNN model f. a learnable projection head 6, and downstream task dataset = {(), y1) w.)..... ., (9m. Fm) }- our target is to obtain a task-specific graph prompl 96 : G - G parameterized by Φ. The graph prompt (.).) transforms the input graph G into a specific prompted graph (g). And then (o) will replace G as input to the pre-trained GNN model f. During the downstream task training, we select the optimal parameters of é and 0 that maximize the likelihood of predicting the correct labels y without tuning the pre-trained model f. which can be formulated as:
max (v.o(ylga (9))
0.8
During the evaluation stage, the test graph Guss is first transformed by graph prompt go(+), and the resulting prompted graph g (Graa) is processed through the frozen GNN model f.
Practical Usage. In this part, we provide a detailed description of the refined process of graph prompt tuming, which comprises two fundamental steps: template design and prompt optimization.
A. Template Design. Given input graph G, we first generate a graph template G", which includes learnable components in its adjacency matrix A' and feature matrix X". Previous research has attributed the success of prompt tuning to bridging the gap between pre-training tasks and downstream tasks [Liu et al., 2022a]. Consequently. it implies that the specific form of the graph template is influenced by the pre-training strategy employed by the model. For specific pre- pre-training task E T and an input graph G, the graph template G* can be expressed as:
g*: (A*X")) = (9)
(3)

---

where the graph template g" may contain learnable parameters (i.e., tunable links or node features) in its adjacency matrix or feature matrix (similar to the inclusion of learnable soft prompts in a sentence), the candidate space for A is A. and the candidate space for X* is X.
B. Prompt Optimization. Once we have obtained the graph template g", next step is to search for the optimal Á and X within their respective candidate spaces A and X that maximize the likelihood of correctly predicting the labels using the pre-trained model f and a learnable projection head 0. This process can be expressed as:
Py.o(yl"")
max AEX,0
The graph & composed of Å and X can be considered as the the prompted graph go( mentioned in Formula
Practical Challenges. The specific form of the graph template closely tied to the pre-training
task t employed by the model f. However, designing the prompting function wb(()) is challenging
and varies for different pre- traininning tasks. Pioneering works (Sun et al.) 2022 Liu et al., 2023
have proposed corresponding prompting functions (-( for a specific pre- training strategy, with
focus on models pre-trained using edge prediction. However, many other pre-training strategies (Hu et al. 2020a Xia et al. 2022b), such attribute masking and context prediction, are widely utilized in existing pre-trained GNN models, yet no research has been conducted on designing prompting functions for these strategies. Furthermore, existing prompting functions are all intuitively designed, and these manual prompting functions lack guarantee of effectiveness. It raises a natural question: Can we design universal prompting method that can be applied to any pre-trained model, regardless of the underlying pre-training strategy?
3.3 Universal Graph Prompt Design
In this section, we introduce a universal prompting method and its variant. Drawing inspiration from the success of pixel-level Visual Prompt (VP) techniques (Bahng et al. [022 Wu et al. 2022 Xing et al. 2022 in Computer Vision, our methods introduce learnable components to the Ieature space of the input graph. In Section 3.4 we will demonstrate that these prompting methods can theoretically achieve an equivalent effect as any prompting function (-).
Graph Prompt Feature (GPF). GPF focuses on incorporating additional learnable parameters into the feature space of the input graph. Specifically, the learnable component is vector of dimension F, where F corresponds to the dimensionality of the node features. It can be denoted as:
p E R
(5)
The learnable vector p is added to the graph features X to generate the prompted features X*, which can be expressed as:
X {a1, 22, NND X
fa) + p, + ....., UN + p)
(6)
The prompted features X* replace the initial features X and are processed by the pre-trained model.
Graph Prompt Feature- (GPF-plus). Building upon GPF, we introduce variant called GPF-plus, which assigns an independent learnable vector p. to each node e; in the graph. It can be expressed as:
P1 PN F
(7)
(8) Similarly to GPF, the prompted features X* replace the initial features X and are processed by the pre-trained model. However, such a design is not universally suitable for all scenarios. For instance, when training graphs have different scales (i.e., varying node numbers), it is challenging to train such a series of p:- Additionally, when dealing with large-scale input graphs, such design requires substantial amount of storage resources due to its O(N) learnable parameters. To address these issues, we introduce an attention mechanism in the generation of p., making GPF-plus more parameter-efficient and capable of handling graphs with different scales. In practice, we train only k independent basis vectors p², which can be expressed as:
P2. .2...., X .....,,NNN = + P3 EN + pN}
pi.P2.... -p R
(9)
5

---

where k is for node U; we utilize attentive aggregation of these basis vectors with the assistance of & learnable linear projections a. The calculation process can be expressed as:
that can be adjusted based on the downstream dataset. To obtain p:
exp(a
Pi
=
P4.P3 b
aij = Σx Exp((aF
(10)
Subsequently, p; is used to generate the prompted feature X* as described in Formula
3.4 Theoretical Analysis
In this section, we provide theoretical analyses for our proposed GPF and GPF-plus. Our analyses are divided into two parts. First, we certify the universality of our methods. We demonstrate that our approaches can theoretically achieve results equivalent to any prompting function (hx It confirms the versatility and applicability of our methods across different pre-training strategies. Then, we make guarantee of the effectiveness of our proposed methods. Specifically, we demonstrate that our proposed graph prompt tuning is not weaker than full fine-tuning, which means that in certain scenarios, GPF and GPF-plus can achieve superior tuning results compared to fine-tuning. It is important to note that our derivations in the following sections are based on GPF, which adds a global extra vector p to all nodes in the graph. GPF-plus, being an extension of GPF and degenerates to GPF when the hyperparameter &s is set to 1. Therefore, the analyses discussed for GPF are also applicable to GPF-plus.
powerful version, can be seen as
Before we illustrate our conclusions, we first provide some preliminaries. For given pre-training task t E T and an input graph G: (A, X). we assume the existence of prompting function (-) that
generates a graph template g" : (A X*) = (9). The candidate space for A and X* is denoted as A and X, respectively.
Theorem 1. (Universal Capability of GPF) Given a pre-trained GNN model f. an input graph G: (A, X). an arbitrary prompting function (/e(),, for any prompted graph 9: (Å E A, e X) in the candidate space of the graph template G = (9),)), there exists a GPF extra feature vector p that satisfies:
f(A,
(11)
The complete proof of Theorem cann be found in the appendix. Theorem I Timplies that GPF can achieve the thcoretical performance upper bound of any prompting function described in Formula and 4 Specifically, if optimizing the graph template G* generated by a certain prompting function (+)()) can yield satisfactory graph representations, then theoretically, optimizing the vector p of GPF can also achieve the exact graph representations. This conclusion may initially appear counter- intuitive since GPF only adds learnable components node features without explicitly modifying the graph structure. The key lies in understanding that the feature matrix and the adjacency matrix A are not entirely independent during the processing. The impact of graph structure modifications on the final graph representations can also be obtained through appropriate modifications to the node features. Therefore, GPF and GPF-plus, by avoiding the explicit illustration of the prompting function (.).)) adopt simple yet effective architecture that enables them to possess universal capabilities in dealing with pre- trained GNN models under various pre- pre-training strategies.
Next, we make guarantce of the effectiveness of GPF and demonstrate that GPF is not weaker than fine-tuning, which means GPF can achieve better theoretical tuning results in certain situations compared to fine-tuning. In Natural Language Processing, the results obtained from fine- Euninu generally considered the upper bound for prompt tuning results [Lester et al.. 2021a Liu et al.) 2021b Ding et al., 2022). It is intuitive to believe that fine-tuning, which allows for more flexible and comprehensive parameter adjustments in the pre-trained model, can lead to better theoretical results during downstream adaptation. However, in the graph domain, the architecture of graph neural networks magnifies the impact of input space transformation some extent. To further illustrate this point, following previous work [Kumar et al. 2022 Tian et al.) 2023] Wei et al. 2021). we assume that the downstream task utilizes the squared regression loss 1 E, (9: 31)
the final representations to
Theorem 2. (Effectiveness Guarantee of GPF) For a pre-trained GNN model f, a series of graphs Xm). m, fmm) under the non-degeneracy condition, and
D [(G1 : (1), (Gm: (Ams

---

linear projection head 8, there exists y" = {v/,.... .......,uli Vm for 1/- = Vi..... Um = Vm that satisfies:
IGPF = min P.0
E(A(A,, + 0 y:)² y-)2 < lFT = min 5.0
(f(A,, X.) - y1)²
(12)
The detailed proof of Theorem 2 and the description of the degeneracy condition can be found in the appendix. Theorem 2 indicates that GPF obtains a lower minimum loss compared to fine-tuning in certain scenarios, demonstrating its ability to achieve better theoretical tuning results.
Experiments
4.1
Experiment Setup
Model Architecture and Datasets. We adopt the widely used 5-layer GIN et
et al. 2019 as the
underlying architecture for our models, which aligns with the majority of existing pre-trained GNN
models (Xia et al. 2022b Hu et al. 2020a Qiu et al.) 2020a You et al. 2020 Suresh et al.] [2021] Xu
al., 2021b Zhang et al. 2021b You et al. 2022, Xia
[022a]] As for the benchmark datasets,
we employ the chemistry and biology datasets published by Hu et al. 2020al A comprehensive
description of these datasets can be found in the appendix.
Pre-training Strategies. We employ five widely used strategies (tasks) to pre-train the GNN models, including Deep Graph Infomax (denoted by Infomax) |Velickovic al. 2019a]], Edge Prediction (denoted by EdgePred) (Kipf and Welling 2016a]. Attribute Masking (denoted by AttrMasking) (uu al., 2020a]. Context Prediction (denoted by ContextPred) (Hu al., [020al and Graph Contrastive Learning (denoted by GCL) You et al. 22200. A detailed description of these pre-training strategies can be found in the appendix.
We adopt the pre-trained models to downstream tasks with different tuning strategies. Given pre-trained GNN model f. a task-specific projection head 0,
Tuning Strategies.
Fine Tuning (denoted as FT). We tune the parameters of the pre-trained GNN model / and the projection head simultaneously during the downstream training stage.
Graph Prompt Feature (denoted GPF). We freeze the parameters of the pre-trained model f and introduce an extra learnable feature vector p into the feature space of the input graph described as Formula 6 We tune the parameters of the projection head 0 and feature vector p during the downstream training stage.
Graph Prompt Feature-Plus (denoted as GPF-plus). We freeze the parameters of pre- trained model f and introduce & learnable basis vectors pi. ....... Pg with learnable linear projections ......, ag to calculate the node-wise p; as Formula IO We tune the parameters of the projection head 0, basis vectors pf, ,.....,.. P. and linear projections a1.... dg during the downstream training stage.
......Pg
Implementation. We perform five rounds of experiments with different random seeds for each experimental setting and report the average results. The projection head 9 is selected from range of [1, 31-layer MLPs with equal widths. The hyper-paramete of GPF-plus is chosen from the range [5,10,20]. Further details on the haraeeparrmeter settings can be found in the appendix.
4.2 Main Results
We compare the downstream performance of models trained using different pre-training and tuning strategies, and the overall results are summarized in Table T Our systematic study suggests the following observations:
1. Our graph prompt tuning outperforms finettuning in most cases. Based on the results presented in Table [. is evident that GPF and GPF-plus achieve superior performance compared to fine-tuning in the majority of cases. Specifically, GPF outperforms fine-tuning in 28/36 experiments, while GPF. plus outperforms fine-tuning in 29/ /36 experiments. worth noting that the tunable parameters in GPF and GPF-plus are significantly fewer in magnitude than those in fine-tume fine-tuning (details can be found
7

---

Table Test ROC-AUC (%) performance on molecular prediction benchmarks and protein function prediction benchmarks with different pre-training strategies and different tuning strategies.
Pre-training Strategy
Tuning Strategy
BBBP
Tox21
ToxCast
SIDER
Tox
MUV
HIV
BACE
PPI
Avg.
67.55
78.57
65.16
63.34
70.06
81.42
77.71
81.32
71.29
FT
72.93
+2.06
#0.51
+0.53
+0.45
#1.45
+2.65
#0.45
#1.25
#1.79
Infomax
66.83
79.09
66.10
66.17
73.56
80.43
76.49
83.60
77.02
GPF
74.36
+0.86
=0.25
+0.53
+0.81
+3,94
+0.53
+0.18
=1.00
#0.42
67.17
79.13
66.35
65.62
75.12
81.33
77.73
83.67
77.03
GPP-
74.79
#0.36
=0.70
#0.37
#0.74
#2.45
+1.52
#1.14
#1.08
#0.32
66,33
78.28
65,34
66.77
74.46
81.78
77.90
80,94
73.93
FT
73.97
#0.55
#0.05
+0.30
#0.13
#2.82
+1.95
#0.15
#1.99
11.17
AttrMasking
68.09
79.04
66.32
69.13
75,06
82.17
78.86
84.33
78.91
GPF
75.76
+0.38
#0,90
#1.16
+1.02
+0.65
+1.42
+0.54
+0.25
67.71
78.87
66.58
68.65
76.17
81.12
78.13
85.76
78.90
GPP-
75.76
#0.64
#0.31
#0.13
+0.72
#2.98
1.332
1.12
#0.36
#0.
69.65
78.29
66.39
64.45
73.71
82.36
79.20
84.66
72.10
FT
74.53
+0.87
#0.44
#0.57
#0.6
+1.57
+1.22
#0.51
+0.84
11.94
ContextPred
68.48
79.99
67.92
66.18
74.51
84.34
78.62
85.32
77.42
GPF
75.86
+0.88
#0,24
#0.35
#0.46
#2.72
+0.255
1,,46
#00.4
+0.07
69.15
80.05
67.58
66.94
75.25
84.48
78.40
85,81
77.71
GPF-plus
76.15
#0.82
#0.46
#0.54
+0.95
#: 1..88
+0.78
#0.16
#0.43
#0.21
69.49
73.35
62.54
60.63
75.17
69.78
78.26
75,51
67.76
FT
70.27
+0.35
+0.70
#0.26
+1.26
+2, 14
+1.44
+0.73
#2.01
+0.78
GCL
71.11
73.64
62.70
61.26
72.06
70.09
75.52
78.55
67.60
GPF
70.28
+1.20
=0.25
#0.46
+0.53
+2.98
+0.67
#1.09
+0.56
#0.57
72.18
73.35
62.76
62.37
73.90
72.94
77.51
79.61
67.89
GPF-plus +0.93
71.39
#0.43
#0.75
+0.38
+2.47
+1.87
#0.82
#2.06
#0.69
Table 2:
Test ROC-AUC (%) performance on molecular prediction benchmarks and protein function prediction benchmarks for the models pre-trained by Edge Prediction.
Pre-training Strutegy
Tuning Strategy
BBBP
Tox21
ToxCast SIDER
ClinTox
MUV
HIV
BACE
PPI
Avg.
66.56
78.67
66.29
64.35
69.07
79.67
77.44
80,90
71.54
FT
72.72
+3.56
+0.35
10.45
20.78
14.61
#1.70
#0.58
#0.92
40.85
64.13
66.41
60.34
54.86
59.81
63.05
60.54
70.85
56.23
GPPT
61.80
EdgePred
40.14
10.04
10.14
#0.25
20.46
#0.34
20.54
#1.42
#0.27
69.43
78.91
64,86
60.94
62.15
82.06
73.19
70.31
76.85
GPPT (wfo ol)
70.97
#0.18
#0.15
#0.11
#0.18
#0.69
+0.53
#0.19
#f.99
#0.26
69.29
68.09
60,54
58.71
55.37
62.35
59.31
67.70
49.48
GraphPrompt
61.20
+0.19
#0.19
#0.21
#0.13
#0.57
#0.44
+0.93
#1.26
#0.96
69.57
79.74
65.65
67.20
69.49
82.86
77.60
81.57
76.98
GPF
74.51
#0.21
#0.03
+0.30
#0.99
+5.17
+0.23
+1.45
#1.08
+0,20
69.06
80.04
65.94
67.51
68.80
83.13
77.65
81.75
77.00
GPF-plus
74.54
#0.68
+0.06
#0.31
+0.59
#2.58
#0.42
#1.90
#2.09
#0.12
in the appendix). These experimental findings highlight the efficacy of our methods and demonstrate their capability to unleash the power of the pre-trained models.
GPF and GPF-plus exhibit universal capability across various pre-training strategies. GPF and GPF-plus present favorable tuning performance across all pre-training strategies examined in experiments, consistently surpassing the average results obtained from fine-tuning. Specifically, GPF achieves an average improvement of 1. 1.14%, while GPF-plus achieves average improvement of 1.60%. These results signify the universal capability of GPF and GPF-plus, enabling their application models trained with any pre-training strategy.
3. GPF-plus marginally outperforms GPF. Among the two graph prompt tuning methods, GPF-plus performs better than GPF in the majority of experiments (26/36). As discussed in Section 3.3 GPF-plus offers greater flexibility and expressiveness compared to GPF. The results further aftir that GPF- plus is enhanced version of graph prompt tuning, aligning with the theoretical analysis.
4.3 Comparison with Existing Graph Prompt-based Methods
We also conducted a comparative analysis between our proposed methods, GPF and GPF-plus, and existing graph prompt-based tuning approaches (Sun et al. 2022 Liu et al. 2023 Both of them
8

---

95
ROC-AUC(%%)
75
90
AAtraasking/// FT
-Contextrred/F
70
85
-AlMMaskingg GPF
ContextPred / GPF
AmrMaskingg GPP-plus
65
80
ContextPred GPF-plus
20
30
40
50
10
30
40
Steps
Steps
(a) Training Curve
(b) Test Curve
Figure 2: Training and test curves of different tuning methods.
specialize in tuning the models pre-trained by Edge Prediction (also known as Link Prediction). We apply GPPT [Sun et al. [2022], GPPT without orthogonal prompt constraint loss (denoted as GPPT (w/o ol)) [Sun et al. 2022 GraphPrompt [Liu et al. 2023 to the models pre-trained using Edge Prediction, and the results are summarized in Table 2 is worth mentioning that GPPT is originally designed for node classification tasks. Therefore, substituting class-prototy nodes with class-prototyy graphs to adapt it for graph classification tasks. The experimental results indicate that our proposed GPF and GPF-plus outperform existing graph prompt-based tuning methods by a significant margin. On the chemistry and biology benchmarks, GPF and GPF-plus achieve average improvements of 12%, 3%, and 13% over GPPT, GPPT (w/o ol), and GraphPrompt, respectively. These results showcase the ability of GPF and GPF-plus to achieve superior results compared to existing graph prompt-based tuning methods designed specifically for the pre-training strategy. Furthermore, it is worth highlighting that GPF and GPF-plus are the only two graph prompt-based tuning methods that surpass the performance of fine-tuning.
make minor modifications by
4.4 Additional Experiments
Few-shot graph classification. Prompt tuning has also been recognized for its effectiveness in addressing few-shot downstream tasks [Brown et al.) [2020] Schick and Schütze 22200b al Liu et 2021d 2023]. We evaluate the efficacy of our proposed methods in handling tew-shot scenarios. To conduct few-shot graph classification on the chemistry and biology datasets, we limit the number of training samples in the downstream tasks to 50 (compared to the original range of 1.2k to 72k training samples). The results are summarized in Table of the appendix. Compared the full-shot scenarios, our proposed graph prompt tuning demonstrates even more remarkable performance improvement (an average improvement of 2. 2.95% for GPF and 3.42% for GPF-plus) over fine-tuning in the few-shot scenarios. This finding indicates that our solutions retain higher degree of generalization ability in pre-trained models during few-shot downstream adaptations compared to fine-tuning.
Training process analysis. We conducted an analysis of the training process using different tuning methods on the biology datasets with the GNN models that employ Attribute Masking and Context Prediction their pre- pre-training tasks (Hu et al. 2020b) Figure 2 presents the training and test curves during the adaptation stage. From Figure 2 (a). it can be observed that the ROC-AUC scores of the training set consistently increase during the adaptation stage for both proposed methods and fine-tuning. However, from Figure (b). we can find that their behavior on the test set is quite distinct. For fine-tuning, the ROC-AUC scores the test set exhibit fluctuations and continuously decrease after an initial increase. On the other hand, when applying GPF or GPF-plus to adapt pre-trained models, the ROC-AUC scores the test set continue to grow and remain consistently high. These results indicate that fully fine-tuning pre-trained GNN model on a downstream task may lose the model's ' generalization ability. In contrast, employing our proposed graph prompt tuning methods can significantly alleviate this issue and maintain superior performance on the test set.
5
Conclusion
In this paper, we introduce universal prompt-based tuning method for pre-trained GNN models. Our method GPF and its variant GPF-plus operate on the feature space of the downstream input graph. GPF and GPF-plus can theoretically achieve an equivalent effect to any form of prompting function, meaning we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we can adaptively use GPF to obtain the prompted graph for downstream task adaptation. Compared to fine-tuning, the superiority of our method demonstrated both theoretically and empirically, making it compelling alternative for downstream adaptations.

---

6 Acknowledgements
This work was partially supported by Zhejiann NSF (LR22F020005), the National Key Research and Development Project of China (2018AA40101900), and the Fundamental Research Funds for the Central Universities.
References
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. 2022.
Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Bala- murugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks.
ICLR, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu. Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurIPS, 2020.
Leonardo Cotta, Christopher Morris, and Bruno Ribeiro. Reconstruction for powerful graph repre- sentations. NeurIPS, 2021.
Ning Ding, Yujia Qin, Guang Yang. Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang. Juan Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv, abs/2203.0690 2022.
Simon Shaolei Du, Wei Hu, Sham M. Kakade, J. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. ICLR, 2020.
Fabrizio Frasca, Beatrice Bevilacqua. Michael Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. NeurlPS, 2022.
Anna Gaulton, Louisa J. Bellis, A. Patrícia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington. Chembl: a large-scale bioactivity database for drug discovery. Nucleic Acids Research, 40:D1100- D1107, 2012.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurlPS, pages 1034-1034, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'a ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. ArXiv, abs/2111.06311 2021.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Perey Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. ICLR, 2020a.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020b.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. ECCV, 2022.
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self- supervised learning on graphs: Deep insights and new direction. ArXiv, abs/2006.10141, 2020.
Thomas Kipf and Max Welling. Variational graph auto-encoders. ArXiv, 003//611..0700, 2016a.
Thomas Kipf and Max Welling. Variational graph auto-encoders. In ArXiv, volume abs/ abs/161 1.07308. 2016b.
10

---

Thomas N Kipf and Max Welling. Semi-supervise classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. In Neural Information Processing Systems, 2019.
Boris Knyazev, Graham W. Taylor, and Mohamed R. Amer. Understanding attention and generaliza- tion in graph neural networks. In NeurlPS, 2019.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. ICLR, 2022.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP, 2021a.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint 1rXiv:21040086/5 2021b.
Xiang Lisa Li and Perey Liang. Prefix-tuming Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1tth International Joint Conference on Natural Language Processing (Volume I: Long Papers), 02//21010001999 2021.
Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In AAAI Conference on Artificial Intelligence, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13:: 13586, 2021a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys (CSUR), 2022a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tumine tne-tuning universally across scales and tasks. ArXiv, abs/2110.07602. 2021b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 1 2021c.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. ArXiv, 00n//213..1085, 2021d.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL, 2022b.
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering, 35:5879-5900 2021e.
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. Proceedings of the ACM Web Conference 2023, 2023.
Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqing Yang. Vision-and-language pretrained models: A survey. IJCAI, 2022.
Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi. Learning to pre-train graph neural networks. In AAAI, 2021.

---

Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin N. Steijaert, Jörg Kurt Wegner, Hugo Ceulemans, Djork-Ami Améé Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction on chemblt felectronic supplementary information (esi) available: Overview, data collection and clustering, methods, results, appendix. see doi: 10.1039/c8sc001450 Chemical Science, 9:5441 5451, 2018.
Diego Mesquita, Amauri H. de Souza, and Samuel Kaski. Rethinking pooling in graph neural networks. ArXiv, abs/2010.1 1418, 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. AAAI, 2019.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang. Ming Ding, Kuansan Wang. and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020a.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63:1872
1897, 2020b.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer large-scale molecular data, Advances Neural Information Processing Systems, 13:1955111 2020.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Conference of the European Chapter of the Association for Compu- tational Linguistics, 2020a.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. NAACL, 2020b.
T. Sterling and John J. Irwin. Zinc 15 ligand discovery for everyone. Journal of Chemical Information and Modeling, 55:2324 2337, 2015.
Fan- Run Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908. 01000, 2019.
Mingchen Sun, Kaixiong Zhou, Xingbo He, Ying Wang, and Xin Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In Neural Information Processing Systems, 2021.
Junjiao Tian, Xiaoliang Dai, Yao Ma. Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tunin fine-tuning, ArXiv, 00b//303..10200 2023.
Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. NeurlPS, 2020.
Petar Velickovic, William Fedus, William L. Hamilton, Pietro Lio', Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. ICLR, 2019a.
Petar Velickovic, William Fedus, William L. Hamilton, Pietro Lio', Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. ICLR, 2019b.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. ArXiv, abss22106.09226, 2021.
Junyang WU, Xianhang Li, Chen Wei, Huiyu Wang, Alan Loddon Yuille, Yuyin Zhou, and Cihang Xie. Unleashing the power of visual prompting the pixel level. ArXiv, abs/22 10556, 2022.
12

---

Sen Wu, Hongyang Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. ICLR, 2020.
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: A benchmark for molecular machine learning. arXiv: Learning, 2017.
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z. Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. Proceedings of the ACM Web Conference 2022, 2022a.
Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z. Li. A survey of pretraining on graphs: Taxonomy, methods, and applications. arXiv preprint 2xXiv:202.007893, 2022b.
Yinghui Xing, Qirui Wu, De Cheng. Shizhou Zhang, Guoqiang Liang. and Yanning Zhang. Class- aware visual prompt tuning for vision-language pre-trained model. ArXiv, abs/2208.08340, 2022.
Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning under the pre-train and fine-tune paradigm. NAACL, 2021a
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
Minghao Xu, Hang Wang, Binghing Ni, Hongyu Guo, and Jian Tang. Supe-supe graph-level representation learning with local and global structure. In International Conference on Machine Learning, 2021
Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kernlls. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015.
Gilad Yehudai, Ethan Fetaya, Eli A. Meirom, Gal Chechik, and Haggai Maron. From local structures to size generalization in graph neural networks. In ICML, 2021.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. NeurlPS, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121- 121322 PMLR, 2021.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Bringing your own view: Graph contrastive leamning without prefabricated data augmentations. Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, 2022.
Chuxu Zhang, Kaize Ding, Jundong Li, Xiangliang Zhang, Yanfang Ye, N. Chawla, and Huan Liu. Few-shot learning on graphs. International Joint Conference on Artificial Intelligence, 2022.
Muhan Zhang and Pan Li. Nested graph neural networks. NeurIPS, 2021.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, 2016.
Z. Zhang, Q. Liu, H. Wang, C. Lu, and C. K. Lee. Motif-based graph self-supervised learning for molecular property prediction. 2021a.
Zaixin Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self- supervised learning for molecular property prediction. In Neural Information Processing Systems, 2021b.
Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. ICLR, 2022.
Fan Zhou and Chengtai Cao. Overcoming catastrophic forgetting in graph neural networks with experience replay. In AAAI Conference on Artificial Intelligence, 2021.
Marinka Zitnik, Rok Sosi, and Jure Leskovec. Prioritizing network communities. Nature Communi- cations, 9, 2018.
13

---

A Extra Materials for Section 3
A.1 Extension to node-wise tasks
In this section, we illustrate the process of graph prompt tuning for node-wise tasks (node classification and link prediction). These tasks differ from graph classification, requiring node-level representations instead of graph-level representations. To bridge this gap, we employ Subgraph GNNs [Cotta et al. 2021, Zhang and Li 2021 Bevilaequa et al. 2022 Zhao et al. [022] capture the graph-level and node-level representations. Subgraph GNN models utilize MPNNs on sets of subgraphs extracted from the original input graph. They subsequently aggregate the resulting representations [Frasca et al., 2022]. As result, the node representations can also be interpreted as graph representations of the induced subgraphs. Specifically,, the node representation h2 for node vs can be calculated as:
ly = (())
(13)
where f is Subgraph GNN model, and G. is the induced subgraph for node To obtain the node representations through graph prompt tuning, we freeze the model f and introduce a learnable graph prompt Is: G + G. parameterized by o, as indicated in Formula 2, which can be expressed as:
h; =f (G:)
(14)
Once we acquire the node representations. we can seamlessly proceed with downstream node classification or link prediction tasks. Our proposed methods, GPF and GPF-plus, enable the effective execution of node-level tasks through the approach described in Formula
A.2 Proof for Theorem
Theorem 1. (Universal Capability of GPF) Given pre-trained GNN model f. an input graph G: (A, X), arbitrary prompting function (-)))), for any prompted graph G: (Å E A, X) in the candidate space of the graph template G = (9), there exists a GPF extra feature vector p that satisfies:
=
where (G) = g*: XA", A and X are the candidate space for A and X respectively.
To illustrate Theorem we provide the specific architecture of the pre-trained GNN model f. For analytical simplicity, we initially assume that f is a single-layer GIN [Xu et al., 2019] with linear transformation. Subsequently. we extend our conclusions to multi-layer models utilizing various transition matrices [Klicpera et al. 2019]. During the generation of graph representations, we first obtain node representations and then employ a readout function to calculate the final graph representations. Existing pre-trained GNN models commonly employ sum mean pooling for this purpose. Previous research [Mesquita et al. 2020)] suggests that complex pooling mechanisms are unnecessary, as simple readout functions can yield superior performance. It is worth mentioning that the subsequent derivations still hold when we use any other weighted aggregation readout functions, such as average pooling, min/max pooling, and hierarchical pooling. Hence, we assume that the concrete architecture of the pre-trained GNN model ʃ can expressed as:
H = (A + (1+ e) -X. W
(15)
hg = Σ h4
(16)
where W is a linear projection. The parameters and W have been pre-trained in advance and remain fixed during downstream adaptation. Next, we proceed with the derivation of Theorem 1. In Theorem 1, we impose no constraints on the form of the prompting function ()x(),, allowing Å and X to represent the adjacency matrix and feature matrix of any graph. We define the graph- level transformation 9: G + G, which satisfies (Ā 9 X). Consequently, Theorem is cquivalent to Proposition
Proposition 1. Given a pre-trained GNN model an input graph G: (A, X), for any graph-level transformation g: G > G, there exists a GPF extra feature vector P that satisfies:
f(A.X + f(g(A,X))
(17)
14

---

To further illustrate the graph-level transformation g(-), we decompose it into several specific transformations.
Proposition 2. Given input graph G = (), arbitrary graph-level transformation g: G can be decoupled to a series of following transformations:
Feature transformations. Modifying the node features and generating the new feature matrix X' (x)
Link transformations. Adding or removing edges and generating the new adjacency matrix
A' = [A)
Isolated component transformations. Adding or removing isolated components (sub-graphs) and generating the new adjacency matrix and feature matrix A', X' +Jict (A, A.X, X).
Here, the word "isolated" refers to a component (sub-graph) that does not link with the rest of the graph. Proposition 2 suggests that arbitrary graph-level transformation is a combination of the three transformations mentioned above. For instance, the deletion of a node in the imitial graph can be decomposed into two steps: removing its connected edges and then removing the isolated node.
Proposition 3. Given a pre-trained GNN model f, an impul graph G: (A, X), for any feature transformation Oft 9fo (X) X', there exists a GPF extra feature vector that satisfies:
+ p) J(A,X
[(A,X)
(18)
Proof. We set AX = X' - X. Then, we have:
H' (A (1 c) I) X W
(19)
(A + (1+c) I) (X + AX). W
(20)
(A+ I)- X W + (A + (1+c) 1) AX W
(21)
+1++(1+) I) A W
(22)
For GPF p = [a1,
.ar] IxF, we can perform similar split:
Hp = (A+ - (1 e) 1) (X+ [ p) W
(23)
(A+(1+e)))
I) X W (A + (1 +e) +e)-I) [ p W
(24)
-H+ (A (A+ (1+ I) [1] .p .p.W W
(25)
= -H+ + (4: +1+ .p. W
(26)
where [1]N E N denotes column vector with N 1s, [d, + +1+ c| E NR1 denotes a column vector with the value of i-th row is de + 1 + and de represents the degree number of To obtain the same graph representation hg, we have:
hg.ft = hg.p - Sum(H') = Sum(H,)
(27)
where SMm(M) = Σ, my denotes the operation that calculates the sum vector for each row in the matrix. We can further simplify the above equation as:
hg.ft = hg.p
(28)
+ Sum H') = Sum (Hp)
(29)
+ Sum H + (A + (1 + - I) AX W) = Sum + (d, + 1+ e W)
(30)
Sum((A + c) I) AX W) = Sum([4 +1 <N .p. W)
(31)
where the results of ((A + (1 + I) AX) E NxF, the frozen linear transformation
W E FxF" We first calculate Ahg.p = Sum [d. + + 11 M p W) FF. We can obtain that:
F N
Ahg
dy +1 +1+ 0j W
(32)
k=
Σ
D DD+ ND+N+N- W
(33)
j=1
15

---

where heg denotes the value of the i-th dimension in hg.p. D D= degree of all nodes in the whole graph. Øj.J [1, F] denotes the j-th learnable parameter in GPF p. and W J.a= E [1, F), E [1, F"] denotes the frozen parameter W. As for Ahg.
-= d2 denotes the total
Sum( (1 +) I) AX W), we assume (A+ -I AX
B RNP., Then we have:
N Beg) W
Ahg. =
2 J=1
(34)
where Bujjk k E NJ, E [1, P] denotes the learnable parameter in B. According to above analysis,
to obtain same graph representation lig,s with a certain hig.tt, we have:
hegp nig.ee for every i E [1, FM
(35)
+ Ahg Ahg
(36)
Σn=1 BKG oj N
j [1
(37)
Therefore, for an arbitrary feature transformation 9ft, there exists GPF p that satisfies the above conditions and can obtain the exact graph representtiinn for the pre-trained GNN model f.
n
Proposition 3 demonstrates the comprehensive coverage of proposed GPF for all graph-level feature transformations. GPF introduces a uniform feature modification p E IR to each node in the graph. However, it can achieve an equivalent effect to adding independent feature modifications to each node individually under the pre-trained GNN model described above.
Proposition 4. Given a pre-trained GNN model f, an input graph G: (A, for any link transfor- mation que gu (A) A', there exists GPF extra feature vector P that satisfies:
J(A,X + p) = ,(A,,X)
(38)
Proof. The proof of Proposition 4 is similar that of Proposition We AA A'- A. It is worth mentioning that A, A' E {0, 1 and AA E (-1, 0, 1 which means they are of the same size, the values of A, A' can only be or and the values of AA can be -1, 1. We have:
H' (A' (1 e) X.
(39)
((A + AA) + (1 +6) I). W.
(40)
H AA X W
(41)
From the proof of Proposition
we obtain:
H, = H + [d, + +1+ p W
(42)
where p- [a1 ar) E RI denotes our learnable GPF, [d, + + d. RN1 denotes
column vector with the value of i-th line is d, 11 and d, represents the degree number of vi-
With Ahg.p = Sum((d, + +
k..We RF., we can obtain:
F N
Ahg.o
(d +1+6) W
(43)
1=
(D+N+N-C).0g W
(44)
1=1
where higp denotes the value of the i-th dimension in hg.p. D D= EN dy denotes the total degree of all nodes in the whole graph, Jj.j j E [1, F] denotes the j-th learnable parameter in GPF and
W J. E [1, F), [1, F'] denotes the frozen parameter in W. As for Ahg. Sum(A A X. W),
we have:
A Ahg.u
Σ
Σ (A.D)ENN
(AKKI-31.) W
(45)
16

---

where A k E [1, 2 E [1 denotes the element of AA, and zı,, [1, e F]
denotes the element of X. To obtain same graph representation higs with certain hg.u we have:
1ig lig.t. for every E [1,
(46)
A Ahgg,e
(47)
EKKNENNN A Fi
0 D
- [1,
(48)
Therefore, for an arbitrary link transformation Str, there exists a GPF P that satisfies above conditions and can obtain the same graph representation for pre-trained GNN model f.
Proposition 4 demonstrates that GPF can also encompass all link transformations. Intuitively, link transformations are closely associated with changes in the adjacency matrix, which are independent of node features. However, our findings reveal that, within the architecture of most existing GNN models, modifications in the feature space and modifications in the structural space can produce equivalent effects.
Proposition 5. Given pre-trained GNN model f. an input graph 9: (A,X), for any isolated component transformation Giot giot A, X) A', X', there exists GPF extra feature vector p that satisfies:
(49)
Proof. Unlike feature transformations and linear transformations, isolated component transformations will change the number of nodes the graph, which means the scale of modified A' and X' is uncertain. We first express the isolated component transformation in more details. The adjacency matrix and feature matrix X can be divided into several isolated components, which can be expressed as:
A1 0
0
X X
A
X
(50)
Am
Removing isolated component Cg (Ax, Xx), E [1.m] means removing both A in the adjacency matrix and corresponding X₃ in the feature matrix. Adding a new isolated component Cm+++ (Am+i, X (Am+ 1 means adding Am+t the adjacency matrix A, and adding X to the corresponding position of X. Then we have:
hg. = Σ Sum((Au + (1+ Xx W)
(51)
To align with the proofs of Proposition and Proposition we set Ahg,ist = hgg Sum((A c)+ I)- X W), and it can be expressed as:
+
Anggist Σ (um (1+6) Xx W)
(52)
where 1 is an indicator that satisfies:
O if Cs has no change
4
1 if Cj is additional component
(53)
if C₁ is removed component
From the proof of Proposition
3 we have following conclusions:
H H [d. + + cN p W
(54)
where p (o aF) RIXF denotes our learnable GPF, (d, 1 + c) Rx1 denotes
column vector with the value of i-th line is d, + 1 and d, represents the degree number of 2
17

---

With A = Sum( [d; + + te W) R we can obtain:
Anigg
dh +1++) ji
(55)
j=
F (D Wj
(56)
where h denotes the value of the i-th dimension in hg.p. D D= E de denotes the total degree of all nodes in the whole graph, az. E [1, F] denotes the j-th learnable parameter in GPF p. and W J,a.J E [1, F). E [1, F") denotes the frozen parameter in W. To obtain a same graph representation hg.p with a certain hggist, we have:
h/g p isg for every i E [1,
(57)
A Ahi
(58)
0 Σx Ix Sum((A& + (1+ x
[1
(59)
D N + N.c
where X denotes the j-th column of the matrix X. Therefore, for an arbitrary isolated component transformation giet. there exists GPF that satisfies above conditions and can obtain the same graph representation for pre-trained GNN model f.
The isolated component transformation possesses the capability to alter the scale of a graph, and previous studies have paid limited attention this type of graph-level transformation.
Proposition 6. Given a pre-trained GNN model f. an input graph G: (A,X), for a series of transformations g = (g1,92, 02,. .. } composed of gsr, gie and gists there exists a GPF extra feature vector P that satisfies:
(60)
Proof. Without loss of generality, we consider g (91, 92} with two transformations described in Proposition 2
Now prove there exists a GPF p that satisfies:
f(A, X + p) = (92((1) (A,,
(61)
We assume gi (A,X) X) = A', X According to Proposition 3
there exists a GPF أI that satisfies:
f(A,X
= = (A,
(62)
and fa that satisfies:
f(A,X xx)+ f(92(A',
(63)
Therefore, there is a a==pp P1 + P2 that satisfies:
f(A, + = f0z(01 (o,
(64)
Based on the preceding analysis, we have established that the GPF can replicate the effects of any graph-level transformation the pre-trained GNN model defined by Formula 15 and and Hence, we
have successfully demonstrated Proposition and Theorem within the framework the simple model architecture. Next, we aim generalize our findings to more intricate scenarios.
Extension to other GNN backbones. We use GIN Xu et al. [019 as the default backbone model in our previous derivation. When replacing GIN with another GNN model, only slight modifications are required to ensure that all propositions remain valid. As is described in Klicpera et al. (2019). various GNN models can be represented as:
-
(65)
18

---

where S E RNN denotes the diffusion matrix (e.g., A + (1 +) I the diffusion matrix for GIN),
and W denotes the linear projection. In this case, we modify the Formula 26 42 and [4 as follows:
H = H + d; + +1
W st. p W
(66)
H = H
(67)
where st, is the element of the diffusion matrix S. With these modifications, Propositions 3 and remain valid.
Extension to multi-layer models. For analytical simplification, similar to many previous works, we consider multi-layer GNN models without non-lineat activation functions between layers, which can be expressed
(()) X
(68)
H (A) = (M) H(K-1) (())
(69)
where S(k) the diffusion matrix described as Formula 65 of the k-th layer, and W (k) is the linear projection of the k-th layer. With such architecture, the node representations of the k-th layer (k) can also be obtained as:
(R)
(
w W
(70)
where S' = IT i= (t) and W W (i) By substituting S = S' and W W Formula 65
can find that Proposition 3 and still hold true.
A.3 Proof for Theorem
Theorem 2. (Effectiiveness Guarantee of GPF) For a pre-trained GNN model f. a series of graphs
D = {(G: (A1, 11), (.)...)..... (9m; (Am M,), ymm) under the non-degeneracy condition. and
linear projection head 0, there exists y {y 3m for y1
V .i...., Ym V'm that satisfies:
m2 min p.8
ICFF
Σ
+ p) ---)²² IFT min 5
E(((A.
9
To demonstrate Theorem 2, we need to provide more detailed description of the architecture of the GNN model f. We assume that the graph representations hG, for G. are obtained through the following process:
H. = S X, hg, Sum(uuHH
(71)
(72)
where S; E R...NN. denotes the diffusion matrix of G; as Formula 65 N; denotes the node number of the graph Gj, W E FxF denotes the linear projection, F is the dimension of node features, and Sum MM) Σ denotes the operation that calculates the sum vector for each row in the matrix M. When we employ our proposed GPF in the above model f. the graph representations & are calculated as:
HGPF S, (X; p)
p) W
(73)
GPF
Sum
(74)
where [1]T is column vector with all I's, and p E R the extra learnable vector of GPF. The squared regression loss can be expressed as:
=
Σ (eg 0 y1)²
(75)
19

---

where 8 E IR'x is linear projection head. In such case, the optimal tuning loss of fine-tuning 4FT and GPF 1GPF can be expressed as:
4PT
min W.
(Sum(S X, W) W y.)²
(76)
m
|GPF min p.o
(Sum(S, (X; + [1]" p) W) W0-0- - y;) y;)²
(77)
Before we prove Theorem 2, we first illustrate the following proposition.
Proposition 7. Given a series of graphs D {91 (A1, (r,x...).. .......... 9m (Am: Xm)} and linear
projection W, there exist P TR. FR, θ IRFF' Fxx1, x1, and 6 > that satisfy:
m
Σ [S (X.ISSS--X+++ [XJ+ -) W) 0 - Sum(S, X; w') 0 >
(78)
for any W E FFFF 0' E
For analytical simplification, we gather all the unique node features in D into set Set Sx (- . Consequently, the node feature matrix X; of any graph Gi can be constructed from elements in the set SX. Next, we proceed to expand the function Sum(.) as:
N. N,
hg. =Sum(S; X; W)
ΣΣ
S S.UAA) X.(1.)) W
(79)
ng.
=Sum(S, (X, + [1]T P) -W)
N >Σ>+ S.GAA) X.(k)))
Ni
N N.
W
ΣΣ S S.G.)) P W
(80)
where S..(jk) denotes the element the j-th row and k-th column S, and (x..))) denotes the k-th row vector of X, In order to express the representations of different graphs in unified form, we rewrite the above formulas to calculate the graph representations he, and hg, representations in Sx:
GPF from the node
N: S.G.A) X((RA)) W
hg.
Σ
Ca. Ij W
(81)
N N
N. N.
GPF hgg.
S..(.A) X.(Ka)) W +
Σ
S (i.(.)) p W
k
à
Σ C W + Σ P.
(82)
where Gyy is the coefficient of zy for the graph Gi which is calculated as G.j = Et Ex
S. J)
for all k's that satisfy (.,kk.) = Ty. We can also get that Ej CrS EE E,²: S. rewrite Formula into matrix form as follows:
.)) Then, we
(C Xsx + Caall" P) W. - C Xsx W 0 0'122> 8
(83)
where Xsx E R denotes the feature matrix for Sx that satisfies Xsx (1-) = Tyj. C E mxx denotes the coefficient matrix and the element in the i-th and k-th column C(t.) is equal to G.ji EE, Ca denotes column vector with the value of i-th row is E, C(a))- We represent
W'. 0' = w. 0 + AW 0, then we have:
II(C Xsx Eaanl () W. Ó- C. XX (W 0 0 AW 0)
(84)
= -11Σ Ca P W.W.W -C- Xsx AW. 0|1
(85)
20

---

As described by the condition of Proposition W' and 6' can be chosen arbitrarily, which means AW- f can be equal any u RPx1 Therefore, Proposition can be reformed below.
Given a series of graphs D {91: (A1, x.), 9m: (Am: Xm)} and a linear projection W, there exist p E RRIF,, 0 E R FRFF* x1, and > that satisfy:
1 Casil' P. W -C- C. Xsx 0112> 28
(86)
for any E ERI.
We make the assumption that C is column full-rank matrix, which implies that there is uniform feature distribution shared among different graphs, aligning with real-world scenarios. is important to note that W is pre-traeieee pre-trained beforehand, and for any p and 0 satisfying P W 0 & O, the non- degeneracy condition for Formula 86 is as follows:
ΣΣ Canl D. W. -C Xsx u/12 #0
(87)
Xsx = [1] has no solution v/
(88)
where [1]T E [ is column vector with all I's. Therefore, Formula 86 and Proposition 7 hold for all X5x for which X5x v = [1]7 has no solution v'.
Finally, we revisit Theorem 2. Given a series of graphs under the non-degeneracy condition, a pre-
trained linear projection W, 9 and 0 that satisfy P W 0 o O, we can construct y" = .. yn as:
J/ (a.)) Xsx W. W.
+CC P W.O
(89)
Under these conditions, the optimal theoretical tuning results for GPF and fine-tuning can be expressed
as::
lGFF = min |I(C Xsx + 1 p) - (C- + Ecanl p) w. 0112 =
p.u
j
(90)
IF = min W
C Xxx W. (C Xxx + Cant P) W 0112 >
(91)
Here, loPF can be achieved with á and = 9, while ler consistently remains greater than 0, as stated in Proposition 7 Theorem 2.
Therefore, can conclude that lopp lpT, thus establishing the proof of
21

---

B More Information on Experiments
B.1 Details of the datasets
Dataset overview We utilize the datasets provided by Hu et al. 2020a] for our pre-training datasets. These datasets consist of two domains: chemistry and biology. The chemistry domain dataset comprises 2 million unlabeled molecules sampled from the ZINC15 database (Sterling and Irwin 2015], along with 256K labeled molecules obtained from the preprocessed ChEMBL dataset [Mayr al. 2018 Gaulton et al., 2012]. On the other hand, the biology domain dataset includes 395K unlabeled protein ego-networks and 88K labeled protein ego-networks extracted from PPI networks. For the models pre-trained on the chemistry dataset,
employ eight binary graph classification datasets available in MoleculeNet (Wu et al. 2017] as downstream tasks. As for the models pre- trained on the biology dataset, apply the pre- trained models 40 binary classification tasks, with each task involving the prediction of a specific fine-grained biological function.
Pre-training datasets. The datasets provided by Hu al. [2200a] consist of two distinct datasets: Biology and Chemistry, corresponding to the biology domain and chemistry domain, respectively. The Biology dataset contains 395K unlabeled protein ego- networks obtained from PPI networks of 50 species. These networks are used for node-level self-supervised pre-training. Additionally, 88K labeled protein ego networks serve as the training data for predicting 5000 coarse-grained biological functions. This graph-level multi-task supervised pre-training aims to predict these functions jointly. Regarding the Chemistry dataset, it comprises 2 million unlabeled molecules sampled from the ZINC15 database (Sterling and Irwin 20155] These molecules are utilized for node-level self- supervised pre-traie pre-training. For graph-level multi-task supervised pre-training, a preprocessed ChEMBL dataset (Mayr et al. 2018 Gaulton et al., 2012] is employed. This dataset contains 456K molecules and covers 1310 different biochemical assays.
Downstream datasets. The statistics of the downstream datasets utilized for the models pre-trained on Biology and Chemistry are presented in Table 3
Table 3: Statistics of datasets for downstream tasks.
Dataset
BBBP
Tox21
ToxCast
SIDER
Clin'Tox
MUV
HIV
BACE
PPI
Proteins / Molecules # Binary prediction tasks
2039
7831
8575
1427
1478
93087
41127
1513
88K
12
617
27
17
1
1
40
B.2 Details of pre-training strategies
We adopt five widely used strategies (tasks) to pre-train the GNN models, which are listed as below:
Deep Graph Infomax (denoted by Infomax). It is first proposed by Velickovic et al. [019a]] Deep graph infomax obtains expressive representations for graphs nodes via maximizing the mutual information between graph-level representations and substructure-level represen- tations of different granularity.
Edge Prediction (denoted by EdgePred). It is regular graph reconstruction task used by many models, such as GAE (Kipf and Welling 2016a]. The prediction target is the existence of edge between pair of nodes.
Auribute Masking (denoted by AttrMasking). It is proposed by Hu et al. 2020a masks node/edge attributes and then let GNNs predict those attributes based on neighboring structure.
Context Prediction (denoted by ContextPred). It is also proposed by Hu et al. [2020a]. Context prediction uses subgraphs predict their surrounding graph structures, and aims to mapping nodes appearing in similar structural contexts to nearby embeddings.
Graph Contrastive Learning (denoted by GCL). embeds augmented versions of the anchor close to each other (positive samples) and pushes the embeddings of other samples (negatives) apart. We use the augmentation strategies proposed You et al. 2020 for generating the positive and negative samples.
22

---

To pre-train our models, we follow the training steps outlined in Hu et al. [200aa] for Infomax, EdgePred, AttrMasking, and ContextPred tasks. We then perform supervised graph-level property prediction to enhance the performance of the pre-trained models further. For models pre-trained using GCI,, we follow the training steps detailed in You al. 2020
B.3 Results of few-shot graph classification
The results for 50-shot scenarios. Table Yable4Summin summarizes the results for 50-shot graph classification.
Table 4: 50-shot test ROC-AUC (%) performance on molecular prediction benchmarks and protein function prediction benchmarks.
Pre-training Tuning Strategy Strategy
BBBP
Tox21
ToxCast
SIDER
Clin
MUV
HIV
BACE
PP1
Avg.
53.81
61.42
53.93
50.77
58.6
66.12
65.09
52.64
48.79
FT
56.79
#3.35
#1.19
#0.59
+2.27
+3.48
#0.63
#1.17
#2.64
#1.32
Infomax
55.52
65.56
56.76
50.29
62.44
68.00
67.68
54.49
54.03
GPF
59.41
+1.84
#0.64
#0.54
#1.61
#4.1 11
+0.61
#1.09
#2.54
+0,34
58.09
65.71
57.13
51.33
62.96
67.88
66.80
56,56
53.78
GPF-plus
60.02
#2.12
#0.37
+0.48
1.11
#3.27
+0.42
+1.43
#6.811
#0.45
48.88 88
60.95
55.73
51.30
57.78
66.88
64.22
61.27
47,62
FT
57.18
20.68
#1.46
#0.43
#2,21
#4.03
+0.53
#1.57
#6,10
#1.50
EdgePred
50.53
64.46
$7.33
51.35
68.74
68.08
66.22
62.85
52.81
GPF
60.26
+1.35
#0.93
#0.65
#0.76
46.03
+0.39
+1.90
+5.91
+0.38
54.49
64.99
57.69
$1,30
66.64
68.16
62.05
62.60
53.30
GPF-plus
60.13
#4.60
#0.53
#0.61
#1.18
+2.40
+0.48
#33 ,9
#2.48
#0.34
51.26
60.28
53.47
50.11
61.51
59.35
67.18
55
48,17
FT
56.32
+2.33
#1.73
#0.46
#1.63
#1.45
+1.31
11.59
#5504
+2.44
AttrMasking
54.24
64.24
56.84
50.62
65.34
61.34
67.94
57.31
51.26
GPF
58.79
#0.74
=0.40
+0.28
#0.88
#1.93
+0.60
#0.48
#6.71
#0.32
58.10
64.39
56.78
50.30
63.34
63.84
68.05
57.29
$1.35
GPF-plus
59.27
#1.92
=0.30
#0.25
=0.78
=0.85
#1.13
#0.97
#4.46
#0.32
49.45
58.77
54.46
49.89
48.60
56.14
60.91
56.37
46.33
FT
53.43
+5.74
=0.70
#0.54
#1.16
+3.40
+:4.2
11.84
#1.90
#1.76
ContextPred
52.55
59.73
55.70
50.54
53.03
61.93
60.59
59.91
50.14
GPF
56.01
+1.24
#0.86
+0.29
+0.91
#5.9
+ 84
+1.28
#-6.31
+0.33
53.76
60.59
55.91
51.44
52.37
64.51
60.84
64.21
50.52
GPF-plus
57.12
+4.47
10.51
#0.22
#: 1,70
#4.30
+:4.88
#1.11
#7.30
#0.41
54.40
48.35
50.29
53.23
54.05
46.73
60.05
49.87
49.94
FT
51.62
+2.87
#1.67
#0.19
+0.87
16.16
+1.88
+:3 30
#1.78
+1.77
GCL
53.87
50.58
52.64
53.86
64.44
47.22
64.86
67.56
50.40
GPF
55.62
#2.17
#0.49
#0.50
#0.45
#4.64
43.555
11.29
+2.29
#1.17
55.89
50.14
53.25
55,46
65.22
47.88
63.99
64.10
51.19
GPF-plus 11.58
55.89
#1.09
#0.95
+0.96
#4.55
1..77
11.60
11.85
#1.53
The results for 100-shot scenarios. We also conducted experiments where the number of training samples in the downstream task was limited to 100 for both the chemistry and biology datasets. The summary of the overall results can found Table [5] The experimental findings align with those observed in the 50-shot scenarios. Our proposed graph prompt tuning method achieves the best results in 42 out of 45 cases (14 out of 45 for GPF and out of 45 for GPF-plus). Additionally, the average results of both GPF and GPF-plus surpass the average results of fine-tuning across all pre-training strategies, showcasing the superiority of our proposed graph prompt tuning approach.
B.4 Parameter efficiency analysis
We have computed the number of tunable parameters for full fine-tuning, GPF, and GPF-plus (excluding the task-specific projection head 0) on the chemistry and biology datasets. The statistics are presented in Table 6 The results indicate that the number of tunable parameters GPF and GPF-plus is several orders of magnitude smaller than that of fine-tuning. Specifically, GPF utilizes no more than 0.02% of the tunable parameters used in fine-tuning, while GPF-plus utilizes no more than 0.7% of the tunable parameters used in fine-tuning, Our proposed graph prompt tuning methods
23

---

Table 100-shot ROC-AUC (%) performance on molecular prediction benchmarks and protein
function prediction benchmarks.
Pre-training Strategy
Tuning Strategy
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
PPI
Avg.
$6.29
60.46
55.34
50.49
50.90
65.88
65.81
57.35
49.74
FT
56.91
+1.65
#0.66
#0.20
#1.29
+4.92
+1.76
+1.43
#2.67
+0.72
Infomax
56.38
61.54
57.31
$4.49
56.49
66.52
68.02
61.67
54.57
GPF
59.66
+2.84
=0.28
#0.35
#0.72
#1.98
#0.67
#1.22
#2.76
#0,51
56.97
62.48
57.64
54.86
$7.68
67.00
67.66
61.76
54,66
GPP-
60.07
#3.46
=0.80
#0.30
#0.49
40.89
+0.47
#1.13
#2.80
#0.52
51.27
61.48
58.28
52.23
58,50
64,32
59.82
50.86
48.06
FT
56.09
#3.89
#1.21
#0.81
+.227
+2.54
+2.48
#1.47
#0.85
+2.00
EdgePred
55.13
63.35
59.09
52.30
65.02
65.47
63.19
48.64
52.52
GPF
58.30
+1.27
#0.54
#4.13
#0.31
+1.49
#6.46
$4.20
64.80
59.42
52.47
62.73
65.37
63.18
50.02
53,00
GPP-plus
58.35
+5.05
#0.95
#0.23
+0.64
#3.54
+0.37
+1.28
#:5.65
#0.44
54.56
60.95
55.84
50.64
61.16
64,90
61.65
59.03
47.29
FT
57.33
+4,82
#1.28
+0.40
#1.16
11,19
+1.43
+3.31
+2.89
11.43
AttrMasking
55.23
63.36
57.66
50.08
63.05
65.58
69.79
59.37
52.31
GPF
59.60
#3.14
#0.61
+0.34
+0.59
#4,41
+0.69
+1.78
#3.
#0.44
53.58
63.89
57.72
51.70
62.68
66,47
69.35
58.50
52.28
GPF-plus
59.57
#2.19
#0.58
+0.37
#0.61
+2.50
+0.43
11.58
#2.36
#0.89
50.42
60.74
56.00
$1.81
51.48
64.87
$9.82
50.43
45.39
$4.55
+0.57
#0.88
+0.29
#1.77
42,86
+2.30
+2,00
43.74
#0.42
ContextPred
52.33
63.91
57.32
53.55
54.31
65.80
68.51
54.70
50.44
GPF
57.87
+5.07
=0.82
#0.30
#0.88
+2.58
+0.45
#2.23
#5.89
#0.64
53.62
64.89
58.02
54.13
54.02
65.89
68.75
54.41
50.79
GPF-plus
58.28
#6.59
=0.89
#0.52
1138
#2.38
+0.54
+3.80
#5.85
#0.50
44.06
48.47
51.91
56.10
48.13
$3.93
32.63
55.41
49.44
FT
48.89
+2.55
#1.63
+0.33
#0.45
=33,23
+0.91
+0.93
#1.28
#1.53
GCL
51.34
55,46
53.78
53.37
60.44
54.06
44.23
49.20
54.35
GPF
52.91
#1.01
=1.54
+0.58
#0.99
#4.10
#2.56
=0.75
#2.94
20.65
52.47
57.42
53.07
$2.90
60.22
55.68
46.24
51.64
54.47
GPF-plus
53.79
#1.19
#1.36
#0.81
#1.05
#3.81
11.25
#3.35
+4.25
#1.25
exhibit significant advantages in terms of parameter efficiency compared fine-tuning. leads to reduced training time and storage space required for downstream adaptations.
Table 6: The number tunable parameters for different tuning strategies.
Dataset
Tuning Strategy
Tunable Parameters Relative Ratio (%)
FT
1.8M
100
Chemistry
GPF
0.3K
0.02
GPF-plus
~ 3-12K
0.17-0.68
FT
2.7M
100
Biology
GPF
0.3K
0.01
GPF-plus
~ 3-12K
0.11-0.44
B.5 Comparison with linear probing
In the field of Natural Language Processing and Computer Vision, linear probing [Kumar et al. 2022]
Wu al., 2020 Tripuraneni al. 2020 Du et al. 2020 is widely employed method for adapting pre-trained models to downstream tasks. This approach involves freezing the parameters of the pre-trained model f and solely optimizing the linear projection head To evaluate the effectiveness of linear probing, we conducted experiments on the chemistry datasets Toxcast and SIDER, and the results are summarized Table 7
evident from the results that linear probing exhibits significant performance decline compared to fine-tuning and our proposed graph prompt tuning. The primary distinction between linear probing and our proposed graph prompt tuning lies in the incorporation of additional learnable graph prompt 9s(-) in the input space. The substantial
24

---

Table 7: Test ROC-AUC (%) performance on Toxcast and SIDER with the linear probing.
Pre-training Strategy
Tuning Strategy
ToxCast SIDER
Avg.
65.16
63.34
FT
64.25
+0.53
+0.45
Infomax
63.84
59.62
Linear Probing
61.73
40.10
+0.73
66.10
GPF
66.13
#0.53
#0.81
66.35
65.62
GPE-plus
65.98
#0,37
+0,74
66.29
64.35
FT
65.32
+0.45
+0,78
EdgePred
65.25
61.47
Lincar Probing
63.36
+0.09
#0.03
65.65
67
GPF
66.42
40.30
+0.99
65.94
67.51
GPP-plus
66.72
#0.31
#0.59
65.34
66.77
FT
+0.30
#0.13
66.05
AttMMasking
64.75
62.60
Linear Probing
63.67
+0.07
#0.57
66.32
69.13
GPF
67.72
#0.42
#1,16
66.58
68.65
GPF-plus
67.61
#0.13
#0.72
66.39
64.45
FT
65.42
+0.57
+0.60
ContextPred
65,35
61.28
Linear Probing
63.31
+0.09
=0.39
67.92
66.18
GPF
67.05
40.35
:0.46
67.58
66.94
GPF-plus
67.26
#0.54
=0.95
62.54
60.63
FT
61.58
#0.26
+1 .22
GCL
50.92
52.91
Linear Probing
51.91
+0,22
62.70
61.26
GPF
61.98
#0.46
#0.53
62.76
62.37
GPF-plus
62.56
40.75
performance gap observed between these approaches underscores the importance of integrating graph prompt for the effective adaptation pre-trained trainarieed models.
B.6 Comparison with other tuning methods
also compare our proposed graph prompt tuning with other tuning methods described as follows:
PARTIAL-K: We tune the last & layers of the pre-trained model with projection head
and freeze other parts, which is utilized in Zhang et al. [2016). He et al. [202T] Jia et al. 2202
ML.P-:: We freeze the pre-trained model / and utilize multi-layer perceptron (MLP) with k layers the projection head to perform the classification.
We conduct the experiments on the biology datasets (PPI), and Table B summarizes the results. The experimental results indicate that our methods outperform other tuning methods in all cases.
B.7
Extra results on GCC
Another popular pre-training strategy graph contrastive learning involves following training steps outlined in Qiu et al. 2020aa. First, we introduce the datasets utilized for pre-training GCC. The supersueee pre-training task of GCC is conducted on six graph datasets, and provides detailed statistics for each dataset. For the downstream tasks of the pre- pre-trained GCC, we employ
25

---

Table Test ROC-AUC (%) performance on protein function prediction benchmarks with different
tuning methods.
Pre-training Strategy
FT
MLP-3
Partial-1
Partial-3
GPF
GPF-plus
Infomax
71.29 =1.79
74.68
+0.56
74.36
=0.92
73.28 =0.18
77.02 #0.42
77.03 #0.32
EdgePred
71,54 =0.85
74.60
#0.88
73.24 +0.68
73.35 #0.77
76.98
#0.20
77.00
#0.12
AttrMasking
73.93 =1.17
77.99
#0.42
75.91 #0.10
74.02
#0.37
78,91
#0.25
78.90 #0.11
Contex(Pred
72.10
11.9
76.01
#0.68
76.62
+0.92
74.86 +0.79
77.42 #0.07
77.71 #0.21
IMDB-BINARY and IMDB-MULTI datasets Yanardag and Vishwanathan 2015 Each dataset
consists of a collection of graphs associated with specific target labels. We evaluate GPF on these datasets, and the results are presented in Table [T, The experimental findings consistently demonstrate that GPF outperforms fine-tuning when adapting models pre-trained using the GCC strategy.
Table 9: Statistics of datasets for pre-training
Dataset
Academia
DBLP(SNALP) DBLP(NetRep)
IMDB
Facebook
LiveJournal
137,969
317,080 2,099,732
540,486 30,491,158
896,305
3,097, 165 47,334,788
4, 4,843,953 85,691,368
E
739,984
7,564,894
Table 10:
Test accuracy (%) performance of GCC graph classification benchmarks.
Pre-training Strategy
Tuning Strategy
IMDB-B
IMDB-M
Avg.
FT
72.60 #4.72
49.07 +3.59
60.83
GCC (E2E)
GPF
73.40 +3.80
49.17 #3.12
61.28
FT GPF
71.70
#4.98
48.07
+2.91
59.88
GCC (MoCo)
72.50
+3.20
49.33
#3.93
60.91
B.8
Hyper-parameter settings
This section presents the hyper- apper-parameters used during the adaptation stage of pre-trained GNN models on downstream tasks for our proposed graph prompt tuning. Table [T] summarizes the hyper-parameter settings. You can also visit our code repository to obtain the specific commands for reproducing the experimental results.
Table 11: The hyper-parameter settings.
Pre-training
Prompt Dimension
Learning Rate
Weight Decay
Batch Size
Training Epoch
Dataset
Strategy
Infomax
300
0.001
32
50
EdgePred
300
0.001
0
32
50
Biology
Masking
300
0.001
0
32
50
ContextPred
300
0.001
32
50
GCL
300
0.0001
32
50
Infomax
300
0.001
0
32
100
EdgePred
300
0.001
32
100
Chemistry
Masking
300
0.001
0
32
100
ContextPred
300
0.001
32
100
GCL
300
0.001
0
32
100
IMDB-B
GCC
64
0.005
Linear Linear
128
100
IMDB-M
GCC
64
0.005
128
100
26


Reviews:
[{"review_id": "DYr4hcN6ms", "reviewer": "Reviewer_aqDC", "summary": "This paper proposes the Graph Prompt Feature (GPF) to improve Graph Neural Networks (GNNs) performance amidst scarce labeled data and low out-of-distribution generalization. GPF, a universal prompt-based tuning method, operates on the input graph's feature space and is applicable to any GNN architecture. The authors also present a stronger variant, GPF-plus, providing diverse prompted features for different nodes. Both methods show better performance than fine-tuning, validated through theoretical analyses and extensive experiments.\n", "questions": "1. How do GPF and GPF-plus perform in scenarios where the feature space is noisy or inadequately represented? Are there strategies to mitigate these potential issues?\n2. Could you provide more detailed explanation or real-world examples to further illustrate the theoretical guarantees of GPF and GPF-plus?\n3. How would GPF and GPF-plus integrate with other advancements in GNNs or machine learning in general?", "limitations": "Not really.  Dear authors, please enumerate some limitations of your work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. GPF and GPF-plus are universal, model-agnostic solutions that can be applied to any pre-trained GNN model, enhancing their general applicability in diverse scenarios.\n2. The paper provides theoretical guarantees for the effectiveness of GPF and GPF-plus, strengthening the scientific rigor of the presented methods.\n3. The authors demonstrate that their proposed methods outperform existing fine-tuning strategies and specialized prompt-based tuning methods in both full-shot and few-shot scenarios, showcasing their practical effectiveness.\n\n", "weaknesses": "1. The effectiveness of GPF and GPF-plus largely depends on the quality and representation of the feature space, which may not be optimally prepared in all real-world applications.\n2. While the paper provides theoretical analysis for the methods' effectiveness, more detailed explanation or examples of the theoretical proofs could enhance the clarity and comprehensibility.\n3. The real-world applicability and performance of the proposed methods could be further substantiated with experiments on a wider range of tasks and datasets.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response**\n\nDear reviewer aqDC,  \nWe appreciate your comments and your support for our work. We hope our response can address your concerns. Please find our detailed response below.\n\n1. (Weakness 1 \\& Question 1) Dealing with scenarios where the feature space is noisy or inadequately represented.\n\nOur proposed GPF and GPF-plus methods operate on the feature space of the input graph. We understand that your concern may lie in the fact that node features can be severely corrupted or contain unacceptable levels of noise in certain datasets. Notably, reducing dependence on node features often leads to better results in such cases. In these situations, we suggest replacing the raw features with synthetic features, such as positional encoding features [1,2] or Gaussian features [3], before applying GPF and GPF-plus to the generated node features. In Appendix B.7, we present experimental results on GCC that demonstrate the effectiveness of GPF on synthetic features. In this particular experiment, all node features are generated as positional encodings, and in this scenario, GPF still outperforms fine-tuning.\n\n2. (Weakness 2 \\& Question 2) Detailed explanation or examples of the theoretical proofs.\n\nOur theoretical analysis ensures the universal capability and effectiveness of our method. We appreciate your suggestion to include additional examples, and we have incorporated concrete examples in our latest revision to illustrate the theorems more comprehensively. Regarding Theorem 1, which establishes the universal capability of GPF, we have enhanced the theorem proofs by incorporating specific examples. These examples encompass existing graph templates like GPPT as well as intuitively designed graph templates. By demonstrating the equivalence of these templates to specific forms of GPF, we aim to provide readers with a deeper understanding of the universal capability of our method. As for Theorem 2, which showcases the effectiveness of GPF, we have manually designed a specific graph task to illustrate how GPF outperforms fine-tuning by providing a more optimal solution. This example serves as evidence that GPF can achieve superior performance in certain scenarios. Thanks for your detailed suggestion, and we believe that these enhancements and additional examples strengthen our theoretical claims. \n\n3. (Weakness 3 \\& Question 3) The integration with other advancements in GNN models.\n\nYour suggestion is indeed reasonable. We have included additional experimental results utilizing more powerful backbone models. You can find these results in B.1 of the global response.\n\n\n\n**Ref**\n\n[1] Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks.\n\n[2] On the Equivalence Between Positional Node Embeddings and Structural Graph Representations.\n\n[3] Random Features Strengthen Graph Neural Networks."}, {"review_id": "lRxLOgtA6U", "reviewer": "Reviewer_GUF8", "summary": "This paper introduces the Graph Prompt Feature (GPF) approach, which aims to adapt pre-trained Graph Neural Networks (GNNs) for downstream tasks by appending tunable embeddings onto the frozen node embeddings. By doing so, the authors achieve a significant reduction in the number of parameters required for the downstream task compared to full fine-tuning. The experimental results demonstrate promising performance in binary classification tasks on Molecular and Protein datasets when compared to full fine-tuning.", "questions": "see weaknesses", "limitations": "The author did not address any limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is clearly written, the contribution is easy to understand and provide theoretical proof on effectiveness.", "weaknesses": "There are several concerns that I would like to see addressed:\n\n1) In the formulation of GPF, its relation to the concept of prompting seems unclear. It is challenging to connect this approach to the motivating concept of prompting in NLP. Instead, GPF appears to resemble task-specific adaptive fine-tuning in transformers, such as LoRA[1]. It would be helpful if the authors could elaborate more on where the similarity between GPF and prompting lies.\n\n2) The choice of downstream tasks seems to be limited to small binary classification tasks, and the comparison is only made against fine-tuning as a baseline. Why are there no results presented for multi-class classification tasks? This decision weakens the potential of the proposed method.\n\n3) Unlike previous graph prompt methods, GPF does not make any modifications to the graph's structure. It would be beneficial if the authors could explain how graph classification highlights the strength of GPF, as the additional results provided may seem somewhat redundant.\n\n4) There is no detail provided on the number of features appended into the embedding space for each node. Since GPF-plus suggests a simple size range, does this mean that the authors used different settings for GPF-plus for each dataset? This is not discussed in the experimental settings and might make the comparison somewhat incomparable.\n\n[1] https://arxiv.org/abs/2106.09685", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response**\n\nDear reviewer GUF8,  \nWe hope our point-to-point responses can address your concerns and provide you with better clarification of the contributions and value of our work.\n\n\n1. (Weakness 1) The relationship between our methods and prompting methods.\n\nOur method, GPF, is a general prompt tuning method that provides theoretically guaranteed prompts for any pre-training task within the context of GNNs. What distinguishes GPF from conventional parameter tuning techniques, such as LoRA, is its consideration of the influence of the pre-training task. Specifically, in the field of NLP, prompt tuning methods transform downstream tasks into sentence completion tasks to align them more closely with the pre-training tasks. Similarly, in the field of graphs, researchers aim to bridge the gap between downstream and pre-training tasks for GNN models. Graph prompt tuning involves selecting suitable graph templates based on the pre-training tasks, and this process is formally described in Section 3.2 of our paper. All existing graph prompt tuning methods, such as [1,2], satisfy our definition, with the only difference lying in the selection of distinct $\\psi_i(\\cdot)$ in Formula 3. Built upon the principles of graph prompt tuning, our method, GPF, is a general graph template that is theoretically applicable to any pre-training task. While the form of GPF may appear relatively simple, its effectiveness in bridging the gap between pre-training tasks and downstream tasks is theoretically guaranteed (Theorem 1). Traditional parameter tuning methods are primarily designed to optimize parameter efficiency. However, our method is designed with the primary goal of bridging the gap between pre-training tasks and downstream tasks. Therefore, our method should be categorized as a prompt tuning method that also offers advantages in terms of parameter efficiency.\n\n2. (Weakness 2) The results for multi-class classification tasks.\n\nNot all experiments involved are conducted on binary classification tasks. The IMDB-M dataset illustrates a multi-class classification task on graphs, and you can refer to the corresponding results in Table 10 in the appendix. We have included additional results in B.2 of the global response to address your concerns about GPF's performance on multi-class tasks. These experimental results showcase the satisfactory performance of our methods in handling multi-class classification tasks, and we hope they can dispel your concern.\n\nIn addition to comparing the results with fine-tuning, we have also conducted a comparison between GPF and existing graph prompting methods, as presented in Table 2. Furthermore, we have assessed GPF's performance against linear probing, as shown in Table 7, and compared GPF with other model tuning methods, as presented in Table 8. Through these extensive comparisons, we aim to evaluate GPF's performance comprehensively.\n\n3. (Weakness 3) The modifications to the graph structure.\n\nOur method, GPF, operates on the feature space of the input graph, but it does not imply that we overlook modifications to the graph structure. While we focus on adding extra learnable vectors to the feature space, it is important to note that we have demonstrated the ability of this operation to achieve an equivalent effect to any structure modification. Detailed proofs regarding this equivalence can be found in Propositions 4 and 5 in the appendix. Consequently, our method does not need explicit structural modifications. There are several advantages to this simplification. The adjacency matrix, an N*N matrix, is much larger than the feature matrix. By working on the feature matrix, our method significantly reduces resource consumption and minimizes training difficulties.\n\nFurthermore, it is worth noting that our method, GPF, is not restricted to graph classification tasks alone. Section A.1 in the appendix provides detailed elaboration on how GPF extends to encompass node-wise tasks (node classification and link prediction). To address your concern regarding GPF's performance on node-wise tasks, we have included additional experimental results in B.1 of the global response.\n\n4. (Weakness 4) Details about the experiment settings.\n\nThe dimension of the basic prompt features matches that of the input graph node features. In our experiments, when applying the GCC framework, the dimension of the prompt features is set to 64, while for other frameworks, it is set to 300. You can find the relevant description in the main text at line 203 and Section B.4 in the appendix. Regarding the hyper-parameter $k$ of GPF-plus, it determines the number of prompt feature bases utilized. This parameter can be adjusted manually and vary across different datasets. You can find a detailed description of this parameter in Equation 9 and lines 300-301 of the main text. We are sorry to make you confused, and we have included an additional hyper-parameter list in the latest revision to provide more clarity on these parameters.\n\n\n\nWe hope our explanation can dispel your concerns. If you have any other questions or concerns, please feel free to let us know.\n\n\n\n**Ref**\n\n[1] GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks.\n\n[2] GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks."}, {"review_id": "0fF65kCJaw", "reviewer": "Reviewer_Cfab", "summary": "This paper aims for efficient adaptation of pre-trained graph neural networks to downstream tasks. A simple prompt tuning method (i.e. GPF) is proposed for adaptation and is applicable to GNN pretrained with any objectives. The main idea of GPF is to add a learnable vector on all the node features in the input graph, while an improved version (i.e. GPF-plus) leverages a set of learnable vectors to compose the bias added to each node feature. Experiments show that GPF and GPF-plus achieve better results than the full finetuning.", "questions": "The author is suggested to address the concerns in the weakness section.", "limitations": "Limitation is not discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strength \n1.\tFigure 1 illustrates the difference between the proposed method and existing methods.\n\n2.\tProvide a theoretical analysis of the proposed method.\n\n3.\tThe proposed method is simple and easy to understand.", "weaknesses": "Major\n1.\tThe novelty of GSF may be limited because it just adds a bias term on the input of the pretrained network and there is nothing specific for the graph structure. However, the bias-term finetuning is not a new idea [a, b].\n\n2.\tThe authors claim that the developed method is universal for all the graph learning. However, the theoretical proof is only valid for graph classification downstream tasks where the node features are sum-pooled as the representation. There is no evidence showing that GSF is applicable to other types of downstream tasks.\n\n3.\tBaselines missing in Table 1 & 2. For example, Linear Probing is a standard baseline, and residual adapter in [c] can also be a baseline.\n\n4.\tGiven the variance shown in Table 1 & 2, the improvement of GPF-plus over GPF is marginal. Why is it an enhanced version? If GPF is theoretically feasible to address the problem (section 3.4), why there GPF-plus is needed?\n\n[a] BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. ACL’22.\n[b] Tinytl: Reduce memory, not parameters for efficient on-device learning. NeurIPS’20.\n[c] CLIP-Adapter: Better Vision-Language Models with Feature Adapters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response**\n\nDear reviewer Cfab,  \nWe hope our point-to-point responses can address your concerns and better clarify the contributions and value of our work.\n\n1. (Weakness 1) The novelty of GPF and the comparison with existing methods.\n\nThe main contributions of our work compared to existing methods can be summarized as follows:\n\na. We have pioneered the general prompt tuning method on pre-trained GNN models, offering rigorous derivations to fill the gap in the lack of theoretical validity for the effectiveness of graph prompt tuning.\nSpecifically, GPF can be seen as a general graph template theoretically applicable to any pre-training task. Despite its relatively simple form, GPF's effectiveness in bridging the gap between pre-training and downstream tasks is theoretically guaranteed (Theorem 1). While the bias-term tuning methods you mentioned are designed for parameter efficiency, our method stands out by considering the influence of the pre-training task, which distinguishes our method from conventional parameter tuning techniques.\n\nb. Our work provides valuable insights that can guide future investigations in this field. We aim to find a universal prompt tuning method that can provide effective prompts for any graph pre-training strategy. Our paper introduces GPF and GPF-plus as the most intuitive and concise options for universal graph prompt tuning. Based on our deduction and analysis, designing more complex prompt tuning methods is possible. We highlight an updated work, \"All in One: Multi-task Prompting for Graph Neural Networks\", which received the best research paper award of SIGKDD 2023 recently and successfully incorporated our innovative ideas with multi-task meta-learning. Our work serves as its theoretical foundation, providing indispensable theoretical guarantees for their proposed method.\n\nOur method, GPF, operates on the feature space of the input graph, but it does not imply that we overlook modifications to the graph structure. While we focus on adding extra learnable vectors to the feature space, it is important to note that we have demonstrated the ability of this operation to achieve an equivalent effect to any structure modification. Detailed proofs regarding this equivalence can be found in Propositions 4 and 5 in the appendix. Consequently, our method does not need explicit structural modifications. There are several advantages to this simplification. The adjacency matrix, an N*N matrix, is much larger than the feature matrix. By working on the feature matrix, our method significantly reduces resource consumption and minimizes training difficulties.\n\n2. (Weakness 2) Applicability of theoretical proof and other downstream tasks.\n\nThe theoretical proof provided can also be extended to node-level tasks. From a subgraph aggregation perspective, the node representations can be considered as the subgraph representations of the central nodes. This means that obtaining node representations can be transformed into obtaining subgraph representations. To further explain the application of GPF on node-level tasks, please refer to Section A.1 in the appendix. Furthermore, in Section 3.4 of the main text, we demonstrate that GPF achieves high flexibility and effectiveness in generating graph representations. This implies that when applied to node-level tasks, GPF can also obtain sufficiently flexible and effective node representations. We are sorry to cause your concern and want to assure you that we have included a theoretical discussion on the effectiveness of GPF for node-level tasks in our latest revision. To address your concerns, we also provide additional empirical results on node classification tasks, which can be found in B.1 of the global response.\n\nRegarding the sum pooling assumption in the theorem derivation, it is not a necessary condition for the theorem to hold. Our analysis can be extended to accommodate any weighted aggregation readout function, including average pooling, max/min pooling, and classical hierarchical pooling. These pooling methods are all encompassed within our analysis. We use the sum pooling assumption to facilitate a straightforward and comprehensible analysis. And we have included additional discussions on other pooling techniques in our latest revision.\n\n3. (Weakness 3) The comparison with linear probing and residual adapter.\n\nWe have already conducted a thorough performance comparison of our method with linear probing, and the corresponding results can be found in Section B.5 within the appendix. Additionally, we have evaluated our method against other classical tuning methods, and the corresponding results can be found in Section B.6 within the appendix. Regarding your suggestion to compare our method with the residual adapter, we have taken it into consideration and provided the results in B.3 of the global response.\n\n\n4. (Weakness 4) Advantages of GPF-plus over GPF.\n\nWe introduce GPF-plus to enhance the expressiveness and scalability of our method. Both GPF and GPF-plus are theoretically guaranteed to be universal. However, GPF-plus offers greater flexibility by allowing for different prompted features to be provided for each node.\nWhen GPF and GPF-plus are employed to approximate a specific graph template, GPF-plus possesses a larger set of feasible solutions than GPF due to its enhanced flexibility. Consequently, GPF-plus has a greater chance of obtaining better solutions during the practical training stage. Additionally, while the number of parameters in GPF is fixed according to the dimensionality of the node features, GPF-plus provides the option to adjust the number of parameters freely by selecting the number of prompt feature bases. This adaptability enables GPF-plus to flexibly adjust to the characteristics of downstream datasets.\n\nWe hope our explanation can dispel your concerns. If you have any other questions or concerns, please feel free to let us know."}, {"review_id": "qF11WIseD5", "reviewer": "Reviewer_dbJu", "summary": "This paper proposed a universal prompt-based tuning method, called GPF, for pre-trained GNN models. The idea is to operate on the feature space of the downstream input graph. The authors theoretically showed that GPF can achieve results equivalent to any prompting function, and is not weaker than full fine-tuning. Experiments showed the competitive results.", "questions": "See the weaknesses.\n\n", "limitations": "Potential limitations of the proposed GPF not discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposed a universal graph prompting approach, which basically operates on the input graph feature space. As far as I know, the idea of augmenting input graph node features for prompting is novel.\n\n2. The authors provide theoretical analysis, which showed that GPF can achieve results equivalent to any prompting function, and is not weaker than full fine-tuning.\n\n3. Experiments results showed the performance gains.", "weaknesses": "1. The motivation for a universal prompting needs better explained, and advantages over specialized prompting approaches. \n\n2. Comparison with other prompting meothds is only perfomed on the models pre-trained by edge predcition.\n\nSome minor comments:\n1. It is not clear to me how the avg. in Table 1 and Table 2 obtained, and what does it mean. It is just the average across all datasets?\n2. I suggest the authors include the detailed pretraining settings in experiments (e.g., datasets used) instead of in the appendix.\n3. Fig. 1 is not informative. The authors should use a figure to better illustrate how GPF works, and the main differences with existing prompting approaches. \n4. Table captions should be on top of tables insead of below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response**\n\nDear reviewer dbJu,  \nWe really appreciate your comments on our work. We hope our response can address your concerns.\n\n1. (Weakness 1) The motivation and advantages of universal prompting.\n\nThe universal graph prompt tuning method that we proposed offers three main advantages over specialized prompting methods:\n\na. Generalization without knowledge of pre-training details: One of the primary advantages of our method is that it eliminates the need for an in-depth understanding of the pre-training strategies employed by the pre-trained GNN models. In practical applications, gathering such detailed knowledge can be challenging and inconvenient. Our universal method can achieve satisfactory results without requiring extra information about the model's pre-training details.\n\nb. Applicability to complex graph pre-training tasks: Existing graph prompt tuning methods predominantly focus on the pre-training task of link prediction. However, when dealing with more abstract and complex graph pre-training tasks, such as graph infomax and graph contrastive learning, it becomes challenging for researchers to design appropriate prompt templates intuitively. Our universal method circumvents the need for an explicit characterization of concrete prompt templates, thus making it applicable to models trained on any graph pre-training task.\n\nc. Practical and theoretical effectiveness: Our universal method is easy to apply in practice and comes with strict theoretical guarantees of effectiveness. It offers valuable insights for future investigations in this field.\n\nWe appreciate your suggestion to include additional discussion, and we have added similar content in our latest revision.\n\n\n\n2. (Weakness 2) Comparing existing graph prompt tuning methods under alternative pre-training strategies.\n\nWe compare our approach with existing graph prompt tuning methods in the context of edge prediction because both GPPT and GraphPrompt explicitly state in their papers that they are designed specifically for the pre-training task of link prediction. These methods explicitly transform node classification tasks into link prediction tasks, which limits their ability to bridge the gap between upstream and downstream tasks when applied to other pre-training strategies. To provide more evidence, we present additional experimental results that indicate the limitations of these methods in dealing with other pre-training tasks. As an example, the table below shows the ROC-AUC scores (%) of these methods on the model pre-trained by graph infomax.\n\n|         |             | BBBP           | Tox21         | ToxCast        | SIDER          | ClinTox        | BACE           |\n| ------- | ----------- | -------------- | ------------- | -------------- | -------------- | -------------- | -------------- |\n| Infomax | FT          | **67.55±2.06** | 78.57±0.51    | 65.16±0.53     | 63.34±0.45     | 70.06±1.45     | 81.32±1.25     |\n|         | GPPT        | 56.92±0.46     | 59.37±0.84    | 53.73±0.92     | 48.23±0.79     | 53.22±0.84     | 56.22±0.16     |\n|         | GPPT(w/olo) | 62.87±0.05     | 71.50±0.70     | 57.55±0.13     | 55.77±0.19     | 54.49±0.40     | 75.37±0.30     |\n|         | GraphPrompt | 62.95±0.89     | 61.65±0.40    | 54.98±0.32     | 51.21±0.21     | 47.53±0.12     | 54.77±0.99     |\n|         | GPF         | 66.83±0.86     | 79.09±0.25    | 66.1±0.53      | **66.17±0.81** | 73.56±3.94     | 83.60±1.00     |\n|         | GPF-plus    | 67.17±0.36     | **79.13±0.70** | **66.35±0.37** | 65.62±0.74     | **75.12±2.45** | **83.67±1.08** |\n\n\n\n3. Collection of minor comments.\n\n(a) You are right, and the Avg. in Table 1 and 2 presents the average values across all datasets. \n\n(b) Thanks for your suggestion. We have moved the detailed experiment settings in our main text.\n\n(c) Thanks for your suggestion. We have adjusted the last two figures to make their meaning clearer.\n\n(d) Thanks for your detailed advice, and we have resolved this mistake."}, {"review_id": "bKfVIOFqrm", "reviewer": "Reviewer_KwHM", "summary": "The paper introduces a universal prompt-based tuning method called Graph Prompt Feature (GPF) and its variation (GPF-plus) for pre-trained Graph Neural Network (GNN) models. GPF is a universal method that can be applied to any pre-trained GNN model under any pre-training strategy. It operates on the input graph's feature space and can achieve an equivalent effect to any form of prompting function. \nGPF introduces a learnable vector $p$ of dimension $F$, which is added to the node features, where $F$ corresponds to the dimensionality of the node features. The authors also purpose GPF-plus, which assigns an independent learnable vector $p_i$ to each node $u_i$ in the graph, instead of a single vector $p$. Experimental results show that GPF outperforms fine-tuning, with an average improvement of about 1.4% in full-shot scenarios and 3.2% in few-shot scenarios.", "questions": "1) Explanation of the relationship between GPF and linear probing: The paper compares GPF with linear probing, which involves a trainable layer in the final layer. It would be beneficial if the authors could discuss the relationship between these two approaches and explain why GPF, with its learnable vector added to the input features, achieves superior performance compared to linear probing. Is there any inherent advantage or characteristic of GPF that enables it to outperform linear probing?\n\n2) Generalizability of Theorem 2: The paper presents Theorem 2, which focuses on a simple linear 1-layer GNN without activations. However, many practical GNN architectures incorporate activation functions to introduce non-linearity. It would be helpful if the authors could discuss the generalizability of the theorem's conclusions to more complex GNN models commonly used in real-world applications. Can the findings of Theorem 2 be extended to GNNs with non-linear activations? If not, what are the limitations or implications of the theorem in the context of practical GNN architectures?\n\n3) Have you considered or experimented with using more powerful GNN architectures, such as subgraph GNNs or higher order GNNs, as the backbone GNN for the finetuning stage? If so, how does the performance of GPF and GPF-plus compare when utilizing these alternative GNN architectures?\n\nI will be happy to increase my score for the paper, if the authors adequately address the mentioned weaknesses and provide satisfactory explanations and improvements in response to the questions and suggestions raised. \n\n\n*** After rebuttal increased score to \"Borderline Accept\" ** ", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) Theoretical Guarantees: The authors provide rigorous derivations and theoretical analyses to demonstrate the universality and effectiveness of GPF. They prove that GPF can achieve an equivalent effect to any form of prompting function and can outperform fine-tuning in certain cases. This theoretical foundation strengthens the credibility of their proposed method.\n\n2) Experimental Validation - Reproducibility: The paper includes extensive experiments conducted under various pre-training strategies, including both full-shot and few-shot scenarios. The experimental results consistently show that GPF outperforms fine-tuning, achieving an average improvement of about 1.4% in full-shot scenarios and 3.2% in few-shot scenarios. Furthermore, GPF surpasses existing specialized prompt-based tuning methods designed for specific pre-training strategies. Moreover, the availability of the source code enhances the reproducibility of the experiments. \n\n", "weaknesses": "1) A weakness of the paper is the lack of a clear explanation or motivation regarding why the addition of the learnable vector p to the input features in the GPF model leads to better results compared to linear probing with a trainable layer in the final layer. While the Appendix demonstrates the superior performance of GPF, the paper does not provide a comprehensive analysis or reasoning behind this improvement. The absence of a clear explanation may leave readers questioning the underlying mechanisms and factors that contribute to the observed performance gain. Without a proper understanding of the motivations and justifications for the proposed approach, it becomes challenging to assess the significance and generalizability of the findings. \n\n2) Theorem 2 assumes a simple linear 1-layer GNN (without activations). While the theorem provides theoretical insights into the convergence properties of this specific type of GNN, it may not accurately capture the behavior of more complex GNN architectures commonly used in practice. In real-world applications, GNNs often incorporate activation functions to introduce non-linearity and capture more intricate patterns in graph data. By focusing solely on a linear 1-layer GNN without activations, the theorem may limit the generalizability of its conclusions and overlook important aspects of GNN models commonly used in practical scenarios. Therefore, the applicability of the theorem's findings to more sophisticated GNN architectures with non-linear activations remains uncertain.\n\n3) One weakness of the paper is the exclusive use of Graph Isomorphism Network (GIN) as the backbone GNN for finetuning. While GIN is a widely used GNN architecture, it may not be the most expressive or optimal choice for every graph-related task[1,2]. The paper does not provide a comprehensive exploration or comparison of different backbone GNN architectures in the finetuning process. Moreover, maybe a more powerful GNN would be able to achieve better performance in the fine-tuning stage. Therefore, it would be interesting for the authors to examine if their approach leads also to better results when a more powerful GNN is used. \n\n[1] Frasca, Fabrizio, et al. \"Understanding and extending subgraph gnns by rethinking their symmetries.\" Advances in Neural Information Processing Systems 35 (2022): 31376-31390.\n[2]  Morris, Christopher, et al. \"Weisfeiler and leman go neural: Higher-order graph neural networks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response**\n\nDear reviewer KwHM,  \nWe hope our point-to-point responses can address your concerns and provide you with better clarification.\n\n1. (Weakness 1 \\& Question 1) Comparison with linear probing.\n\nThe difference between linear probing and GPF lies in the introducing of additional learnable parameters in GPF that modify the input graph. While linear probing focuses solely on training the last linear head $\\theta$ of the model, GPF not only trains $\\theta$ but also introduces a learnable vector $p$ into the input graph's feature space. To emphasize the theoretical advantage of GPF over linear probing, we define the pre-trained model as $f$, the input graph as $G$, the linear head as $\\theta$, and the target matrix as $Y$. Let $f(G)=H$ represent the representation matrix obtained from the pre-trained model. In linear probing, the goal is to find the optimal $\\theta$ that minimizes the discrepancy between $H\\theta$ and $Y$. Meanwhile, GPF incorporates additional learnable parameters $p$ into the input graph $G$, resulting in $f(G+p)=H^\\prime$. When applying GPF, we can simultaneously adjust $H^\\prime$ and $\\theta$ to make $H^\\prime\\theta$ close to $Y$. This simultaneous adjustment yields better results compared to only modifying $\\theta$, thus establishing the theoretical superiority of GPF over linear probing. Notably, linear probing cannot modify the representations $H$ obtained from the pre-trained model. In contrast, GPF addresses this limitation by manipulating the input to modify the representations obtained by the pre-trained model. Additionally, the comparison between GPF and linear probing can be regarded as an extension of Theorem 2. When the GNN model does not contain any learnable parameters, Theorem 2 highlights the theoretical advantage of GPF over linear probing. Furthermore, akin to the success of prompt tuning in the NLP domain, the merit of GPF can also be attributed to its capacity to bridge the disparity between pre-training and downstream tasks through input transformation. Consequently, compared to linear probing, GPF exhibits advantages in both intuition and theory.\n\n2. (Weakness 2 \\& Question 2) Generalizability of Theorem 2.\n\nTheorem 2 can also be applied to multi-layer GNN models, albeit with slight variations in some coefficients during the derivation process. However, incorporating non-linear activation functions presents a significant challenge for theoretical analysis. The crux lies in accurately estimating the expressive power of the multi-layer perceptron (MLP) architecture within the model. The universal approximation theorem describes the upper bound on the expressive power of MLP, stating that MLP networks with infinite depth or infinite width can theoretically approximate any function. Consequently, comparing the theoretical optimal expressive power of any network with an MLP architecture becomes meaningless. However, in practical scenarios, especially for GNN models with limited dimensions and depth, the real impact of multi-layer linear transformations with non-linear activation functions is not significantly different from that of single-layer linear transformations [1,2]. Therefore, similar to many existing works that analyze the theoretical capacity of models [3,4], we omit the consideration of non-linear activation functions in Theorem 2. Nonetheless, the presented derivation can be generalized to practical applications with only minor discrepancies.\n\n3. (Weakness 3 \\& Question 3) Results on more powerful GNN backbones.\n\nYour suggestion is rather rational, and we have included additional experimental results using more powerful GNN backbones. The results can be found in B.1 of the global response. From the experimental results, we can find that our method still achieves better results on these powerful models.\n\nWe hope our explanation can dispel your concerns. If you have any other questions or concerns, please feel free to let us know.\n\n\n\n**Ref**\n\n[1] Simplifying Graph Convolutional Networks\n\n[2] DFG-NAS: Deep and Flexible Graph Neural Architecture Search\n\n[3] Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution\n\n[4] In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning"}]

Author rebuttal:
### **Global Response**

Dear all reviewers,  
We appreciate your valuable comments on our work. We provide the following clarification and additional experimental results based on feedback.

**A. Contributions and influence**

We propose a universal graph prompt tuning method that can be applied to any pre-trained GNN models, and its effectiveness is theoretically guaranteed. By considering the impact of the pre-training task, our method is distinguished from conventional parameter tuning techniques. Importantly, our work is the pioneer in providing rigorous theoretical analysis on the effectiveness of graph prompting methods, offering valuable insights for future investigations in this field. We highlight an updated work, "All in One: Multi-task Prompting for Graph Neural Networks", which received the best research paper award of SIGKDD 2023 recently and successfully incorporated our innovative ideas with multi-task meta-learning, leading to remarkable performance improvements. Our work serves as its theoretical foundation, providing indispensable theoretical guarantees for their proposed method.

**B. Additional experiment results**

1. The performance of advanced backbone models on node classification.

We conduct experiments with advanced backbone models SUN(EGO+) [1] and 1-2-GNN [2] with subgraph aggregation on four node classification datasets pre-trained by Infomax. The following table presents the accuracy (%).

| SUN(EGO+) |      Cora      |    CiteSeer    |     PubMed     |   Ogbn-arxiv   |
| :-------: | :------------: | :------------: | :------------: | :------------: |
|    FT     |   83.15±0.23   |   71.10±0.27   |   79.11±0.21   |   70.89±0.11   |
|    GPF    |   83.46±0.33   |   72.81±0.67   |   79.91±0.42   |   70.92±0.09   |
| GPF-plus  | **83.70±0.45** | **72.89±0.53** | **80.79±0.95** | **70.95±0.17** |

| 1-2-GNN (Subgraph) |      Cora      |   CiteSeer    |     PubMed     |   Ogbn-arxiv   |
| :----------------: | :------------: | :-----------: | :------------: | :------------: |
|         FT         |   83.47±0.49   |  71.33±0.84   |   78.27±0.95   |   70.32±0.97   |
|        GPF         |   83.85±0.25   |  71.29±0.76   |   79.87±0.44   | **71.03±0.05** |
|      GPF-plus      | **84.01±0.87** | **71.56±0.10** | **79.95±0.20** |   70.82±0.40   |



2. The performance on multi-class classification.

We conduct experiments on the multi-class dataset Ogbg-ppa, and the following table presents the accuracy (%).

| Ogbg-ppa |    Infomax     |    EdgePred    |  AttrMasking   |  ContextPred   |      GCL       |
| :------: | :------------: | :------------: | :------------: | :------------: | :------------: |
|    FT    |   69.17±0.52   |   69.59±0.82   |   69.02±0.55   |   69.35±0.97   |   68.43±0.66   |
|   GPF    |   69.55±0.43   | **70.03±0.37** |   69.76±0.69   |   69.40±0.34   |   69.03±0.51   |
| GPF-plus | **69.95±0.27** |   69.73±0.37   | **69.88±0.43** | **69.51±0.37** | **69.11±0.63** |

Besides, some datasets in Table 1 and Table 2 of our paper contain multiple binary classification tasks. When we combine these binary classifications into a multi-class task and conduct experiments, we obtain the following ROC-AUC scores (%).

|          |          |     Tox21      |    ToxCast     |     SIDER      |    ClinTox     |
| :------: | :------: | :------------: | :------------: | :------------: | :------------: |
| Infomax  |    FT    |   63.84±0.80    |   52.18±0.19   |   52.55±0.51   |   60.11±0.95   |
|          |   GPF    |   64.17±0.95   |   53.35±0.47   | **55.21±0.19** |   63.81±0.33   |
|          | GPF-plus | **64.28±0.13** | **53.54±0.28** |   54.81±0.54   | **65.38±0.94** |
| EdgePred |    FT    |   63.75±0.72   |   52.36±0.09   |   53.45±0.75   |   59.30±0.44   |
|          |   GPF    |   64.85±0.18   |   52.78±0.55   |   56.35±0.05   | **59.60±0.51** |
|          | GPF-plus | **65.13±0.47** | **52.95±0.99** | **56.61±0.30** |   58.82±0.45   |



3. Comparison with the adapter.

We conduct experiments with an adapter [3] before the final linear head. The final output representation dimension of the adapter is the same as the output representation dimension of the pre-trained model. The following table presents the ROC-AUC scores (%).

|          |          |      BBBP      |     Tox21      |    ToxCast     |     SIDER      |    ClinTox     |      BACE      |
| :------: | :------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: |
| Infomax  | Adapter  |   64.19±0.46   |   76.30±0.02    |    63.21±0.20    |   62.14±0.39   |   69.33±0.01   |   80.51±0.26   |
|          |   GPF    |   66.83±0.86   |   79.09±0.25   |   66.10±0.53    | **66.17±0.81** |   73.56±3.94   |   83.60±1.00   |
|          | GPF-plus | **67.17±0.36** | **79.13±0.70** | **66.35±0.37** |   65.62±0.74   | **75.12±2.45** | **83.67±1.08** |
| EdgePred | Adapter  |   63.95±0.25   |   75.21±0.41   |   63.72±0.26   |   61.78±0.23   |   67.12±0.28   |   79.45±0.40    |
|          |   GPF    | **69.57±0.21** |   79.74±0.03   |   65.65±0.30    |   67.20±0.99    | **69.49±5.17** |   81.57±1.08   |
|          | GPF-plus |   69.06±0.68   | **80.04±0.06** | **65.94±0.31** | **67.51±0.59** |   68.80±2.58   | **81.75±2.09** |



Due to limited time, some experiments are not validated under all pre-training strategies and datasets. We will complete them later.



**Ref**

[1] Understanding and extending subgraph gnns by rethinking their symmetries.

[2] Weisfeiler and leman go neural: Higher-order graph neural networks.

[3] Parameter-Efficient Transfer Learning for NLP.

Meta-review: The paper explores the adaptation of pre-trained graph neural networks for downstream tasks. It introduces a simple method referred to as 'prompt tuning.' This approach involves enhancing the graph node features with a learnable component. The paper later presents a refinement of this concept. The method is termed 'universal' because it can be applied to GNNs pre-trained with various objectives. The authors provide both theoretical guarantees and experimental evaluations.

The reviewers raised several concerns about the presentation, including a lack of intuition regarding the reasons for improvements over the baselines, the relationship with prompt tuning in Language Model Models, limited experiments. The authors responded by providing comprehensive explanations and conducting additional experiments that persuaded the reviewers to increase their scores and lean toward accepting the paper. Overall, the proposed method offers a general method with good performance that covers a variety of GNNs adaptation setups. I recommend an accept.
Decision: Accept (poster)