Title: Universal Prompt Tuning for Graph Neural Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

prompt tuning
Abstract: In recent years, prompt tuning has sparked a research surge in adapting pre-trained models. Unlike the unified pre-training strategy employed in the language field, the graph field exhibits diverse pre-training strategies, posing challenges in designing appropriate prompt-based tuning methods for graph neural networks. While some pioneering work has devised specialized prompting functions for models that employ edge prediction as their pre-training tasks, these methods are limited to specific pre-trained GNN models and lack broader applicability. In this paper, we introduce a universal prompt-based tuning method called Graph Prompt Feature (GPF) for pre-trained GNN models under any pre-training strategy. GPF operates on the input graph's feature space and can theoretically achieve an equivalent effect to any form of prompting function. Consequently, we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we employ GPF to obtain the prompted graph for the downstream task in an adaptive manner. We provide rigorous derivations to demonstrate the universality of GPF and make guarantee of its effectiveness. The experimental results under various pre-training strategies indicate that our method performs better than fine-tuning, with an average improvement of about 1.4% in full-shot scenarios and about 3.2% in few-shot scenarios. Moreover, our method significantly outperforms existing specialized prompt-based tuning methods when applied to models utilizing the pre-training strategy they specialize in. These numerous advantages position our method as a compelling alternative to fine-tuning for downstream adaptations.

Universal Prompt Tuning for Graph Neural Networks

Abstract

In recent years, prompt tuning has sparked a research surge in adapting pre-trained models. Unlike the unified pre-training strategy employed in the language field, the graph field exhibits diverse pre-training strategies, posing challenges in designing appropriate prompt-based tuning methods for graph neural networks. While some pioneering work has devised specialized prompting functions for models that employ edge prediction as their pre-training tasks, these methods are limited to specific pre-trained GNN models and lack broader applicability. In this paper, we introduce a universal prompt-based tuning method called Graph Prompt Feature (GPF) for pre-trained GNN models under any pre-training strategy. GPF operates on the input graph's feature space and can theoretically achieve an equivalent effect to any form of prompting function. Consequently, we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we employ GPF to obtain the prompted graph for the downstream task in an adaptive manner. We provide rigorous derivations to demonstrate the universality of GPF and make guarantee of its effectiveness. The experimental results under various pre-training strategies indicate that our method performs better than finetuning, with an average improvement of about 1 . 4% in full-shot scenarios and about 3 . 2% in few-shot scenarios. Moreover, our method significantly outperforms existing specialized prompt-based tuning methods when applied to models utilizing the pre-training strategy they specialize in. These numerous advantages position our method as a compelling alternative to fine-tuning for downstream adaptations. Our code is available at: https://github.com/zjunet/GPF .

1 Introduction

Graph neural networks (GNNs) have garnered significant attention from researchers due to their remarkable success in graph representation learning [Kipf and Welling, 2017, Hamilton et al., 2017, Xu et al., 2019]. However, two fundamental challenges hinder the large-scale practical applications of GNNs. One is the scarcity of labeled data in the real world [Zitnik et al., 2018], and the other is the low out-of-distribution generalization ability of the trained models [Hu et al., 2020a, Knyazev et al., 2019, Yehudai et al., 2021, Morris et al., 2019]. To overcome these challenges, researchers have made substantial efforts in designing pre-trained GNN models [Xia et al., 2022b, Hu et al., 2020a,b, Lu et al., 2021] in recent years. Similar to the pre-trained models in the language field, pre-trained GNN models undergo training on extensive pre-training datasets and are subsequently adapted to downstream tasks. Most existing pre-trained GNN models obey the 'pre-train, fine-tune' learning strategy [Xu et al., 2021a]. Specifically, we train a GNN model with a massive corpus of pre-training graphs, then we utilize the pre-trained GNN model as initialization and fine-tune the model parameters based on the specific downstream task.

Pretrained

GNN

(a

)

Fine-tuning

(b

)

(c

)

Results on PPI

+

Feature

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

ROC-AUC Score

Tunable parameters (%)

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Fine-tuning

Pretrained

GNN

Downstream Tasks

Downstream Tasks

(a

)

Fine-tuning

(b

)

Graph Prompt Tuning

(c

)

Results on PPI

+

Feature

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

ROC-AUC Score

Tunable parameters (%)

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Fine-tuning

Pretrained

GNN

Downstream Tasks

Downstream Tasks

(a

)

Fine-tuning

(b

)

Graph Prompt Tuning

(c

)

Results on PPI

+

Feature

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

ROC-AUC Score

Tunable parameters (%)

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Fine-tuning

Pretrained

GNN

Downstream Tasks

Downstream Tasks

(a

)

Fine-tuning

(b

)

Graph Prompt Tuning

+

Feature

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

Pretrained

GNN

Downstream Tasks

Downstream Tasks

(a

)

Fine-tuning

(b

)

Graph Prompt Tuning

(c

)

Results on PPI

+

Feature

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

ROC-AUC Score

Tunable parameters (%)

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Fine-tuning

Pretrained

GNN

Downstream Tasks

Downstream Tasks

(a

)

Fine-tuning

(b

)

Graph Prompt Tuning

+

Feature

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

ROC-AUC Score

?

?

Specialized Graph

Template

Graph Prompt

Feature

?

?

?

?

Pretrained

GNN

Downstream Tasks

Downstream Tasks

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

ROC-AUC Score

Pretrained

GNN

Downstream Tasks

Downstream Tasks

(a

)

Fine-tuning

(b

)

Graph Prompt Tuning

(c

)

Results on PPI

+

Feature

Graph Prompt

Graph

Input

Pretrained

GNN

Tuned

Frozen

ROC-AUC Score

Tunable parameters (%)

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Fine-tuning

(a) Fine-tuning

(b) Specialized Graph

(b) Prompt Tuning

(c) Prompt Tuning

Feature

ROC-AUC Score

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Fine-tuning

Fine-tuning

Supervised AttrMasking

Supervised Infomax

Supervised

AttrMasking

Graph Prompt

Fine-tuning

Tunable parameters (%)

Tunable parameters (%)

(c

(c

)

Results on PPI

Results on PPI

Tunable parameters (%)

Results on PPI

)

However, the 'pre-train, fine-tune' framework of pre-trained GNN models also presents several critical issues [Jin et al., 2020]. First, there is a misalignment between the objectives of pre-training tasks and downstream tasks [Liu et al., 2022a]. Most existing pre-trained models employ selfsupervised tasks [Liu et al., 2021e] such as edge prediction and attribute masking as the training targets during pre-training, while the downstream tasks involve graph or node classification. This disparity in objectives leads to sub-optimal performance [Liu et al., 2023]. Additionally, ensuring that the model retains its generalization ability is challenging. Pre-trained models may suffer from catastrophic forgetting [Zhou and Cao, 2021, Liu et al., 2020] during downstream adaptation. This issue becomes particularly acute when the downstream data is small in scale, approaching the fewshot scenarios [Zhang et al., 2022]. The pre-trained model tends to over-fit the downstream data in such cases, rendering the pre-training process ineffective.

'If your question isn't getting the desired response, try rephrasing it. ' In recent years, a novel approach called prompt tuning has emerged as a powerful method for downstream adaptation, addressing the aforementioned challenges. This technique has achieved significant success in Natural Language Processing [Li and Liang, 2021, Lester et al., 2021a, Liu et al., 2022a,b] and Computer Vision [Bahng et al., 2022, Jia et al., 2022]. Prompt tuning provides an alternative method for adapting pre-trained models to specific downstream tasks: it freezes the parameters of the pre-trained model and modifies the input data. Unlike fine-tuning, prompt tuning diverges from tuning the parameters of the pretrained model and instead focuses on adapting the data space by transforming the input.

Despite that, applying prompt tuning on pre-trained GNN models poses significant challenges and is far from straightforward. First, the diverse pre-training strategies employed on graphs make it difficult to design suitable prompting functions. Previous research [Liu et al., 2022a] suggests that the prompting function should be closely aligned with the pre-training strategy. For pre-trained language models, the typical pre-training tasks involve masked sentence completion [Brown et al., 2020]. In order to align with this task, we may modify a sentence like "I received a gift" to "I received a gift, and I feel [Mask]" to make it closer to the task of sentence completion. However, in the case of graph pre-training, there is no unified pre-training task, making it challenging to design feasible prompting functions. Some pioneering studies [Sun et al., 2022, Liu et al., 2023] have applied prompt-based tuning methods to models pre-trained by edge prediction [Kipf and Welling, 2016b]. They introduce virtual class-prototype nodes/graphs with learnable links into the original graph, making the adaptation process more akin to edge prediction. However, these methods have limited applicability and are only compatible with specific models. When it comes to more intricate pre-training strategies, it becomes challenging to design manual prompting functions in the same manner as employed for link prediction. Consequently, no prompt-based tuning method is available for models pre-trained using alternative strategies, such as attribute masking [Hu et al., 2020a]. Furthermore, existing prompt-based tuning methods for GNN models are predominantly designed based on intuition, lacking theoretical guarantees for their effectiveness.

In this paper, we address the aforementioned issues for graph prompt tuning. To deal with the diversity of graph pre-training strategies, we propose a universal prompt-based tuning method that can be

applied to the pre-trained GNN models that employ any pre-training strategy. Figure 1 illustrates the distinction between our universal prompt-based tuning method and existing approaches. Our solution, called Graph Prompt Feature (GPF), operates on the input graph's feature space and involves adding a shared learnable vector to all node features in the graph. This approach is easily applicable to any GNN architecture. We rigorously demonstrate that GPF can achieve comparable results to any form of prompting function when applied to arbitrary pre-trained GNN models. Consequently, instead of explicitly illustrating the prompting function corresponding to each pre-training strategy, we adopt GPF to dynamically obtain the prompted graph for downstream tasks. We also introduce a theoretically stronger variant of GPF, named GPF-plus, for practical application, which incorporates different prompted features for different nodes in the graph. To guarantee the effectiveness of our proposed GPF and GPF-plus, we provide theoretical analyses to prove that GPF and GPF-plus are not weaker than full fine-tuning and can obtain better theoretical tuning results in some cases. Furthermore, we conduct extensive experiments to validate the efficacy of our methods. Despite using a significantly smaller number of tunable parameters than fine-tuning, GPF and GPF-plus achieve better results across all pre-training strategies. For models pre-trained using edge prediction, GPF and GPF-plus exhibit a substantial performance advantage over existing specialized prompt-based tuning methods. Overall, the contributions of our work can be summarized as follows:

To the best of our knowledge, we present the first investigation of universal prompt-based tuning methods for existing pre-trained GNN models. We propose GPF and its variant, GPF-plus, as novel approaches for universal graph prompt tuning. Our methods can be applied to the pre-trained GNN models that employ any pre-training strategy.

We provide theoretical guarantees for the effectiveness of GPF and GPF-plus. We demonstrate that GPF and GPF-plus can achieve an equivalent effect to any prompting function and can obtain better tuning results in some cases compared to fine-tuning.

We conduct extensive experiments (both full-shot and few-shot scenarios) to validate the effectiveness of GPF and GPF-plus. The experimental results indicate that GPF and GPFplus can perform better than fine-tuning, with an average improvement of about 1 . 4% in full-shot scenarios and about 3 . 2% in few-shot scenarios. Furthermore, GPF and GPF-plus significantly outperform existing prompt-based tuning methods when applied to models that utilize the pre-training strategy they specialize in.

2 Related work

Pre-trained GNN Models Inspired by the remarkable achievements of pre-trained models in Natural Language Processing [Qiu et al., 2020b] and Computer Vision [Long et al., 2022], substantial efforts have been dedicated to pre-trained GNN models (PGMs) [Xia et al., 2022b] in recent years. These methods utilize self-supervised strategies [Jin et al., 2020] to acquire meaningful representations from extensive pre-training graphs. GAE [Kipf and Welling, 2016a] first uses edge prediction as the objective task to train graph representations. Deep Graph Infomax (DGI) [Velickovic et al., 2019b] and InfoGraph [Sun et al., 2019] are proposed to garner nodes or graph representations by maximizing the mutual information between graph-level and substructure-level representations of different granularity. Hu et al. [2020a] employ attribute masking and context prediction as pre-training tasks to predict molecular properties and protein functions. Both GROVER [Rong et al., 2020] and MGSSL [Zhang et al., 2021a] propose to predict the presence of the motifs or generate them with the consideration that rich domain knowledge of molecules hides in the motifs. Graph Contrastive Learning (GCL) is another widely adopted pre-training strategy for GNN models. GraphCL [You et al., 2020] and JOAO [You et al., 2021] propose various augmentation strategies to generate different augmented views for contrastive learning. In summary, there exists a diverse range of pre-training strategies for GNN models, each characterized by unique objectives.

Prompt-based Tuning Methods Prompt-based tuning methods, originating from Natural Language Processing, have been widely used to facilitate the adaptation of pre-trained language models to various downstream tasks [Liu et al., 2021a]. Research has also explored the design of soft prompts to achieve optimal performance [Lester et al., 2021b, Liu et al., 2021c]. These methods freeze the parameters of the pre-train models and introduce additional learnable components in the input space, thereby enhancing the compatibility between inputs and pre-trained models. Aside from the success of prompts in the language field, the prompting methods are utilized in other areas. Jia et al. [2022] and Bahng et al. [2022] investigate the efficacy of adapting large-scale models in the vision field by

modifying input images at the pixel level. In the realm of graph neural networks, the exploration of prompt-based tuning methods is still limited. Some pioneering work [Sun et al., 2022, Liu et al., 2023] applies prompt-based tuning methods on the models pre-trained by edge prediction [Kipf and Welling, 2016b]. These methods introduce virtual class-prototype nodes/graphs with learnable links into the input graph, making the downstream adaptation more closely resemble edge prediction. However, these methods are specialized for models pre-trained using edge prediction and cannot be applied to models trained with other strategies. We are the first to investigate the universal prompt-based tuning methods that can be applied to the GNN models under any pre-training strategy.

3 Methodology

We introduce graph prompt tuning for adapting pre-trained GNN models to downstream tasks. It is important to note that there are several types of downstream tasks in graph analysis, including node classification, link prediction, and graph classification. We first concentrate on the graph classification task and then extend our method to node-wise tasks. We define the notations in Section 3.1, then illustrate the process of graph prompt tuning in Section 3.2. We introduce our universal graph prompt tuning method in Section 3.3 and provide theoretical analyses in Section 3.4. Finally, we present the extension of our method to node-wise tasks (node classification and link prediction) in the appendix.

3.1 Preliminaries

Let G = ( V , E ) ∈ G represents a graph, where V = { v 1 , v 2 , . . . , v N } , E ⊆ V × V denote the node set and edge set respectively. The node features can be denoted as a matrix X = { x 1 , x 2 , . . . , x N } ∈ R N × F , where x i ∈ R F is the feature of the node v i , and F is the dimensionality of node features. A ∈ { 0 , 1 } N × N denotes the adjacency matrix, where A ij = 1 if ( v i , v j ) ∈ E .

Fine-Tuning Pre-trained Models. Given a pre-trained GNN model f , a learnable projection head θ and a downstream task dataset D = { ( G 1 , y 1 ) , . . . , ( G m , y m ) } , we adjust the parameters of the pre-trained model f and the projection head θ to maximize the likelihood of predicting the correct labels y of the downstream graph G :

3.2 Graph Prompt Tuning

Overall Process. Our proposed graph prompt tuning works on the input space by drawing on the design of the prompt tuning in the language field [Liu et al., 2022a]. Given a frozen pre-trained GNN model f , a learnable projection head θ , and a downstream task dataset D = { ( G 1 , y 1 ) , . . . , ( G m , y m ) } , our target is to obtain a task-specific graph prompt g ϕ : G → G parameterized by ϕ . The graph prompt g ϕ ( · ) transforms the input graph G into a specific prompted graph g ϕ ( G ) . And then g ϕ ( G ) will replace G as input to the pre-trained GNN model f . During the downstream task training, we select the optimal parameters of ϕ and θ that maximize the likelihood of predicting the correct labels y without tuning the pre-trained model f , which can be formulated as:

During the evaluation stage, the test graph G test is first transformed by graph prompt g ϕ ( · ) , and the resulting prompted graph g ϕ ( G test ) is processed through the frozen GNN model f .

Practical Usage. In this part, we provide a detailed description of the refined process of graph prompt tuning , which comprises two fundamental steps: template design and prompt optimization .

A. Template Design. Given an input graph G , we first generate a graph template G ∗ , which includes learnable components in its adjacency matrix A ∗ and feature matrix X ∗ . Previous research has attributed the success of prompt tuning to bridging the gap between pre-training tasks and downstream tasks [Liu et al., 2022a]. Consequently, it implies that the specific form of the graph template is influenced by the pre-training strategy employed by the model. For a specific pre-training task t ∈ T and an input graph G , the graph template G ∗ can be expressed as:

where the graph template G ∗ may contain learnable parameters ( i.e. , tunable links or node features) in its adjacency matrix or feature matrix (similar to the inclusion of learnable soft prompts in a sentence), the candidate space for A ∗ is A , and the candidate space for X ∗ is X .

B. Prompt Optimization. Once we have obtained the graph template G ∗ , our next step is to search for the optimal ˆ A and ˆ X within their respective candidate spaces A and X that maximize the likelihood of correctly predicting the labels y using the pre-trained model f and a learnable projection head θ . This process can be expressed as:

The graph ˆ G composed of ˆ A and ˆ X can be considered as the the prompted graph g ϕ ( G ) mentioned in Formula 2.

Practical Challenges. The specific form of the graph template is closely tied to the pre-training task t employed by the model f . However, designing the prompting function ψ t ( · ) is challenging and varies for different pre-training tasks. Pioneering works [Sun et al., 2022, Liu et al., 2023] have proposed corresponding prompting functions ψ t ( · ) for a specific pre-training strategy, with a focus on models pre-trained using edge prediction. However, many other pre-training strategies [Hu et al., 2020a, Xia et al., 2022b], such as attribute masking and context prediction, are widely utilized in existing pre-trained GNN models, yet no research has been conducted on designing prompting functions for these strategies. Furthermore, existing prompting functions are all intuitively designed, and these manual prompting functions lack a guarantee of effectiveness. It raises a natural question: Can we design a universal prompting method that can be applied to any pre-trained model, regardless of the underlying pre-training strategy?

3.3 Universal Graph Prompt Design

In this section, we introduce a universal prompting method and its variant. Drawing inspiration from the success of pixel-level Visual Prompt (VP) techniques [Bahng et al., 2022, Wu et al., 2022, Xing et al., 2022] in Computer Vision, our methods introduce learnable components to the feature space of the input graph. In Section 3.4, we will demonstrate that these prompting methods can theoretically achieve an equivalent effect as any prompting function ψ t ( · ) .

Graph Prompt Feature (GPF). GPF focuses on incorporating additional learnable parameters into the feature space of the input graph. Specifically, the learnable component p is a vector of dimension F , where F corresponds to the dimensionality of the node features. It can be denoted as:

The learnable vector p is added to the graph features X to generate the prompted features X ∗ , which can be expressed as:

The prompted features X ∗ replace the initial features X and are processed by the pre-trained model.

Graph Prompt Feature-Plus (GPF-plus). Building upon GPF, we introduce a variant called GPF-plus, which assigns an independent learnable vector p i to each node v i in the graph. It can be expressed as:

Similarly to GPF, the prompted features X ∗ replace the initial features X and are processed by the pre-trained model. However, such a design is not universally suitable for all scenarios. For instance, when training graphs have different scales (i.e., varying node numbers), it is challenging to train such a series of p i . Additionally, when dealing with large-scale input graphs, such design requires a substantial amount of storage resources due to its O ( N ) learnable parameters. To address these issues, we introduce an attention mechanism in the generation of p i , making GPF-plus more parameter-efficient and capable of handling graphs with different scales. In practice, we train only k independent basis vectors p b , which can be expressed as:

where k is a hyper-parameter that can be adjusted based on the downstream dataset. To obtain p i for node v i , we utilize attentive aggregation of these basis vectors with the assistance of k learnable linear projections a . The calculation process can be expressed as:

Subsequently, p i is used to generate the prompted feature X ∗ as described in Formula 8.

3.4 Theoretical Analysis

In this section, we provide theoretical analyses for our proposed GPF and GPF-plus. Our analyses are divided into two parts. First, we certify the universality of our methods. We demonstrate that our approaches can theoretically achieve results equivalent to any prompting function ψ t ( · ) . It confirms the versatility and applicability of our methods across different pre-training strategies. Then, we make guarantee of the effectiveness of our proposed methods. Specifically, we demonstrate that our proposed graph prompt tuning is not weaker than full fine-tuning, which means that in certain scenarios, GPF and GPF-plus can achieve superior tuning results compared to fine-tuning. It is important to note that our derivations in the following sections are based on GPF, which adds a global extra vector p to all nodes in the graph. GPF-plus, being a more powerful version, can be seen as an extension of GPF and degenerates to GPF when the hyperparameter k is set to 1 . Therefore, the analyses discussed for GPF are also applicable to GPF-plus.

Before we illustrate our conclusions, we first provide some preliminaries. For a given pre-training task t ∈ T and an input graph G : ( A , X ) , we assume the existence of a prompting function ψ t ( · ) that generates a graph template G ∗ : ( A ∗ , X ∗ ) = ψ t ( G ) . The candidate space for A ∗ and X ∗ is denoted as A and X , respectively.

Theorem 1. (Universal Capability of GPF) Given a pre-trained GNN model f , an input graph G : ( A , X ) , an arbitrary prompting function ψ t ( · ) , for any prompted graph ˆ G : ( ˆ A ∈ A , ˆ X ∈ X ) in the candidate space of the graph template G ∗ = ψ t ( G ) , there exists a GPF extra feature vector ˆ p that satisfies:

The complete proof of Theorem 1 can be found in the appendix. Theorem 1 implies that GPF can achieve the theoretical performance upper bound of any prompting function described in Formula 3 and 4. Specifically, if optimizing the graph template G ∗ generated by a certain prompting function ψ t ( · ) can yield satisfactory graph representations, then theoretically, optimizing the vector p of GPF can also achieve the exact graph representations. This conclusion may initially appear counterintuitive since GPF only adds learnable components to node features without explicitly modifying the graph structure. The key lies in understanding that the feature matrix X and the adjacency matrix A are not entirely independent during the processing. The impact of graph structure modifications on the final graph representations can also be obtained through appropriate modifications to the node features. Therefore, GPF and GPF-plus, by avoiding the explicit illustration of the prompting function ψ t ( · ) , adopt a simple yet effective architecture that enables them to possess universal capabilities in dealing with pre-trained GNN models under various pre-training strategies.

Next, we make guarantee of the effectiveness of GPF and demonstrate that GPF is not weaker than fine-tuning, which means GPF can achieve better theoretical tuning results in certain situations compared to fine-tuning. In Natural Language Processing, the results obtained from fine-tuning are generally considered the upper bound for prompt tuning results [Lester et al., 2021a, Liu et al., 2021b, Ding et al., 2022]. It is intuitive to believe that fine-tuning, which allows for more flexible and comprehensive parameter adjustments in the pre-trained model, can lead to better theoretical results during downstream adaptation. However, in the graph domain, the architecture of graph neural networks magnifies the impact of input space transformation on the final representations to some extent. To further illustrate this point, following previous work [Kumar et al., 2022, Tian et al., 2023, Wei et al., 2021], we assume that the downstream task utilizes the squared regression loss l = ∑ i (ˆ y i -y i ) 2 .

Theorem 2. (Effectiveness Guarantee of GPF) For a pre-trained GNN model f , a series of graphs D = { ( G 1 : ( A 1 , X 1 ) , y 1 ) , . . . , ( G m : ( A m , X m ) , y m ) } under the non-degeneracy condition, and a

linear projection head θ , there exists Y ′ = { y ′ 1 , . . . , y ′ m } for y 1 = y ′ 1 , . . . , y m = y ′ m that satisfies:

The detailed proof of Theorem 2 and the description of the degeneracy condition can be found in the appendix. Theorem 2 indicates that GPF obtains a lower minimum loss compared to fine-tuning in certain scenarios, demonstrating its ability to achieve better theoretical tuning results.

4 Experiments

4.1 Experiment Setup

Model Architecture and Datasets. We adopt the widely used 5-layer GIN [Xu et al., 2019] as the underlying architecture for our models, which aligns with the majority of existing pre-trained GNN models [Xia et al., 2022b, Hu et al., 2020a, Qiu et al., 2020a, You et al., 2020, Suresh et al., 2021, Xu et al., 2021b, Zhang et al., 2021b, You et al., 2022, Xia et al., 2022a]. As for the benchmark datasets, we employ the chemistry and biology datasets published by Hu et al. [2020a]. A comprehensive description of these datasets can be found in the appendix.

Pre-training Strategies. We employ five widely used strategies (tasks) to pre-train the GNN models, including Deep Graph Infomax (denoted by Infomax) [Velickovic et al., 2019a], Edge Prediction (denoted by EdgePred) [Kipf and Welling, 2016a], Attribute Masking (denoted by AttrMasking) [Hu et al., 2020a], Context Prediction (denoted by ContextPred) [Hu et al., 2020a] and Graph Contrastive Learning (denoted by GCL) [You et al., 2020]. A detailed description of these pre-training strategies can be found in the appendix.

Tuning Strategies. We adopt the pre-trained models to downstream tasks with different tuning strategies. Given a pre-trained GNN model f , a task-specific projection head θ ,

Fine Tuning (denoted as FT). We tune the parameters of the pre-trained GNN model f and the projection head θ simultaneously during the downstream training stage.

Graph Prompt Feature (denoted as GPF). We freeze the parameters of the pre-trained model f and introduce an extra learnable feature vector p into the feature space of the input graph described as Formula 6. We tune the parameters of the projection head θ and feature vector p during the downstream training stage.

Graph Prompt Feature-Plus (denoted as GPF-plus). We freeze the parameters of pretrained model f and introduce k learnable basis vectors p b 1 , . . . , p b k with k learnable linear projections a 1 , . . . , a k to calculate the node-wise p i as Formula 10. We tune the parameters of the projection head θ , basis vectors p b 1 , . . . , p b k , and linear projections a 1 , . . . , a k during the downstream training stage.

Implementation. We perform five rounds of experiments with different random seeds for each experimental setting and report the average results. The projection head θ is selected from a range of [1, 2, 3]-layer MLPs with equal widths. The hyper-parameter k of GPF-plus is chosen from the range [5,10,20]. Further details on the hyper-parameter settings can be found in the appendix.

4.2 Main Results

We compare the downstream performance of models trained using different pre-training and tuning strategies, and the overall results are summarized in Table 1. Our systematic study suggests the following observations:

1. Our graph prompt tuning outperforms fine-tuning in most cases. Based on the results presented in Table 1, it is evident that GPF and GPF-plus achieve superior performance compared to fine-tuning in the majority of cases. Specifically, GPF outperforms fine-tuning in 28 / 36 experiments, while GPFplus outperforms fine-tuning in 29 / 36 experiments. It is worth noting that the tunable parameters in GPF and GPF-plus are significantly fewer in magnitude than those in fine-tuning (details can be found

in the appendix). These experimental findings highlight the efficacy of our methods and demonstrate their capability to unleash the power of the pre-trained models.

2. GPF and GPF-plus exhibit universal capability across various pre-training strategies. GPF and GPF-plus present favorable tuning performance across all pre-training strategies examined in our experiments, consistently surpassing the average results obtained from fine-tuning. Specifically, GPF achieves an average improvement of 1 . 14% , while GPF-plus achieves an average improvement of 1 . 60% . These results signify the universal capability of GPF and GPF-plus, enabling their application to models trained with any pre-training strategy.

3. GPF-plus marginally outperforms GPF. Among the two graph prompt tuning methods, GPF-plus performs better than GPF in the majority of experiments ( 26 / 36 ). As discussed in Section 3.3, GPF-plus offers greater flexibility and expressiveness compared to GPF. The results further affirm that GPF-plus is an enhanced version of graph prompt tuning, aligning with the theoretical analysis.

4.3 Comparison with Existing Graph Prompt-based Methods

We also conducted a comparative analysis between our proposed methods, GPF and GPF-plus, and existing graph prompt-based tuning approaches [Sun et al., 2022, Liu et al., 2023]. Both of them





























































specialize in tuning the models pre-trained by Edge Prediction (also known as Link Prediction). We apply GPPT [Sun et al., 2022], GPPT without orthogonal prompt constraint loss (denoted as GPPT (w/o ol)) [Sun et al., 2022], GraphPrompt [Liu et al., 2023] to the models pre-trained using Edge Prediction, and the results are summarized in Table 2. It is worth mentioning that GPPT is originally designed for node classification tasks. Therefore, we make minor modifications by substituting class-prototype nodes with class-prototype graphs to adapt it for graph classification tasks. The experimental results indicate that our proposed GPF and GPF-plus outperform existing graph prompt-based tuning methods by a significant margin. On the chemistry and biology benchmarks, GPF and GPF-plus achieve average improvements of 12% , 3% , and 13% over GPPT, GPPT (w/o ol), and GraphPrompt, respectively. These results showcase the ability of GPF and GPF-plus to achieve superior results compared to existing graph prompt-based tuning methods designed specifically for the pre-training strategy. Furthermore, it is worth highlighting that GPF and GPF-plus are the only two graph prompt-based tuning methods that surpass the performance of fine-tuning.

4.4 Additional Experiments

Few-shot graph classification. Prompt tuning has also been recognized for its effectiveness in addressing few-shot downstream tasks [Brown et al., 2020, Schick and Schütze, 2020b,a, Liu et al., 2021d, 2023]. We evaluate the efficacy of our proposed methods in handling few-shot scenarios. To conduct few-shot graph classification on the chemistry and biology datasets, we limit the number of training samples in the downstream tasks to 50 (compared to the original range of 1.2k to 72k training samples). The results are summarized in Table 4 of the appendix. Compared to the full-shot scenarios, our proposed graph prompt tuning demonstrates even more remarkable performance improvement (an average improvement of 2 . 95% for GPF and 3 . 42% for GPF-plus) over fine-tuning in the few-shot scenarios. This finding indicates that our solutions retain a higher degree of generalization ability in pre-trained models during few-shot downstream adaptations compared to fine-tuning.

Training process analysis. We conducted an analysis of the training process using different tuning methods on the biology datasets with the GNN models that employ Attribute Masking and Context Prediction as their pre-training tasks [Hu et al., 2020b]. Figure 2 presents the training and test curves during the adaptation stage. From Figure 2 (a), it can be observed that the ROC-AUC scores of the training set consistently increase during the adaptation stage for both our proposed methods and fine-tuning. However, from Figure 2 (b), we can find that their behavior on the test set is quite distinct. For fine-tuning, the ROC-AUC scores on the test set exhibit fluctuations and continuously decrease after an initial increase. On the other hand, when applying GPF or GPF-plus to adapt pre-trained models, the ROC-AUC scores on the test set continue to grow and remain consistently high. These results indicate that fully fine-tuning a pre-trained GNN model on a downstream task may lose the model's generalization ability. In contrast, employing our proposed graph prompt tuning methods can significantly alleviate this issue and maintain superior performance on the test set.

5 Conclusion

In this paper, we introduce a universal prompt-based tuning method for pre-trained GNN models. Our method GPF and its variant GPF-plus operate on the feature space of the downstream input graph. GPF and GPF-plus can theoretically achieve an equivalent effect to any form of prompting function, meaning we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we can adaptively use GPF to obtain the prompted graph for downstream task adaptation. Compared to fine-tuning, the superiority of our method is demonstrated both theoretically and empirically, making it a compelling alternative for downstream adaptations.

References

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. 2022.

Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, G. Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. ICLR , 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurIPS , 2020.

Leonardo Cotta, Christopher Morris, and Bruno Ribeiro. Reconstruction for powerful graph representations. NeurIPS , 2021.

Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv , abs/2203.06904, 2022.

Simon Shaolei Du, Wei Hu, Sham M. Kakade, J. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. ICLR , 2020.

Fabrizio Frasca, Beatrice Bevilacqua, Michael Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. NeurIPS , 2022.

Anna Gaulton, Louisa J. Bellis, A. Patrícia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington. Chembl: a large-scale bioactivity database for drug discovery. Nucleic Acids Research , 40:D1100 D1107, 2012.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS , pages 1024-1034, 2017.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. ArXiv , abs/2111.06377, 2021.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. ICLR , 2020a.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020b.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. ECCV , 2022.

Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Selfsupervised learning on graphs: Deep insights and new direction. ArXiv , abs/2006.10141, 2020.

Thomas Kipf and Max Welling. Variational graph auto-encoders. ArXiv , abs/1611.07308, 2016a.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. In ArXiv , volume abs/1611.07308, 2016b.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learning. In Neural Information Processing Systems , 2019.

Boris Knyazev, Graham W. Taylor, and Mohamed R. Amer. Understanding attention and generalization in graph neural networks. In NeurIPS , 2019.

Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. ICLR , 2022.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP , 2021a.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021b.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , abs/2101.00190, 2021.

Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In AAAI Conference on Artificial Intelligence , 2020.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 , 2021a.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys (CSUR) , 2022a.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ArXiv , abs/2110.07602, 2021b.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 , 2021c.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. ArXiv , abs/2103.10385, 2021d.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL , 2022b.

Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering , 35:5879-5900, 2021e.

Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. Proceedings of the ACM Web Conference 2023 , 2023.

Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqing Yang. Vision-and-language pretrained models: A survey. IJCAI , 2022.

Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi. Learning to pre-train graph neural networks. In AAAI , 2021.

Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin N. Steijaert, Jörg Kurt Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction on chembl† †electronic supplementary information (esi) available: Overview, data collection and clustering, methods, results, appendix. see doi: 10.1039/c8sc00148k. Chemical Science , 9:5441 - 5451, 2018.

Diego Mesquita, Amauri H. de Souza, and Samuel Kaski. Rethinking pooling in graph neural networks. ArXiv , abs/2010.11418, 2020.

Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. AAAI , 2019.

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gcc: Graph contrastive coding for graph neural network pre-training. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020a.

Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences , 63:1872 - 1897, 2020b.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems , 33:12559-12571, 2020.

Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Conference of the European Chapter of the Association for Computational Linguistics , 2020a.

Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. NAACL , 2020b.

Sterling and John J. Irwin. Zinc 15 - ligand discovery for everyone. Journal of Chemical Information and Modeling , 55:2324 - 2337, 2015.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 , 2019.

Mingchen Sun, Kaixiong Zhou, Xingbo He, Ying Wang, and Xin Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2022.

Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In Neural Information Processing Systems , 2021.

Junjiao Tian, Xiaoliang Dai, Chih-Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. ArXiv , abs/2303.10720, 2023.

Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity. NeurIPS , 2020.

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Lio', Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. ICLR , 2019a.

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Lio', Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. ICLR , 2019b.

Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. ArXiv , abs/2106.09226, 2021.

Junyang Wu, Xianhang Li, Chen Wei, Huiyu Wang, Alan Loddon Yuille, Yuyin Zhou, and Cihang Xie. Unleashing the power of visual prompting at the pixel level. ArXiv , abs/2212.10556, 2022.

Sen Wu, Hongyang Zhang, and Christopher Ré. Understanding and improving information transfer in multi-task learning. ICLR , 2020.

Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: A benchmark for molecular machine learning. arXiv: Learning , 2017.

Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z. Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. Proceedings of the ACM Web Conference 2022 , 2022a.

Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z Li. A survey of pretraining on graphs: Taxonomy, methods, and applications. arXiv preprint arXiv:2202.07893 , 2022b.

Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Classaware visual prompt tuning for vision-language pre-trained model. ArXiv , abs/2208.08340, 2022.

Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning - under the pre-train and fine-tune paradigm. NAACL , 2021a.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR , 2019.

Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning , 2021b.

Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kernels. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2015.

Gilad Yehudai, Ethan Fetaya, Eli A. Meirom, Gal Chechik, and Haggai Maron. From local structures to size generalization in graph neural networks. In ICML , 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. NeurIPS , 2020.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning , pages 12121-12132. PMLR, 2021.

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Bringing your own view: Graph contrastive learning without prefabricated data augmentations. Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , 2022.

Chuxu Zhang, Kaize Ding, Jundong Li, Xiangliang Zhang, Yanfang Ye, N. Chawla, and Huan Liu. Few-shot learning on graphs. In International Joint Conference on Artificial Intelligence , 2022.

Muhan Zhang and Pan Li. Nested graph neural networks. NeurIPS , 2021.

Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV , 2016.

Zhang, Q. Liu, H. Wang, C. Lu, and C. K. Lee. Motif-based graph self-supervised learning for molecular property prediction. 2021a.

Zaixin Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph selfsupervised learning for molecular property prediction. In Neural Information Processing Systems , 2021b.

Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. ICLR , 2022.

Fan Zhou and Chengtai Cao. Overcoming catastrophic forgetting in graph neural networks with experience replay. In AAAI Conference on Artificial Intelligence , 2021.

Marinka Zitnik, Rok Sosi, and Jure Leskovec. Prioritizing network communities. Nature Communications , 9, 2018.

A Extra Materials for Section 3

A.1 Extension to node-wise tasks

In this section, we illustrate the process of graph prompt tuning for node-wise tasks (node classification and link prediction). These tasks differ from graph classification, requiring node-level representations instead of graph-level representations. To bridge this gap, we employ Subgraph GNNs [Cotta et al., 2021, Zhang and Li, 2021, Bevilacqua et al., 2022, Zhao et al., 2022] to capture the graph-level and node-level representations. Subgraph GNN models utilize MPNNs on sets of subgraphs extracted from the original input graph. They subsequently aggregate the resulting representations [Frasca et al., 2022]. As a result, the node representations can also be interpreted as graph representations of the induced subgraphs. Specifically, the node representation h i for node v i can be calculated as:

where f is a Subgraph GNN model, and G i is the induced subgraph for node v i . To obtain the node representations through graph prompt tuning, we freeze the model f and introduce a learnable graph prompt g ϕ : G → G , parameterized by ϕ , as indicated in Formula 2, which can be expressed as:

Once we acquire the node representations, we can seamlessly proceed with downstream node classification or link prediction tasks. Our proposed methods, GPF and GPF-plus, enable the effective execution of node-level tasks through the approach described in Formula 14.

A.2 Proof for Theorem 1

Theorem 1. (Universal Capability of GPF) Given a pre-trained GNN model f , an input graph G : ( A , X ) , an arbitrary prompting function ψ t ( · ) , for any prompted graph ˆ G : ( ˆ A ∈ A , ˆ X ∈ X ) in the candidate space of the graph template G ∗ = ψ t ( G ) , there exists a GPF extra feature vector ˆ p that satisfies:

where ψ t ( G ) = G ∗ : ( A ∗ , X ∗ ) , A and X are the candidate space for A ∗ and X ∗ respectively.

To illustrate Theorem 1, we provide the specific architecture of the pre-trained GNN model f . For analytical simplicity, we initially assume that f is a single-layer GIN [Xu et al., 2019] with a linear transformation. Subsequently, we extend our conclusions to multi-layer models utilizing various transition matrices [Klicpera et al., 2019]. During the generation of graph representations, we first obtain node representations and then employ a readout function to calculate the final graph representations. Existing pre-trained GNN models commonly employ sum or mean pooling for this purpose. Previous research [Mesquita et al., 2020] suggests that complex pooling mechanisms are unnecessary, as simple readout functions can yield superior performance. It is worth mentioning that the subsequent derivations still hold when we use any other weighted aggregation readout functions, such as average pooling, min/max pooling, and hierarchical pooling. Hence, we assume that the concrete architecture of the pre-trained GNN model f can be expressed as:

where W is a linear projection. The parameters ϵ and W have been pre-trained in advance and remain fixed during downstream adaptation. Next, we proceed with the derivation of Theorem 1. In Theorem 1, we impose no constraints on the form of the prompting function ψ t ( · ) , allowing ˆ A and ˆ X to represent the adjacency matrix and feature matrix of any graph. We define the graphlevel transformation g : G → G , which satisfies ( ˆ A , ˆ X ) = g ( A , X ) . Consequently, Theorem 1 is equivalent to Proposition 1.

Proposition 1. Given a pre-trained GNN model f , an input graph G : ( A , X ) , for any graph-level transformation g : G → G , there exists a GPF extra feature vector ˆ p that satisfies:

To further illustrate the graph-level transformation g ( · ) , we decompose it into several specific transformations.

Proposition 2. Given an input graph G = { A , X } , an arbitrary graph-level transformation g : G → G can be decoupled to a series of following transformations:

Feature transformations. Modifying the node features and generating the new feature matrix X ′ = g ft ( X ) .

Link transformations. Adding or removing edges and generating the new adjacency matrix A ′ = g lt ( A ) .

Isolated component transformations. Adding or removing isolated components (sub-graphs) and generating the new adjacency matrix and feature matrix A ′ , X ′ = g ict ( A , X ) .

Here, the word 'isolated' refers to a component (sub-graph) that does not link with the rest of the graph. Proposition 2 suggests that an arbitrary graph-level transformation is a combination of the three transformations mentioned above. For instance, the deletion of a node in the initial graph can be decomposed into two steps: removing its connected edges and then removing the isolated node.

Proposition 3. Given a pre-trained GNN model f , an input graph G : ( A , X ) , for any feature transformation g ft : g ft ( X ) = X ′ , there exists a GPF extra feature vector ˆ p that satisfies:

Proof. We set ∆ X = X ′ -X . Then, we have:

For GPF p = [ α 1 , · · · , α F ] ∈ R 1 × F , we can perform a similar split:

where [1] N ∈ R N × 1 denotes a column vector with N 1 s, [ d i +1+ ϵ ] N ∈ R N × 1 denotes a column vector with the value of i -th row is d i +1+ ϵ and d i represents the degree number of v i . To obtain the same graph representation h G , we have:

where Sum ( M ) = ∑ i m i denotes the operation that calculates the sum vector for each row in the matrix. We can further simplify the above equation as:

where the results of (( A + (1 + ϵ ) · I ) · ∆ X ) ∈ R N × F ,and the frozen linear transformation W ∈ R F × F ′ . We first calculate ∆ h G ,p = Sum ([ d i +1+ ϵ ] N · p · W ) ∈ R F ′ . We can obtain that:

where h i G ,p denotes the value of the i -th dimension in h G ,p , D = ∑ N k =1 d k denotes the total degree of all nodes in the whole graph, α j , j ∈ [1 , F ] denotes the j -th learnable parameter in GPF p , and W j,i , j ∈ [1 , F ] , i ∈ [1 , F ′ ] denotes the frozen parameter in W . As for ∆ h G ,ft = Sum (( A +(1+ ϵ ) · I ) · ∆ X · W ) , we assume ( A +(1+ ϵ ) · I ) · ∆ X = B ∈ R N × F . Then we have:

where β k,j , k ∈ [1 , N ] , j ∈ [1 , F ] denotes the learnable parameter in B . According to above analysis, to obtain a same graph representation h G , ˆ p with a certain h G ,ft , we have:

Therefore, for an arbitrary feature transformation g ft , there exists a GPF ˆ p that satisfies the above conditions and can obtain the exact graph representation for the pre-trained GNN model f .

Proposition 3 demonstrates the comprehensive coverage of our proposed GPF for all graph-level feature transformations. GPF introduces a uniform feature modification p ∈ R F to each node in the graph. However, it can achieve an equivalent effect to adding independent feature modifications to each node individually under the pre-trained GNN model described above.

Proposition 4. Given a pre-trained GNN model f , an input graph G : ( A , X ) , for any link transformation g lt : g lt ( A ) = A ′ , there exists a GPF extra feature vector ˆ p that satisfies:

Proof. The proof of Proposition 4 is similar to that of Proposition 3. We set ∆ A = A ′ -A . It is worth mentioning that A , A ′ ∈ { 0 , 1 } N × N and ∆ A ∈ {-1 , 0 , 1 } N × N , which means they are of the same size, the values of A , A ′ can only be 0 or 1 , and the values of ∆ A can be -1 , 0 or 1 . We have:

From the proof of Proposition 3, we obtain:

where p = [ α 1 , · · · , α F ] ∈ R 1 × F denotes our learnable GPF, [ d i + 1 + ϵ ] N ∈ R N × 1 denotes a column vector with the value of i -th line is d i +1+ ϵ and d i represents the degree number of v i . With ∆ h G ,p = Sum ([ d i +1+ ϵ ] N · p · W ) ∈ R F ′ , we can obtain:

where h i G ,p denotes the value of the i -th dimension in h G ,p , D = ∑ N k =1 d k denotes the total degree of all nodes in the whole graph, α j , j ∈ [1 , F ] denotes the j -th learnable parameter in GPF p , and W j,i , j ∈ [1 , F ] , i ∈ [1 , F ′ ] denotes the frozen parameter in W . As for ∆ h G ,lt = Sum (∆ A · X · W ) , we have:

where ∆ a k,l , k ∈ [1 , N ] , l ∈ [1 , N ] denotes the element of ∆ A , and x l,j , l ∈ [1 , N ] , j ∈ [1 , F ] denotes the element of X . To obtain a same graph representation h G , ˆ p with a certain h G ,lt , we have:

Therefore, for an arbitrary link transformation g lt , there exists a GPF ˆ p that satisfies above conditions and can obtain the same graph representation for pre-trained GNN model f .

Proposition 4 demonstrates that GPF can also encompass all link transformations. Intuitively, link transformations are closely associated with changes in the adjacency matrix, which are independent of node features. However, our findings reveal that, within the architecture of most existing GNN models, modifications in the feature space and modifications in the structural space can produce equivalent effects.

Proposition 5. Given a pre-trained GNN model f , an input graph G : ( A , X ) , for any isolated component transformation g ict : g ict ( A , X ) = A ′ , X ′ , there exists a GPF extra feature vector ˆ p that satisfies:

Proof. Unlike feature transformations and linear transformations, isolated component transformations will change the number of nodes in the graph, which means the scale of modified A ′ and X ′ is uncertain. We first express the isolated component transformation in more details. The adjacency matrix A and feature matrix X can be divided into several isolated components, which can be expressed as:

Removing an isolated component C k = { A k , X k } , k ∈ [1 , m ] means removing both A k in the adjacency matrix and corresponding X k in the feature matrix. Adding a new isolated component C m + l = { A m + l , X m + l } , l ≥ 1 means adding A m + l to the adjacency matrix A , and adding X m + l to the corresponding position of X . Then we have:

To align with the proofs of Proposition 3 and Proposition 4, we set ∆ h G ,ist = h G ,ist -Sum (( A + (1 + ϵ ) · I ) · X · W ) , and it can be expressed as:

where I k is an indicator that satisfies:

From the proof of Proposition 3, we have following conclusions:

where p = [ α 1 , · · · , α F ] ∈ R 1 × F denotes our learnable GPF, [ d i + 1 + ϵ ] N ∈ R N × 1 denotes a column vector with the value of i -th line is d i +1+ ϵ and d i represents the degree number of v i .

With ∆ h G ,p = Sum ([ d i +1+ ϵ ] N · p · W ) ∈ R F ′ , we can obtain:

where h i G ,p denotes the value of the i -th dimension in h G ,p , D = ∑ N k =1 d k denotes the total degree of all nodes in the whole graph, α j , j ∈ [1 , F ] denotes the j -th learnable parameter in GPF p , and W j,i , j ∈ [1 , F ] , i ∈ [1 , F ′ ] denotes the frozen parameter in W . To obtain a same graph representation h G , ˆ p with a certain h G ,ist , we have:

where X j denotes the j -th column of the matrix X . Therefore, for an arbitrary isolated component transformation g ict , there exists a GPF ˆ p that satisfies above conditions and can obtain the same graph representation for pre-trained GNN model f .

The isolated component transformation possesses the capability to alter the scale of a graph, and previous studies have paid limited attention to this type of graph-level transformation.

Proposition 6. Given a pre-trained GNN model f , an input graph G : ( A , X ) , for a series of transformations g = { g 1 , g 2 , · · · , g k } composed of g ft , g lt and g ist , there exists a GPF extra feature vector ˆ p that satisfies:

Proof. Without loss of generality, we consider g = { g 1 , g 2 } with two transformations described in Proposition 2. Now we prove there exists a GPF ˆ p that satisfies:

We assume g 1 ( A , X ) = A ′ , X ′ . According to Proposition 3, 4, 5, there exists a GPF ˆ p 1 that satisfies:

Therefore, there is a ˆ p = ˆ p 1 + ˆ p 2 that satisfies:

Based on the preceding analysis, we have established that the GPF can replicate the effects of any graph-level transformation on the pre-trained GNN model defined by Formula 15 and 16. Hence, we have successfully demonstrated Proposition 1 and Theorem 1 within the framework of the simple model architecture. Next, we aim to generalize our findings to more intricate scenarios.

Extension to other GNN backbones. We use GIN Xu et al. [2019] as the default backbone model in our previous derivation. When replacing GIN with another GNN model, only slight modifications are required to ensure that all propositions remain valid. As is described in Klicpera et al. [2019], various GNN models can be represented as:

and a ˆ p 2 that satisfies:

where S ∈ R N × N denotes the diffusion matrix ( e.g. , A +(1+ ϵ ) · I is the diffusion matrix for GIN), and W denotes the linear projection. In this case, we modify the Formula 26, 42 and 54 as follows:

where s i,j is the element of the diffusion matrix S . With these modifications, Propositions 3, 4, and 5 remain valid.

Extension to multi-layer models. For analytical simplification, similar to many previous works, we consider multi-layer GNN models without non-linear activation functions between layers, which can be expressed as:

where S ( k ) is the diffusion matrix described as Formula 65 of the k -th layer, and W ( k ) is the linear projection of the k -th layer. With such architecture, the node representations of the k -th layer H ( k ) can also be obtained as:

where S ′ = ∏ k i =1 S ( i ) and W ′ = ∏ k i =1 W ( i ) . By substituting S = S ′ and W = W ′ in Formula 65, we can find that Proposition 3, 4 and 5 still hold true.

A.3 Proof for Theorem 2

Theorem 2. (Effectiveness Guarantee of GPF) For a pre-trained GNN model f , a series of graphs D = { ( G 1 : ( A 1 , X 1 ) , y 1 ) , . . . , ( G m : ( A m , X m ) , y m ) } under the non-degeneracy condition, and a linear projection head θ , there exists Y ′ = { y ′ 1 , . . . , y ′ m } for y 1 = y ′ 1 , . . . , y m = y ′ m that satisfies:

To demonstrate Theorem 2, we need to provide a more detailed description of the architecture of the GNN model f . We assume that the graph representations h G i for G i are obtained through the following process:

where S i ∈ R N i × N i denotes the diffusion matrix of G i as Formula 65, N i denotes the node number of the graph G i , W ∈ R F × F ′ denotes the linear projection, F is the dimension of node features, and Sum ( M ) = ∑ i m i denotes the operation that calculates the sum vector for each row in the matrix M . When we employ our proposed GPF in the above model f , the graph representations h GPF G i are calculated as:

where [1] T is a column vector with all 1's, and p ∈ R 1 × F is the extra learnable vector of GPF. The squared regression loss l can be expressed as:

where θ ∈ R F ′ × 1 is a linear projection head. In such case, the optimal tuning loss of fine-tuning l FT and GPF l GPF can be expressed as:

Before we prove Theorem 2, we first illustrate the following proposition.

Proposition 7. Given a series of graphs D = {G 1 : ( A 1 , X 1 ) , . . . , G m : ( A m , X m ) } and a linear projection ˆ W , there exist ˆ p ∈ R 1 × F , ˆ θ ∈ R F ′ × 1 , and δ > 0 that satisfy:

For analytical simplification, we gather all the unique node features in D into a set S X = { x 1 , · · · , x l } . Consequently, the node feature matrix X i of any graph G i can be constructed from elements in the set S X . Next, we proceed to expand the function Sum ( · ) as:

where S i, ( j,k ) denotes the element in the j -th row and k -th column of S i , and X i, ( k, :) denotes the k -th row vector of X i . In order to express the representations of different graphs in a unified form, we rewrite the above formulas to calculate the graph representations h G i and h GPF G i from the node representations in S X :

where c i,j is the coefficient of x j for the graph G i , which is calculated as c i,j = ∑ k ∑ N i j ′ S i, ( j ′ ,k ) for all k 's that satisfy X i, ( k, :) = x j . We can also get that ∑ l j c i,j = ∑ N i k ∑ N i j ′ S i, ( j ′ ,k ) . Then, we rewrite Formula 78 into a matrix form as follows:

where X S X ∈ R l × F denotes the feature matrix for S X that satisfies X S X , ( j, :) = x j , C ∈ R m × l denotes the coefficient matrix and the element in the i -th row and k -th column C ( i,j ) is equal to c i,j , [ ∑ j C ( i,j ) ] T denotes a column vector with the value of i -th row is ∑ j C ( i,j ) . We represent W ′ · θ ′ = ˆ W · ˆ θ +∆ W · θ , then we have:

As described by the condition of Proposition 7, W ′ and θ ′ can be chosen arbitrarily, which means ∆ W · θ can be equal to any v ∈ R F × 1 . Therefore, Proposition 7 can be reformed as below.

Given a series of graphs D = {G 1 : ( A 1 , X 1 ) , . . . , G m : ( A m , X m ) } and a linear projection ˆ W , there exist ˆ p ∈ R 1 × F , ˆ θ ∈ R F ′ × 1 , and δ > 0 that satisfy:

̸

for any v ∈ R F × 1 .

We make the assumption that C is a column full-rank matrix, which implies that there is no uniform feature distribution shared among different graphs, aligning with real-world scenarios. It is important to note that W is pre-trained beforehand, and for any ˆ p and ˆ θ satisfying ˆ p · ˆ W · ˆ θ = 0 , the nondegeneracy condition for Formula 86 is as follows:

̸

where [1] T ∈ R m × 1 is a column vector with all 1's. Therefore, Formula 86 and Proposition 7 hold for all X S X for which X S X · v ′ = [1] T has no solution v ′ .

̸

Finally, we revisit Theorem 2. Given a series of graphs under the non-degeneracy condition, a pretrained linear projection ˆ W , ˆ p and ˆ θ that satisfy ˆ p · ˆ W · ˆ θ = 0 , we can construct Y ′ = { y ′ 1 , . . . , y ′ m } as:

Under these conditions, the optimal theoretical tuning results for GPF and fine-tuning can be expressed as::

Here, l GPF can be achieved with p = ˆ p and θ = ˆ θ , while l FT consistently remains greater than 0, as stated in Proposition 7. Therefore, we can conclude that l GPF < l FT, thus establishing the proof of Theorem 2.

B More Information on Experiments

B.1 Details of the datasets

Dataset overview We utilize the datasets provided by Hu et al. [2020a] for our pre-training datasets. These datasets consist of two domains: chemistry and biology. The chemistry domain dataset comprises 2 million unlabeled molecules sampled from the ZINC15 database [Sterling and Irwin, 2015], along with 256K labeled molecules obtained from the preprocessed ChEMBL dataset [Mayr et al., 2018, Gaulton et al., 2012]. On the other hand, the biology domain dataset includes 395K unlabeled protein ego-networks and 88K labeled protein ego-networks extracted from PPI networks. For the models pre-trained on the chemistry dataset, we employ eight binary graph classification datasets available in MoleculeNet [Wu et al., 2017] as downstream tasks. As for the models pretrained on the biology dataset, we apply the pre-trained models to 40 binary classification tasks, with each task involving the prediction of a specific fine-grained biological function.

Pre-training datasets. The datasets provided by Hu et al. [2020a] consist of two distinct datasets: Biology and Chemistry, corresponding to the biology domain and chemistry domain, respectively. The Biology dataset contains 395K unlabeled protein ego-networks obtained from PPI networks of 50 species. These networks are used for node-level self-supervised pre-training. Additionally, 88K labeled protein ego networks serve as the training data for predicting 5000 coarse-grained biological functions. This graph-level multi-task supervised pre-training aims to predict these functions jointly. Regarding the Chemistry dataset, it comprises 2 million unlabeled molecules sampled from the ZINC15 database [Sterling and Irwin, 2015]. These molecules are utilized for node-level selfsupervised pre-training. For graph-level multi-task supervised pre-training, a preprocessed ChEMBL dataset [Mayr et al., 2018, Gaulton et al., 2012] is employed. This dataset contains 456K molecules and covers 1310 different biochemical assays.

Downstream datasets. The statistics of the downstream datasets utilized for the models pre-trained on Biology and Chemistry are presented in Table 3.

B.2 Details of pre-training strategies

We adopt five widely used strategies (tasks) to pre-train the GNN models, which are listed as below:

Deep Graph Infomax (denoted by Infomax). It is first proposed by Velickovic et al. [2019a]. Deep graph infomax obtains expressive representations for graphs or nodes via maximizing the mutual information between graph-level representations and substructure-level representations of different granularity.

Edge Prediction (denoted by EdgePred). It is a regular graph reconstruction task used by many models, such as GAE [Kipf and Welling, 2016a]. The prediction target is the existence of edge between a pair of nodes.

Attribute Masking (denoted by AttrMasking). It is proposed by Hu et al. [2020a]. It masks node/edge attributes and then let GNNs predict those attributes based on neighboring structure.

Context Prediction (denoted by ContextPred). It is also proposed by Hu et al. [2020a]. Context prediction uses subgraphs to predict their surrounding graph structures, and aims to mapping nodes appearing in similar structural contexts to nearby embeddings.

Graph Contrastive Learning (denoted by GCL). It embeds augmented versions of the anchor close to each other (positive samples) and pushes the embeddings of other samples (negatives) apart. We use the augmentation strategies proposed in You et al. [2020] for generating the positive and negative samples.

To pre-train our models, we follow the training steps outlined in Hu et al. [2020a] for Infomax, EdgePred, AttrMasking, and ContextPred tasks. We then perform supervised graph-level property prediction to enhance the performance of the pre-trained models further. For models pre-trained using GCL, we follow the training steps detailed in You et al. [2020].

B.3 Results of few-shot graph classification

The results for 50-shot scenarios. Table 4 summarizes the results for 50-shot graph classification.

The results for 100-shot scenarios. We also conducted experiments where the number of training samples in the downstream task was limited to 100 for both the chemistry and biology datasets. The summary of the overall results can be found in Table 5. The experimental findings align with those observed in the 50-shot scenarios. Our proposed graph prompt tuning method achieves the best results in 42 out of 45 cases (14 out of 45 for GPF and 28 out of 45 for GPF-plus). Additionally, the average results of both GPF and GPF-plus surpass the average results of fine-tuning across all pre-training strategies, showcasing the superiority of our proposed graph prompt tuning approach.

B.4 Parameter efficiency analysis

We have computed the number of tunable parameters for full fine-tuning, GPF, and GPF-plus (excluding the task-specific projection head θ ) on the chemistry and biology datasets. The statistics are presented in Table 6. The results indicate that the number of tunable parameters in GPF and GPF-plus is several orders of magnitude smaller than that of fine-tuning. Specifically, GPF utilizes no more than 0 . 02% of the tunable parameters used in fine-tuning, while GPF-plus utilizes no more than 0 . 7% of the tunable parameters used in fine-tuning. Our proposed graph prompt tuning methods

exhibit significant advantages in terms of parameter efficiency compared to fine-tuning. It leads to reduced training time and storage space required for downstream adaptations.

B.5 Comparison with linear probing

In the field of Natural Language Processing and Computer Vision, linear probing [Kumar et al., 2022, Wu et al., 2020, Tripuraneni et al., 2020, Du et al., 2020] is a widely employed method for adapting pre-trained models to downstream tasks. This approach involves freezing the parameters of the pre-trained model f and solely optimizing the linear projection head θ . To evaluate the effectiveness of linear probing, we conducted experiments on the chemistry datasets Toxcast and SIDER, and the results are summarized in Table 7. It is evident from the results that linear probing exhibits a significant performance decline compared to fine-tuning and our proposed graph prompt tuning. The primary distinction between linear probing and our proposed graph prompt tuning lies in the incorporation of an additional learnable graph prompt g ϕ ( · ) in the input space. The substantial

performance gap observed between these approaches underscores the importance of integrating a graph prompt for the effective adaptation of pre-trained models.

B.6 Comparison with other tuning methods

We also compare our proposed graph prompt tuning with other tuning methods described as follows:

PARTIALk : We tune the last k layers of the pre-trained model f with a projection head θ and freeze other parts, which is utilized in Zhang et al. [2016], He et al. [2021], Jia et al. [2022].

MLPk : We freeze the pre-trained model f and utilize a multi-layer perceptron (MLP) with k layers as the projection head to perform the classification.

We conduct the experiments on the biology datasets (PPI), and Table 8 summarizes the results. The experimental results indicate that our methods outperform other tuning methods in all cases.

B.7 Extra results on GCC

Another popular pre-training strategy for graph contrastive learning involves following the training steps outlined in Qiu et al. [2020a]. First, we introduce the datasets utilized for pre-training GCC. The self-supervised pre-training task of GCC is conducted on six graph datasets, and Table 9 provides detailed statistics for each dataset. For the downstream tasks of the pre-trained GCC, we employ

IMDB-BINARY and IMDB-MULTI datasets [Yanardag and Vishwanathan, 2015]. Each dataset consists of a collection of graphs associated with specific target labels. We evaluate GPF on these datasets, and the results are presented in Table 10. The experimental findings consistently demonstrate that GPF outperforms fine-tuning when adapting models pre-trained using the GCC strategy.

B.8 Hyper-parameter settings

This section presents the hyper-parameters used during the adaptation stage of pre-trained GNN models on downstream tasks for our proposed graph prompt tuning. Table 11 summarizes the hyper-parameter settings. You can also visit our code repository to obtain the specific commands for reproducing the experimental results.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a universal prompt-based tuning method called Graph Prompt Feature (GPF) and its variation (GPF-plus) for pre-trained Graph Neural Network (GNN) models. GPF is a universal method that can be applied to any pre-trained GNN model under any pre-training strategy. It operates on the input graph's feature space and can achieve an equivalent effect to any form of prompting function. \nGPF introduces a learnable vector $p$ of dimension $F$, which is added to the node features, where $F$ corresponds to the dimensionality of the node features. The authors also purpose GPF-plus, which assigns an independent learnable vector $p_i$ to each node $u_i$ in the graph, instead of a single vector $p$. Experimental results show that GPF outperforms fine-tuning, with an average improvement of about 1.4% in full-shot scenarios and 3.2% in few-shot scenarios.\n\nSTRENGTHS:\n1) Theoretical Guarantees: The authors provide rigorous derivations and theoretical analyses to demonstrate the universality and effectiveness of GPF. They prove that GPF can achieve an equivalent effect to any form of prompting function and can outperform fine-tuning in certain cases. This theoretical foundation strengthens the credibility of their proposed method.\n\n2) Experimental Validation - Reproducibility: The paper includes extensive experiments conducted under various pre-training strategies, including both full-shot and few-shot scenarios. The experimental results consistently show that GPF outperforms fine-tuning, achieving an average improvement of about 1.4% in full-shot scenarios and 3.2% in few-shot scenarios. Furthermore, GPF surpasses existing specialized prompt-based tuning methods designed for specific pre-training strategies. Moreover, the availability of the source code enhances the reproducibility of the experiments.\n\nWEAKNESSES:\n1) A weakness of the paper is the lack of a clear explanation or motivation regarding why the addition of the learnable vector p to the input features in the GPF model leads to better results compared to linear probing with a trainable layer in the final layer. While the Appendix demonstrates the superior performance of GPF, the paper does not provide a comprehensive analysis or reasoning behind this improvement. The absence of a clear explanation may leave readers questioning the underlying mechanisms and factors that contribute to the observed performance gain. Without a proper understanding of the motivations and justifications for the proposed approach, it becomes challenging to assess the significance and generalizability of the findings. \n\n2) Theorem 2 assumes a simple linear 1-layer GNN (without activations). While the theorem provides theoretical insights into the convergence properties of this specific type of GNN, it may not accurately capture the behavior of more complex GNN architectures commonly used in practice. In real-world applications, GNNs often incorporate activation functions to introduce non-linearity and capture more intricate patterns in graph data. By focusing solely on a linear 1-layer GNN without activations, the theorem may limit the generalizability of its conclusions and overlook important aspects of GNN models commonly used in practical scenarios. Therefore, the applicability of the theorem's findings to more sophisticated GNN architectures with non-linear activations remains uncertain.\n\n3) One weakness of the paper is the exclusive use of Graph Isomorphism Network (GIN) as the backbone GNN for finetuning. While GIN is a widely used GNN architecture, it may not be the most expressive or optimal choice for every graph-related task[1,2]. The paper does not provide a comprehensive exploration or comparison of different backbone GNN architectures in the finetuning process. Moreover, maybe a more powerful GNN would be able to achieve better performance in the fine-tuning stage. Therefore, it would be interesting for the authors to examine if their approach leads also to better results when a more powerful GNN is used. \n\n[1] Frasca, Fabrizio, et al. \"Understanding and extending subgraph gnns by rethinking their symmetries.\" Advances in Neural Information Processing Systems 35 (2022): 31376-31390.\n[2]  Morris, Christopher, et al. \"Weisfeiler and leman go neural: Higher-order graph neural networks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.\n\nQUESTIONS:\n1) Explanation of the relationship between GPF and linear probing: The paper compares GPF with linear probing, which involves a trainable layer in the final layer. It would be beneficial if the authors could discuss the relationship between these two approaches and explain why GPF, with its learnable vector added to the input features, achieves superior performance compared to linear probing. Is there any inherent advantage or characteristic of GPF that enables it to outperform linear probing?\n\n2) Generalizability of Theorem 2: The paper presents Theorem 2, which focuses on a simple linear 1-layer GNN without activations. However, many practical GNN architectures incorporate activation functions to introduce non-linearity. It would be helpful if the authors could discuss the generalizability of the theorem's conclusions to more complex GNN models commonly used in real-world applications. Can the findings of Theorem 2 be extended to GNNs with non-linear activations? If not, what are the limitations or implications of the theorem in the context of practical GNN architectures?\n\n3) Have you considered or experimented with using more powerful GNN architectures, such as subgraph GNNs or higher order GNNs, as the backbone GNN for the finetuning stage? If so, how does the performance of GPF and GPF-plus compare when utilizing these alternative GNN architectures?\n\nI will be happy to increase my score for the paper, if the authors adequately address the mentioned weaknesses and provide satisfactory explanations and improvements in response to the questions and suggestions raised. \n\n\n*** After rebuttal increased score to \"Borderline Accept\" **\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposed a universal prompt-based tuning method, called GPF, for pre-trained GNN models. The idea is to operate on the feature space of the downstream input graph. The authors theoretically showed that GPF can achieve results equivalent to any prompting function, and is not weaker than full fine-tuning. Experiments showed the competitive results.

STRENGTHS:
1. This paper proposed a universal graph prompting approach, which basically operates on the input graph feature space. As far as I know, the idea of augmenting input graph node features for prompting is novel.

2. The authors provide theoretical analysis, which showed that GPF can achieve results equivalent to any prompting function, and is not weaker than full fine-tuning.

3. Experiments results showed the performance gains.

WEAKNESSES:
1. The motivation for a universal prompting needs better explained, and advantages over specialized prompting approaches. 

2. Comparison with other prompting meothds is only perfomed on the models pre-trained by edge predcition.

Some minor comments:
1. It is not clear to me how the avg. in Table 1 and Table 2 obtained, and what does it mean. It is just the average across all datasets?
2. I suggest the authors include the detailed pretraining settings in experiments (e.g., datasets used) instead of in the appendix.
3. Fig. 1 is not informative. The authors should use a figure to better illustrate how GPF works, and the main differences with existing prompting approaches. 
4. Table captions should be on top of tables insead of below.

QUESTIONS:
See the weaknesses.

LIMITATIONS:
Potential limitations of the proposed GPF not discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims for efficient adaptation of pre-trained graph neural networks to downstream tasks. A simple prompt tuning method (i.e. GPF) is proposed for adaptation and is applicable to GNN pretrained with any objectives. The main idea of GPF is to add a learnable vector on all the node features in the input graph, while an improved version (i.e. GPF-plus) leverages a set of learnable vectors to compose the bias added to each node feature. Experiments show that GPF and GPF-plus achieve better results than the full finetuning.\n\nSTRENGTHS:\nStrength \n1.\tFigure 1 illustrates the difference between the proposed method and existing methods.\n\n2.\tProvide a theoretical analysis of the proposed method.\n\n3.\tThe proposed method is simple and easy to understand.\n\nWEAKNESSES:\nMajor\n1.\tThe novelty of GSF may be limited because it just adds a bias term on the input of the pretrained network and there is nothing specific for the graph structure. However, the bias-term finetuning is not a new idea [a, b].\n\n2.\tThe authors claim that the developed method is universal for all the graph learning. However, the theoretical proof is only valid for graph classification downstream tasks where the node features are sum-pooled as the representation. There is no evidence showing that GSF is applicable to other types of downstream tasks.\n\n3.\tBaselines missing in Table 1 & 2. For example, Linear Probing is a standard baseline, and residual adapter in [c] can also be a baseline.\n\n4.\tGiven the variance shown in Table 1 & 2, the improvement of GPF-plus over GPF is marginal. Why is it an enhanced version? If GPF is theoretically feasible to address the problem (section 3.4), why there GPF-plus is needed?\n\n[a] BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. ACL’22.\n[b] Tinytl: Reduce memory, not parameters for efficient on-device learning. NeurIPS’20.\n[c] CLIP-Adapter: Better Vision-Language Models with Feature Adapters.\n\nQUESTIONS:\nThe author is suggested to address the concerns in the weakness section.\n\nLIMITATIONS:\nLimitation is not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Graph Prompt Feature (GPF) approach, which aims to adapt pre-trained Graph Neural Networks (GNNs) for downstream tasks by appending tunable embeddings onto the frozen node embeddings. By doing so, the authors achieve a significant reduction in the number of parameters required for the downstream task compared to full fine-tuning. The experimental results demonstrate promising performance in binary classification tasks on Molecular and Protein datasets when compared to full fine-tuning.\n\nSTRENGTHS:\nThe paper is clearly written, the contribution is easy to understand and provide theoretical proof on effectiveness.\n\nWEAKNESSES:\nThere are several concerns that I would like to see addressed:\n\n1) In the formulation of GPF, its relation to the concept of prompting seems unclear. It is challenging to connect this approach to the motivating concept of prompting in NLP. Instead, GPF appears to resemble task-specific adaptive fine-tuning in transformers, such as LoRA[1]. It would be helpful if the authors could elaborate more on where the similarity between GPF and prompting lies.\n\n2) The choice of downstream tasks seems to be limited to small binary classification tasks, and the comparison is only made against fine-tuning as a baseline. Why are there no results presented for multi-class classification tasks? This decision weakens the potential of the proposed method.\n\n3) Unlike previous graph prompt methods, GPF does not make any modifications to the graph's structure. It would be beneficial if the authors could explain how graph classification highlights the strength of GPF, as the additional results provided may seem somewhat redundant.\n\n4) There is no detail provided on the number of features appended into the embedding space for each node. Since GPF-plus suggests a simple size range, does this mean that the authors used different settings for GPF-plus for each dataset? This is not discussed in the experimental settings and might make the comparison somewhat incomparable.\n\n[1] https://arxiv.org/abs/2106.09685\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nThe author did not address any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Graph Prompt Feature (GPF) to improve Graph Neural Networks (GNNs) performance amidst scarce labeled data and low out-of-distribution generalization. GPF, a universal prompt-based tuning method, operates on the input graph's feature space and is applicable to any GNN architecture. The authors also present a stronger variant, GPF-plus, providing diverse prompted features for different nodes. Both methods show better performance than fine-tuning, validated through theoretical analyses and extensive experiments.\n\nSTRENGTHS:\n1. GPF and GPF-plus are universal, model-agnostic solutions that can be applied to any pre-trained GNN model, enhancing their general applicability in diverse scenarios.\n2. The paper provides theoretical guarantees for the effectiveness of GPF and GPF-plus, strengthening the scientific rigor of the presented methods.\n3. The authors demonstrate that their proposed methods outperform existing fine-tuning strategies and specialized prompt-based tuning methods in both full-shot and few-shot scenarios, showcasing their practical effectiveness.\n\nWEAKNESSES:\n1. The effectiveness of GPF and GPF-plus largely depends on the quality and representation of the feature space, which may not be optimally prepared in all real-world applications.\n2. While the paper provides theoretical analysis for the methods' effectiveness, more detailed explanation or examples of the theoretical proofs could enhance the clarity and comprehensibility.\n3. The real-world applicability and performance of the proposed methods could be further substantiated with experiments on a wider range of tasks and datasets.\n\nQUESTIONS:\n1. How do GPF and GPF-plus perform in scenarios where the feature space is noisy or inadequately represented? Are there strategies to mitigate these potential issues?\n2. Could you provide more detailed explanation or real-world examples to further illustrate the theoretical guarantees of GPF and GPF-plus?\n3. How would GPF and GPF-plus integrate with other advancements in GNNs or machine learning in general?\n\nLIMITATIONS:\nNot really.  Dear authors, please enumerate some limitations of your work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}