SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 1. (Universal Capability of GPF) Given a pre-trained GNN model  $f$ , an input graph  $\mathcal{G}$ :  $(\mathbf{A},\mathbf{X})$ , an arbitrary prompting function  $\psi_t(\cdot)$ , for any prompted graph  $\hat{\mathcal{G}}$ :  $(\hat{\mathbf{A}} \in \mathbb{A},\hat{\mathbf{X}} \in \mathbb{X})$  in the candidate space of the graph template  $\mathcal{G}^* = \psi_t(\mathcal{G})$ , there exists a GPF extra feature vector  $\hat{p}$  that satisfies:
$$
f (\mathbf {A}, \mathbf {X} + \hat {p}) = f (\hat {\mathbf {A}}, \hat {\mathbf {X}}) \tag {11}
$$
The complete proof of Theorem 1 can be found in the appendix. Theorem 1 implies that GPF can achieve the theoretical performance upper bound of any prompting function described in Formula 3 and 4. Specifically, if optimizing the graph template  $\mathcal{G}^*$  generated by a certain prompting function  $\psi_t(\cdot)$  can yield satisfactory graph representations, then theoretically, optimizing the vector  $p$  of GPF can also achieve the exact graph representations. This conclusion may initially appear counterintuitive since GPF only adds learnable components to node features without explicitly modifying the graph structure. The key lies in understanding that the feature matrix  $\mathbf{X}$  and the adjacency matrix  $\mathbf{A}$  are not entirely independent during the processing. The impact of graph structure modifications on the final graph representations can also be obtained through appropriate modifications to the node features. Therefore, GPF and GPF-plus, by avoiding the explicit illustration of the prompting function  $\psi_t(\cdot)$ , adopt a simple yet effective architecture that enables them to possess universal capabilities in dealing with pre-trained GNN models under various pre-training strategies.
Next, we make guarantee of the effectiveness of GPF and demonstrate that GPF is not weaker than fine-tuning, which means GPF can achieve better theoretical tuning results in certain situations compared to fine-tuning. In Natural Language Processing, the results obtained from fine-tuning are generally considered the upper bound for prompt tuning results [Lester et al., 2021a, Liu et al., 2021b, Ding et al., 2022]. It is intuitive to believe that fine-tuning, which allows for more flexible and comprehensive parameter adjustments in the pre-trained model, can lead to better theoretical results during downstream adaptation. However, in the graph domain, the architecture of graph neural networks magnifies the impact of input space transformation on the final representations to some extent. To further illustrate this point, following previous work [Kumar et al., 2022, Tian et al., 2023, Wei et al., 2021], we assume that the downstream task utilizes the squared regression loss  $l = \sum_{i}(\dot{y}_i - y_i)^2$ .
Theorem 2. (Effectiveness Guarantee of GPF) For a pre-trained GNN model  $f$ , a series of graphs  $\mathcal{D} = \{(\mathcal{G}_1: (\mathbf{A}_1, \mathbf{X}_1), y_1), \ldots, (\mathcal{G}_m: (\mathbf{A}_m, \mathbf{X}_m), y_m)\}$  under the non-degeneracy condition, and a
linear projection head  $\theta$ , there exists  $\mathcal{Y}' = \{y_1', \ldots, y_m'\}$  for  $y_1 = y_1'$ ,  $\ldots$ ,  $y_m = y_m'$  that satisfies:
$$
l _ {\mathrm {G P F}} = \min  _ {p, \theta} \sum_ {i} ^ {m} \left(f \left(\mathbf {A} _ {i}, \mathbf {X} _ {i} + p\right) \cdot \theta - y _ {i}\right) ^ {2} <   l _ {\mathrm {F T}} = \min  _ {f, \theta} \sum_ {i} ^ {m} \left(f \left(\mathbf {A} _ {i}, \mathbf {X} _ {i}\right) \cdot \theta - y _ {i}\right) ^ {2} \tag {12}
$$
The detailed proof of Theorem 2 and the description of the degeneracy condition can be found in the appendix. Theorem 2 indicates that GPF obtains a lower minimum loss compared to fine-tuning in certain scenarios, demonstrating its ability to achieve better theoretical tuning results.
Model Architecture and Datasets. We adopt the widely used 5-layer GIN [Xu et al., 2019] as the underlying architecture for our models, which aligns with the majority of existing pre-trained GNN models [Xia et al., 2022b, Hu et al., 2020a, Qiu et al., 2020a, You et al., 2020, Suresh et al., 2021, Xu et al., 2021b, Zhang et al., 2021b, You et al., 2022, Xia et al., 2022a]. As for the benchmark datasets, we employ the chemistry and biology datasets published by Hu et al. [2020a]. A comprehensive description of these datasets can be found in the appendix.
Pre-training Strategies. We employ five widely used strategies (tasks) to pre-train the GNN models, including Deep Graph Infomax (denoted by Infomax) [Velickovic et al., 2019a], Edge Prediction (denoted by EdgePred) [Kipf and Welling, 2016a], Attribute Masking (denoted by AttrMasking) [Hu et al., 2020a], Context Prediction (denoted by ContextPred) [Hu et al., 2020a] and Graph Contrastive Learning (denoted by GCL) [You et al., 2020]. A detailed description of these pre-training strategies can be found in the appendix.
Tuning Strategies. We adopt the pre-trained models to downstream tasks with different tuning strategies. Given a pre-trained GNN model  $f$ , a task-specific projection head  $\theta$
- Fine Tuning (denoted as FT). We tune the parameters of the pre-trained GNN model  $f$  and the projection head  $\theta$  simultaneously during the downstream training stage.
- Graph Prompt Feature (denoted as GPF). We freeze the parameters of the pre-trained model  $f$  and introduce an extra learnable feature vector  $p$  into the feature space of the input graph described as Formula 6. We tune the parameters of the projection head  $\theta$  and feature vector  $p$  during the downstream training stage.
- Graph Prompt Feature-Plus (denoted as GPF-plus). We freeze the parameters of pretrained model  $f$  and introduce  $k$  learnable basis vectors  $p_1^b, \ldots, p_k^b$  with  $k$  learnable linear projections  $a_1, \ldots, a_k$  to calculate the node-wise  $p_i$  as Formula 10. We tune the parameters of the projection head  $\theta$ , basis vectors  $p_1^b, \ldots, p_k^b$ , and linear projections  $a_1, \ldots, a_k$  during the downstream training stage.
Implementation. We perform five rounds of experiments with different random seeds for each experimental setting and report the average results. The projection head  $\theta$  is selected from a range of [1, 2, 3]-layer MLPs with equal widths. The hyper-parameter  $k$  of GPF-plus is chosen from the range [5,10,20]. Further details on the hyper-parameter settings can be found in the appendix.
We compare the downstream performance of models trained using different pre-training and tuning strategies, and the overall results are summarized in Table 1. Our systematic study suggests the following observations:
1. Our graph prompt tuning outperforms fine-tuning in most cases. Based on the results presented in Table 1, it is evident that GPF and GPF-plus achieve superior performance compared to fine-tuning in the majority of cases. Specifically, GPF outperforms fine-tuning in 28/36 experiments, while GPF-plus outperforms fine-tuning in 29/36 experiments. It is worth noting that the tunable parameters in GPF and GPF-plus are significantly fewer in magnitude than those in fine-tuning (details can be found
Table 1: Test ROC-AUC (\%) performance on molecular prediction benchmarks and protein function prediction benchmarks with different pre-training strategies and different tuning strategies.
Pre-training Strategy	Tuning Strategy	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	PPI	Avg.
Infomax	FT	67.55±2.06	78.57±0.51	65.16±0.53	63.34±0.45	70.06±1.45	81.42±2.65	77.71±0.45	81.32±1.25	71.29±1.79	72.93
GPF	66.83±0.86	79.09±0.25	66.10±0.53	66.17±0.81	73.56±3.94	80.43±0.53	76.49±0.18	83.60±1.00	77.02±0.42	74.36
GPF-plus	67.17±0.36	79.13±0.70	66.35±0.37	65.62±0.74	75.12±2.45	81.33±1.52	77.73±1.14	83.67±1.08	77.03±0.32	74.79
AttrMasking	FT	66.33±0.55	78.28±0.05	65.34±0.30	66.77±0.13	74.46±2.82	81.78±1.95	77.90±0.18	80.94±1.99	73.93±1.17	73.97
GPF	68.09±0.38	79.04±0.90	66.32±0.42	69.13±1.16	75.06±1.02	82.17±0.65	78.86±1.42	84.33±0.54	78.91±0.25	75.76
GPF-plus	67.71±0.64	78.87±0.31	66.58±0.13	68.65±0.72	76.17±2.98	81.12±1.32	78.13±1.12	85.76±0.36	78.90±0.11	75.76
ContextPred	FT	69.65±0.87	78.29±0.44	66.39±0.57	64.45±0.6	73.71±1.57	82.36±1.22	79.20±0.51	84.66±0.84	72.10±1.94	74.53
GPF	68.48±0.88	79.99±0.24	67.92±0.35	66.18±0.46	74.51±2.72	84.34±0.25	78.62±1.46	85.32±0.41	77.42±0.07	75.86
GPF-plus	69.15±0.82	80.05±0.46	67.58±0.54	66.94±0.95	75.25±1.88	84.48±0.78	78.40±0.16	85.81±0.43	77.71±0.21	76.15
GCL	FT	69.49±0.35	73.35±0.70	62.54±0.26	60.63±1.26	75.17±2.14	69.78±1.44	78.26±0.73	75.51±2.01	67.76±0.78	70.27
GPF	71.11±1.20	73.64±0.25	62.70±0.46	61.26±0.53	72.06±2.98	70.09±0.67	75.52±1.09	78.55±0.56	67.60±0.57	70.28
GPF-plus	72.18±0.93	73.35±0.43	62.76±0.75	62.37±0.38	73.90±2.47	72.94±1.87	77.51±0.82	79.61±2.06	67.89±0.69	71.39
Table 2: Test ROC-AUC (%) performance on molecular prediction benchmarks and protein function prediction benchmarks for the models pre-trained by Edge Prediction.
Pre-training Strategy	Tuning Strategy	BBBP	Tox21	ToxCast	SIDER	ClinTox	MUV	HIV	BACE	PPI	Avg.
EdgePred	FT	66.56±3.56	78.67±0.35	66.29±0.45	64.35±0.78	69.07±4.61	79.67±1.70	77.44±0.58	80.90±0.92	71.54±0.85	72.72
GPPT	64.13±0.14	66.41±0.04	60.34±0.14	54.86±0.25	59.81±0.46	63.05±0.34	60.54±0.54	70.85±1.42	56.23±0.27	61.80
GPPT (w/o ol)	69.43±0.18	78.91±0.15	64.86±0.11	60.94±0.18	62.15±0.69	82.06±0.53	73.19±0.19	70.31±0.99	76.85±0.26	70.97
GraphPrompt	69.29±0.19	68.09±0.19	60.54±0.21	58.71±0.13	55.37±0.57	62.35±0.44	59.31±0.93	67.70±1.26	49.48±0.96	61.20
GPF	69.57±0.21	79.74±0.03	65.65±0.30	67.20±0.99	69.49±5.17	82.86±0.23	77.60±1.45	81.57±1.08	76.98±0.20	74.51
GPF-plus	69.06±0.68	80.04±0.06	65.94±0.31	67.51±0.59	68.80±2.58	83.13±0.42	77.65±1.90	81.75±2.09	77.00±0.12	74.54
in the appendix). These experimental findings highlight the efficacy of our methods and demonstrate their capability to unleash the power of the pre-trained models.
2. GPF and GPF-plus exhibit universal capability across various pre-training strategies. GPF and GPF-plus present favorable tuning performance across all pre-training strategies examined in our experiments, consistently surpassing the average results obtained from fine-tuning. Specifically, GPF achieves an average improvement of  $1.14\%$ , while GPF-plus achieves an average improvement of  $1.60\%$ . These results signify the universal capability of GPF and GPF-plus, enabling their application to models trained with any pre-training strategy.
3. GPF-plus marginally outperforms GPF. Among the two graph prompt tuning methods, GPF-plus performs better than GPF in the majority of experiments (26/36). As discussed in Section 3.3, GPF-plus offers greater flexibility and expressiveness compared to GPF. The results further affirm that GPF-plus is an enhanced version of graph prompt tuning, aligning with the theoretical analysis.
4.3 Comparison with Existing Graph Prompt-based Methods
We also conducted a comparative analysis between our proposed methods, GPF and GPF-plus, and existing graph prompt-based tuning approaches [Sun et al., 2022, Liu et al., 2023]. Both of them
(a) Training Curve
(b) Test Curve
Figure 2: Training and test curves of different tuning methods.
specialize in tuning the models pre-trained by Edge Prediction (also known as Link Prediction). We apply GPPT [Sun et al., 2022], GPPT without orthogonal prompt constraint loss (denoted as GPPT (w/o ol)) [Sun et al., 2022], GraphPrompt [Liu et al., 2023] to the models pre-trained using Edge Prediction, and the results are summarized in Table 2. It is worth mentioning that GPPT is originally designed for node classification tasks. Therefore, we make minor modifications by substituting class-prototype nodes with class-prototype graphs to adapt it for graph classification tasks. The experimental results indicate that our proposed GPF and GPF-plus outperform existing graph prompt-based tuning methods by a significant margin. On the chemistry and biology benchmarks, GPF and GPF-plus achieve average improvements of  $12\%$ ,  $3\%$ , and  $13\%$  over GPPT, GPPT (w/o ol), and GraphPrompt, respectively. These results showcase the ability of GPF and GPF-plus to achieve superior results compared to existing graph prompt-based tuning methods designed specifically for the pre-training strategy. Furthermore, it is worth highlighting that GPF and GPF-plus are the only two graph prompt-based tuning methods that surpass the performance of fine-tuning.
4.4 Additional Experiments
Few-shot graph classification. Prompt tuning has also been recognized for its effectiveness in addressing few-shot downstream tasks [Brown et al., 2020, Schick and Schütze, 2020b,a, Liu et al., 2021d, 2023]. We evaluate the efficacy of our proposed methods in handling few-shot scenarios. To conduct few-shot graph classification on the chemistry and biology datasets, we limit the number of training samples in the downstream tasks to 50 (compared to the original range of 1.2k to 72k training samples). The results are summarized in Table 4 of the appendix. Compared to the full-shot scenarios, our proposed graph prompt tuning demonstrates even more remarkable performance improvement (an average improvement of  $2.95\%$  for GPF and  $3.42\%$  for GPF-plus) over fine-tuning in the few-shot scenarios. This finding indicates that our solutions retain a higher degree of generalization ability in pre-trained models during few-shot downstream adaptations compared to fine-tuning.
Training process analysis. We conducted an analysis of the training process using different tuning methods on the biology datasets with the GNN models that employ Attribute Masking and Context Prediction as their pre-training tasks [Hu et al., 2020b]. Figure 2 presents the training and test curves during the adaptation stage. From Figure 2 (a), it can be observed that the ROC-AUC scores of the training set consistently increase during the adaptation stage for both our proposed methods and fine-tuning. However, from Figure 2 (b), we can find that their behavior on the test set is quite distinct. For fine-tuning, the ROC-AUC scores on the test set exhibit fluctuations and continuously decrease after an initial increase. On the other hand, when applying GPF or GPF-plus to adapt pre-trained models, the ROC-AUC scores on the test set continue to grow and remain consistently high. These results indicate that fully fine-tuning a pre-trained GNN model on a downstream task may lose the model's generalization ability. In contrast, employing our proposed graph prompt tuning methods can significantly alleviate this issue and maintain superior performance on the test set.
In this paper, we introduce a universal prompt-based tuning method for pre-trained GNN models. Our method GPF and its variant GPF-plus operate on the feature space of the downstream input graph. GPF and GPF-plus can theoretically achieve an equivalent effect to any form of prompting function, meaning we no longer need to illustrate the prompting function corresponding to each pre-training strategy explicitly. Instead, we can adaptively use GPF to obtain the prompted graph for downstream task adaptation. Compared to fine-tuning, the superiority of our method is demonstrated both theoretically and empirically, making it a compelling alternative for downstream adaptations.
This work was partially supported by Zhejiang NSF (LR22F020005), the National Key Research and Development Project of China (2018AAA0101900), and the Fundamental Research Funds for the Central Universities.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. 2022.
Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, G. Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. *ICLR*, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. NeurIPS, 2020.
Leonardo Cotta, Christopher Morris, and Bruno Ribeiro. Reconstruction for powerful graph representations. NeurIPS, 2021.
Ning Ding, Yujia Qin, Guang Yang, Fu Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Haitao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juan Li, and Maosong Sun. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv, abs/2203.06904, 2022.
Simon Shaolei Du, Wei Hu, Sham M. Kakade, J. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. ICLR, 2020.
Fabrizio Frasca, Beatrice Bevilacqua, Michael Bronstein, and Haggai Maron. Understanding and extending subgraph gnns by rethinking their symmetries. NeurIPS, 2022.
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington. Chembl: a large-scale bioactivity database for drug discovery. *Nucleic Acids Research*, 40:D1100 - D1107, 2012.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, pages 1024-1034, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. ArXiv, abs/2111.06377, 2021.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. *ICLR*, 2020a.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020b.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. ECCV, 2022.
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self-supervised learning on graphs: Deep insights and new direction. ArXiv, abs/2006.10141, 2020.
Thomas Kipf and Max Welling. Variational graph auto-encoders. ArXiv, abs/1611.07308, 2016a.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. In ArXiv, volume abs/1611.07308, 2016b.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Johannes Klicpera, Stefan Weißenberger, and Stephan Gunnemann. Diffusion improves graph learning. In Neural Information Processing Systems, 2019.
Boris Knyazev, Graham W. Taylor, and Mohamed R. Amer. Understanding attention and generalization in graph neural networks. In NeurIPS, 2019.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. ICLR, 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. EMNLP, 2021a.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101.00190, 2021.
Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In AAAI Conference on Artificial Intelligence, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586, 2021a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys (CSUR), 2022a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. ArXiv, abs/2110.07602, 2021b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021c.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. ArXiv, abs/2103.10385, 2021d.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL, 2022b.
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering, 35:5879-5900, 2021e.
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. Proceedings of the ACM Web Conference 2023, 2023.
Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqing Yang. Vision-and-language pretrained models: A survey. *IJCAI*, 2022.
Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi. Learning to pre-train graph neural networks. In AAAI, 2021.
A Extra Materials for Section 3
A.1 Extension to node-wise tasks
In this section, we illustrate the process of graph prompt tuning for node-wise tasks (node classification and link prediction). These tasks differ from graph classification, requiring node-level representations instead of graph-level representations. To bridge this gap, we employ Subgraph GNNs [Cotta et al., 2021, Zhang and Li, 2021, Bevilacqua et al., 2022, Zhao et al., 2022] to capture the graph-level and node-level representations. Subgraph GNN models utilize MPNNs on sets of subgraphs extracted from the original input graph. They subsequently aggregate the resulting representations [Frasca et al., 2022]. As a result, the node representations can also be interpreted as graph representations of the induced subgraphs. Specifically, the node representation  $h_i$  for node  $v_i$  can be calculated as:
$$
h _ {i} = f \left(\mathcal {G} _ {i}\right) \tag {13}
$$
where  $f$  is a Subgraph GNN model, and  $\mathcal{G}_i$  is the induced subgraph for node  $v_i$ . To obtain the node representations through graph prompt tuning, we freeze the model  $f$  and introduce a learnable graph prompt  $g_{\phi} \colon \mathbb{G} \to \mathbb{G}$ , parameterized by  $\phi$ , as indicated in Formula 2, which can be expressed as:
$$
h _ {i} = f \left(g _ {\phi} \left(\mathcal {G} _ {i}\right)\right) \tag {14}
$$
Once we acquire the node representations, we can seamlessly proceed with downstream node classification or link prediction tasks. Our proposed methods, GPF and GPF-plus, enable the effective execution of node-level tasks through the approach described in Formula 14.
A.2 Proof for Theorem 1
$$
f (\mathbf {A}, \mathbf {X} + \hat {\boldsymbol {p}}) = f (\hat {\mathbf {A}}, \hat {\mathbf {X}})
$$
where  $\psi_t(\mathcal{G}) = \mathcal{G}^* : (\mathbf{A}^*, \mathbf{X}^*)$ ,  $\mathbb{A}$  and  $\mathbb{X}$  are the candidate space for  $\mathbf{A}^*$  and  $\mathbf{X}^*$  respectively.