Instance-Aware Graph Prompt Learning
Authors: Jiazheng Li
Jundong Li
Chuxu Zhang
Venue: Accepted by TMLR ()

Abstract: Graph neural networks stand as the predominant technique for graph representation learning owing to their strong expressive power, yet the performance highly depends on the availability of high-quality labels in an end-to-end manner. Thus the pretraining and fine-tuning paradigm has been proposed to mitigate the label cost issue. Subsequently, the gap between the pretext tasks and downstream tasks has spurred the development of graph prompt learning which inserts a set of graph prompts into the original graph data with minimal parameters while preserving competitive performance. However, the current exploratory works are still limited since they all concentrate on learning fixed task-specific prompts which may not generalize well across the diverse instances that the task comprises. To tackle this challenge, we introduce Instance-Aware Graph Prompt Learning (IA-GPL) in this paper, aiming to generate distinct prompts tailored to different input instances. The process involves generating intermediate prompts for each instance using a lightweight architecture, quantizing these prompts through trainable codebook vectors, and employing the exponential moving average technique to ensure stable training. Extensive experiments conducted on multiple datasets and settings showcase the superior performance of IA-GPL compared to state-of-the-art baselines.

Published in Transactions on Machine Learning Research (02/2025)
Instance-Aware Graph Prompt Learning
Jiazheng Li University of Connecticut
jiazheng. Wi edu
Jundong Li University of Virginia
jl6gk-rvigiina edu
Chuxu Zhang University of Connecticut
chucau, edu
Reviewed on OpenReview: https: // openreview. net/ forun? id= 750H773DHE
Abstract
Graph neural networks stand as the predominant technique for graph representation learning owing to their scalability and end-to-end learning, yet the performance highly depends on the availability of high-quality labels in end-to-end manner. Thus the pretraining and fine-tuning paradigm has been proposed mitigate the label cost issue. Subsequently, the gap between the pretext tasks and downstream tasks has spurred the development of graph prompt learning which inserts a set of graph prompts into the original graph data with minimal parameters while preserving competitive performance. However, the current exploratory works are still limited since they all concentrate on learning fixed task-specific prompts which may not generalize well across the diverse instances that the task comprises. To tackle this challenge, introduce Instance-Awar Graph Prompl Learning (IA-GPL) in this paper, aiming to generate distinct prompts tailored to different input instances. The process involves generating intermediate prompts for cach instance using lightweight architecture, quantizing these prompts through trainable codebook vectors, and employing the exponential moving average technique to ensure stable training. Extensive experiments conducted on multiple datasets and settings showcase the superior performance of LA-GPL compared to state-of-the- art basclines. The code publicly available: litpps://gtt http:///gtthub.ooo http:///ihhbbboomm/iaagggg///// /ljjachhnn09917IIAAGPP
1 Introduction
Graphs function as pervasive data structures employed across various real-world applications, including but not limited to social networks refsocill,,, molecular structures refmolle,rfmmlee, and knowledge graphs refkg, due to their efficacy in modeling intricate relationships. With the rise of deep learning, Graph Neural Networks (GNNs) have emerged as formidable technique for analyzing graph data.
Nevertheless, GNNs trained end-to-end exhibit a strong dependency on large-scale high-quality labeled data for supervision, which can be challenging to obtain in real-world scenarios. To overcome this challenge, researchers have explored self-supervised or pre-trained GNNs gea, edgepred, xia2022simgrace, gel, yu2022sail, ju2023multi, wencoarse inspired by the advancements in vision fan2021multiscal and language bao2021boit domains. The pre-training methodologies using readily accessible label-free graphs aim capture intrinsic graph properties (e.g., node features, node connectivity, sub-graph pattern) that exhibit generality across tasks and graphs within a given domain. The acquired knowledge is then encoded in the weights of pre-trained GNNs. When it comes to downstream tasks, the initial weights can be efficiently refined through a weight fine-tuning step, leveraging a limited set of task-specific labels. However, as discussed in sun2023all, the 'pre-train and fine-tuning" paradigm is susceptible to the negative transfer problem.
Specifically, pre-trained GNN models focus on preserving the intrinsic graph properties, while fine-tuning seeks to optimize the weights on the reamstream tasks, which may significantly differ from the pretext tasks

---

Published in Transactions on Machine Learning Research (02/2025)
employed in pre-training. For instance, consider the scenario where GNN is pre-trained using link prediction objective kipf201 Gvariational, a prevalent, pretext task that aims to bring the representations of adjacent nodes closer in latent space. Subsequently, fine-tuning is performed using the node classification objective. In such a case, the model might exhibit suboptimal performance or even break down, especially if the graph dataset is heterophilic, where adjacent nodes may have different labels.
Consequently, in an effort narrow the gap between pre-training and downstream tasks, several ex- ploratory graph prompting learning frameworks graphprompt, sun2023all have been introduced. The concept of prompt tuning initially found application in the language domain ptuning, softprompt2, bhard- waj2022vector. In general, a piece of fixed or trainable prompt text is appended to the imput text, aligning the downstream task with the text generation capabilities of pre-trained large language models (LLMs). This approach not only preserves performance but also contributes to a reduction in training resource consumption. In the graph domain, prompt learning has recently remonstrated its potential alternative to fine-tuning, exem- plified by methods such as GPPT gppt, Graph- Prompt graphprompt, GPF gpf, and All-in- One (Sun et al., 2023 Similar to language prompts, these methods modify the original input graphs into prompted graphs which are further fed into frozen pre-trained graph models. The distinctions among these methods lic in the approach of inserting prompts into graphs and detailed training strategies. Nevertheless, the existing graph prompt learning approaches collectively operate under an assumption: that the learned task-specific prompts perform well across all imput instances within the task. In other words, these prompts are considered static concerning the input, a limitation that we deem critical. We argue that the dependency of prompts on the input instance is an essential characteristic that aids generalization over unseen samples. Using two molecules from the BBBP dataset as an example, as shown in Figure [ for molecule (a), since the atoms are all carbons, it is acceptable to use one universal prompt vector for all the atoms (nodes). However, for molecule (b) with complex structures, it is evident that these highlighted atoms with red circles (i.e., S, C, and N), contain distinct features and should be prompted in different ways.
NH.
H
Ca)cCcocc
(b) NCNW/Nccnnn-<zzcccccccess
Two example molecules from the BBBP dataset. Molecule (a) with simple structures suffices with universal prompt. However, molccule (b) with diverse atoms and intricate structures requires the use instance-aware prompts.
Figure 1:
To this end, our paper delves into the exploration of instance-aware prompt learning for the graph domain. This non-trivial research problem raises two questions: (1) what model should we use lo generale instance-aware prompts with additional use of a minimal number of parameters? It important to identify an effective and parameter-efficent method to transform the feature space into the prompt space, as the primary advantage of prompting lies in the minimization of trainable parameters. (2) how
ensure the instance-awary prompts are meaningful and distinctive as expected? Employing parameterized methods for prompt generation runs the risk of converging to trivial solutions, where all prompts collapse into singular solution. Consequently, guaranleeing the generation of diverse and meaningful prompls becomes a pivolal aspect of the entire pipeline.
In response to these challenges, we introduce a novel instance- aware graph prompt learning framework named IA-GPL designed to generate distinctive prompts for each instance by leveraging its individual information. Specifically, to tackle the first. question, we feed the representations of the input instance into parameterized hypercomplex multiplication layers (Zhang et al., 2020) which transform the feature space into the prompt space with minimal parameters. To solve the second question, we resort to the vector quantization (VQ) (GRAY) 1998h technique. VQ discretizes the continuous space of intermediate prompts, mapping each prompt to a set of learnable codebook vectors. The mapped vectors after VQ then replace the original prompts and are incorporated into the original features. To train the codebook, the exponential moving average technique utilized to prevent from converging to trivial solutions. To summarize, our main contributions
follows:
We propose TA-GPL, a novel instancc-aware graph prompting framework. To the best of our knowledge, IA-GPL is the first graph prompting method capable of generating distinct prompts based on different. instances within the dataset.
2

---

Published in Transactions on Machine Learning Research (02/2025)
In IA-GPL, we utilize parameter-cfficient bottleneck architecture for prompt generation followed by the vector quantization process via set of codebook vectors and the exponential moving average technique to ensure effectiveness and stability.
We conduct extensive experiments under different settings to evaluate the performance of IA-GPL. Our results demonstrate its superiority over other state-offthe-art competitors.
2 Related Work
Graph Representation Learning. The objective of graph representation learning is to proficiently encode sparse high-dimensional graph-structured data into low-dimensional dense vectors. These vectors are subsequently employed in various downstream tasks, such as mode/graphe classification and link prediction. The
methods span from classic graph embeddings (Grover & Leskovec 2016) to recent graph neural networks (Kipf
& WWelinng 2016 Veličković et al. 2017 Yun et al. 2019 Zhang et al. 2019 Fan et al. 2022 with the
remarkable success of deep learning. GNNs, which derive node representations by recursively aggregating
information from neighbor nodes, have emerged as a predominant standard for graph representation learning.
GNNs find applications in diverse areas, such as social network analysis (Guo & Wang. 2020 Li et al. 2023
bioinformaties (Mercado et al. 2021; Guo et al.) 2021 Wen et al. 2023). recommendation systems (Fan et al, 2019 Tian et al. 22022) and fraud detection (Dou et al. 2020; Liu et al., 2021). This is attributed to the fact that many real-world datasets inherently possess a graph structure, making GNNs well-suited for effectively modeling and extracting meaningful representations from such data. We refer the readers to a comprehensive survey (Ju et al. 2023) for details.
GNN Pre-training. Supervised learning methods applied to graphs heavily depend on graph labels, which may not always be adequate in real-world scenarios. To overcome this limitation, a pre-training and fine-tuning paradigm has been introduced. In this approach, GNNs are initially pre-trained to capture extensive knowledge from a substantial volume of labeled and unlabeled graph data. Subsequently, the implicit. knowledge encoded in the model parameters is transferred to a new domain or task through the fine-tuning of partially pre-trained models. Existing effective pre-training strategics can be implemented at node-level like GCA (Zhu et al. 2002), edge-level like edge prediction (Jin et al.] 2020 Zhao et al., 2023), and graph-level such as GraphCL ( (You al.) (220)) and SimGRACE (Xia et al. 2022). However, these methods overlook the gap that may exist between the pre-training phase and downstream objectives, limiting their overall generalization ability.
Graph Prompt. Learning. Prompt Learning seeks to bridge the gap between pre-training and fine-tuning by formulating task-specific prompts that guide downstream tasks, with the pre-trained model parameters usually kept static during downstream applications. Many effective prompt methods were initially proposod
in the natural language processing community, including hand-crafted prompts (Gao et al. 2020 Schick
& Schütze 2020) and continuous prompts (Gu et al. 2021 Ti & Liang) 2021 Liu et al. 2022). Drawing
inspiration from these works, several exploratory graph prompt learning methods, such as GPPT Sun et al,
2022 GraphPrompt (Liu et al.) 2023). GPF (Fang et al. 2023 All-in-One (Sun et al. 2023). GFT (Wang
et. a.. 2024a), and GIT (Wang et al. 2024b) have been proposed in the last. two years. These existing
methods introduce virtual prast-prototype nodes or graphs with learnable links into the input graph or directly incorporate learnable embeddings into the representations, facilitating closer aligument between downstream applications and the pretext tasks. However, all existing graph prompt tuning methods have predominantly concentratod on task-specific prompts, failing to generate instance-specific prompts which are critical since universal prompt template may not effectively accommodate input nodes and graphs with significant diversity shown in Figure I Note that while GPF-plus (Fang et al., 2023) also incorporates different prompts for different nodes using the attention mechanism, our method has several advantages over GPF-plus: (1) our method includes a lightweight down- and up-sample projector model that transforms the node hidden representations to another prompt vector space, while GPF-plus directly computes attention in the original feature space and then averages the weighted candidate prompts. An additional alignment between these two spaces is beneficial for the disentanglement of distinct information.
(2) instead of using original node features compute similarities, we use node features after the frozen GNN, which contain rich neighbor-aware information, further aiding in the prompt generation process. Thus in this
3

---

Published in Transactions on Machine Learning Research (02/2025)
(a) Pre-training
(b) Fine-tuning
(c) Our method
Tuned
+ Frozen
Node features
Pre-trained GNNs
Pre-trained GNNs
Pre-trained GNNs */c
Prompts
Projection head
Projection head
Pretext tasks
Downstream tasks
Downstream tasks
Figure 2:
Comparison between different paradigns of graph representation learning.
work, we introduce TA-GPI, a novel methodology designed to address the aforementioned issue by generating prompts that leverage the distinctive features individual instances.
3 Preliminaries
Graphs. Let G EV X, A) represent an undirected and unweighted graph, where V the set of nodes and E is the of edges. X R Vixd is the node feature matrix where the i-th row x; is the d-dimensional feature vector of node 8 V. A R VIX/V denotes the binary adjacent matrix with A,,, if ea, E and Ar, otherwisc. (Ve))= {u E (I u) E} represents the neighboring set of node v.
Graph Neural Networks. Generally, GNNs with a message-passinng passing mechanism can be divided into two steps. First, the representation of each node updated by aggregating messages from its local neighboring nodes. Second, the aggregated messages are combined with the node's own representation. Given a node v, these two steps are formulated as:
m (AG (h(-1), Yu (())).
(1)
( COMBINE (4 !
(2)
where m (1) and ( denote the message vector and representation of node v in the I-th layer, respectively. In the first layer, N is initialized as the node features X,, and the output of the last layer h, can be used downstream tasks.
GNN Pre-training and Fine-tuning. Given pre-trained GNN model fo(-) parameterized by 0, a learnable projection head parameterized by & and a downstream graph dataset = [(G,, vi), (G2, v2),... (Gn, Mn)) update the parameters of the pre-trained model and the projection head to maximize the likelihood of predicting the correct labels Y of the dataset G:
maz Ps. (Y)
(3)
0,0
Specifically, if we only update the parameters of the projection head, it is referred to linear probing:
maz Po. (Y.
Graph Prompt Learning. Compared with fine-tuning, prompt learning introduces a prompt generation model that aims to obtain prompted graph go G parameterized by This model transforms an input graph G to prompted graph (G)) which replaces the original graph and is fed into the pre-trained
4

---

Published in Transactions on Machine Learning Research (02/2025)
graph model as normal. The pre-trained graph model is fixed while only the parameters of the projection head and the prompt generation model are updated:
max 0.7
P (ggg))))))
(5)
A visual comparison these methods is presented in Figure 2 Note that, unlike other prevailing prompting frameworks that employ a universal prompt, our model integrates instance-aware prompts.
4 METHODOLOGY
In this section, we introduce the proposed framework of LA-GPL, as depicted in Figure 3 Firstly, we present a conceptual overview of the entire framework in section 4.1 Subscquently, delve into the key components of IA-GPL weight bottleneck architecture consisting of PHM layers in section 1.2 a prompl quantization
process via set of codebook vectors in section 4.3 and model optimization with the exponential moving
average technique in section 4.4
4.1 Naive Approach
In this section, we present an initial, high-level overview of our method, and then delve into more detailed aspects in the subsequent sections. To generale prompts associated with input instances, the first step involves obtaining specific representations of these instances. So naturally we employ the pre-trained graph model f.)-) as an encoder to generate the hidden embeddings:
= fo(G),
READOUT(H),
(6)
where = AX is the input graph, H E g is the obtained node representations and x E Rd is the graph reprosentation after READOUT operation which summarized node reprosentations form a graph-level representation by pooling or aggregation functions.
In IA-GPL, we consider node-level instance-aware prompts which means generate different prompts for cach node, as we unify different tasks into a general graph-level task following (Snn et al. (2023): Liu et al. (0024). Thus, after getting the node representations H, we employ an efficient bottleneck multi- layer perceptron architecture as the prompt generation model to project them into the prompt space. Specifically, we first project H RIVlxd from d to d' dimensions (d' < d) followed by a nonlinear function. Then is projected back d dimensions to get instance-aware prompts P E RIVixd matching the same shape as X that they be added back to the original node features. Mathematically it be formulated as:
P go (H),
X
(7)
(+))) DAAPROBBT(BBLUUUOOOPPPOOOO
(8)
where (+(+) represents the prompt generation model, X is the original node features while Xp is the prompted node features which contain instance-dopendent information. So far, we have established a general yet naive instance-aware prompt learning framework by replacing = (X with G, = (Xp; A), and train the prompt generation model and the projection head using the back- peeck-poppggtion algorithm. To expand on this simple concept, the following sections will elaborate on the details of the lightweight prompt generation model and the optimization process.
4.2 Lightweight Bottleneck Architecture
The prompt generation model which transforms H from the feature space into the prompt space consists of a down-sample projector and up-sample projector. Instead of the common option, FCN layers, we adopt PHM layers (Zhang et al. 2020) which are more parameter-officient.
FCN layers. One straight Etraightforward forward approach for implementing these two projectors is through fully connected layers (FCNs) which transform an input x E Rd into an output y E &R by:
y =FC(x) Wx b
(9)
5

---

Published in Transactions on Machine Learning Research (02/2025)
Overall Framework
Input instance
Instance-agonistic Ps
(a) P rompt Generator
(b)
GNN Back- bone *
Vector
Back- bone
roj. Head
LCB
Quantization
Node representations
Intermediate prompts
Quantized prompts
Node features
(a) Prompt Generator
(b) Vector Quantization
ReLU
Codebook
Logits
0.1
Sum
0.2
Down roject
Up Project
0.5
Multinomial
Tuned
Sample
0.1
Frozen
0.1
Stop
PHM-layers
Cons
Gradient
Figure 3: Overall Framework of IA-GPL.
where the weight. matrix W E Bexd and the bias vector b E R& are trainable parameters. We can control the number of parameters by controlling the hidden dimension d', but efficiency. In other words, it contradicts the original objective of prompt learning, which aims to reduce the number of trainable parameters, if we set d' large to maintain performance.
trade-off between performance and
PHM layers. To mitigate this problem, we turn to parameterized hyper-complex multiplication (PHM) layers as compromise solution which can also be written in the similar way:
y PHM(x) Mx + b,
(10)
where the replaced parameter matrix M E k is constructed by a sum of Kronecker products of several small matrices. The Kronecker product X & is defined as a block matrix:
En Y
Fin Y
YØY=
E R mmxxn
(11)
Tm1 Y
Lmn
where Fiy is the element of X E Tgmx at i-th row and j-th column and Y E RRx9, Given a user-defined hyperparameter Z=o, for 1,2,..., ., n, let each parameter matrix be denoted as A; gaxx and S; E R x Finally the parameter M caleulated by:
M = Σ A.S.
(12)
By replacing W with M, the number of trainable parameters is reduced to x (n x n n th x ==)= As is usually set a small number (e.g., 2, 4, 8), the parameter size of a PHM layer is approximately 1" of that of an FCN layer.
n mxd
In the case our approach, after we have node representations H through the pre-trained graph model, we feed them into the parameter-officicnt PHM layers instcad of standard FCN layers generate instanco-aware intermediate prompts P...
4.3 Prompt Quantization
Directly using Pe as prompts suffers from the high variance problem where the model that is overfitting the training data, thus Vector Quantization (VQ) (RAAY 1998) is utilized to discretize the intermediate prompt

---

Published in Transactions on Machine Learning Research (02/2025)
space P. to p.. VQ is natural and widely used method in signal processing and data compression that represents a set of vectors by a smaller set of representative vectors. Tt offers several advantages, including regularization, improved clustering, better interpretability, enhanced robustness to noise, and support for transfer learning, among others. This approach not only mitigates the high variance introduced by the PHM layers but also groups similar hidden prompt representations, thereby promoling the beneficial property of clustering.
Specifically, maintain k trainable codebook vectors E = (el e,...., ex) R shared across all the intermediate prompts Pc. For every prompt Pe E Pe, sample M codebook vectors from E corresponding to to obtain the quantized Pg- Please note that the quantization process for cach intermediate prompt operates independently of other prompts. In detail, we first compute the squared Euclidean distance d' between the prompt Pe and every codebook vector e, and the corresponding sampling logits Cy
( -
d' lp. e
(13)
where is temperature coeflicient used to control the diversily of the sampling process. Then sample M latent codebook vectors with replacement for prompt Pe from a Multinomial distribution over the logits l:
2
M 2e
Multinomial(1 Pe..... )
(14)
Finally, the quantized prompt Py can be computed by averaging over the M sampled vectors:
1 Pg M
Σ
(15)
After the VQ process, ensure that for semantically similar instances, the quantized prompts will also have similar representations by treating VQ as clustering mechanism. In the meanwhile, the limited set of learnable codebook vectors explicitly constrains the information capacity of prompt representations py> reducing the variance w.r.t. the output PHM layers, Pe.
Notably, we also introduce learnable aerance-agonist prompt Р, which is shared across all instances and incorporated into each quantized prompt Pa to have the final prompts py:
Dy Pv 3ps.
(16)
where 3 is a balancing hyperparameter. This allows us to effectively fuse the learned information from the input-dependent aspects captured by P, with the input-agnostic prompt P..
In summary, given the high-variance prompts Pa after PHM layers in the last section, the application of VQ
discretizes them into robust quantized prompts Pg that encapsulate intrinsic clustering property.
4.4 Model Optimization
The PHM layers PHM(-), instance-indeeeeners static prompt ps; codebook vectors E and the projection head 6 comprise the trainable parameters while we freeze the pre-trained GNN backbone fe(-). The loss function defined as:
L= LoE,YY, + Σ lP.. Pc, 12.
(17)
which consists of two parts: (1) Cross-entropy loss between the ground truth Y and the predicted labels Yp with prompted graphs as input. (2) Consistency loss that encourages the quantized prompts Pg to be consistent with the intermediate prompts Pe after PHM layers for all the instances (nodes) in the graph. These two terms ively aim to preserve performance while minimizing information loss during the vector quantization process. À is a hyperparameter used to balance the two loss terms which is set to 0.01.
However, a potential limitation of directly training the model using back paopppropngation (BP) is representation collapse where all prompts become constant vector that disregards the input, causing our model to degrade
7

---

Published in Transactions on Machine Learning Research (02/2025)
to GPF (Fang al. 2023 To solve this problem, we still use the standard BP algorithm to update the PHM
layers PHM(.), instance-independen static prompt P. and the projection head ø but adopt the exponential
moving average (EMA) strategy to update the codebook vectors E following Angelidis et al. (2021) Roy
ct al. 2018). Specifically, for each batch in the training process, we perform the following two steps:
Step 1: Count the number of times the j-th codebook vector is sampled and update the count Gy:
((ne)) O- (cott)) + (1 a)
Ie = ez]
(18)
Update the embedding of j-th codebook vector ej by calculating the mean of PHM layer outputs for which that codebook vector sampled during Multinomial sampling:
Step 2:
m I|e.-=
e (now) (j(aa) + (1 a)
(19)
Cj
where n and m stand for batch size and sample number, or is a hyperparameter set to 0.99 and I[]] is the indicator function. By incorporating the EMA mechanism, we can avoid the representation collapse problem and also obtain a more stable training process than gradient-based methods.
5 Experiments
5.1
Experimental Setup
Tasks and datasets. We evaluate IA-GPL using both node-level and graph-level tasks. Following Sun et al. (2023); Liu et al. (2024), we unify these tasks into general graph-level task by generating local subgraphs for the nodes of interest. For graph-level tasks, we use eight molecular datasets from MoleculeNet (Wu et al. 2018). For node-level tasks, we use three citation datasets from Yang et al. (2016). These datasets vary in size, labels, and domains, serving as a comprehensive benchmark for our evaluations. A comprehensive description of these datasets can be found in Appendix
Baselines. To evaluate the effectiveness of IA-GPL, we compare it with state-of-thh-art approaches across three primary categories. (1) Supervised learning: we employ GCN (Kipf & Welling 2016), Graph- SAGE (Hamilton et al.) 2017) and GIN (Xu et al. 2018). The base models and the projection head are all trained end-to-end from scratch. (2) Pre-truining and fine-tuning: The base model is pre-trained using edge prediction (Jin et al. 2020) for molecular datasets and graph contrastive learning (You et al.) 2020) for citation datasets. For the complete fine-tuning (F"T), the pre-trained model is fine-tuned along with the projection head. For lincar probing (LP), we freeze the pro-trained model and exclusively train the projection head. (3) Prompl learning: All in One (Sun et al. (202)), GPF (Fang et al. 2023) and GPF-plus (Fang el al.j 2023) are included. They all freeze the pre-trained base model while training the projection head and their respective prompt gencration modcls.
Settings and implementations. domain scenarios, we split the molecular datasets in two distinct manners: random split and scaffold split. Scaffold split is based on the scaffold of the molecules so that the vvain/valLtest set is more structurally different, making it appropriate for evaluating the model's generalization ability. In contrast, the random split is used to assess the model's in-domain prediction ability. We test. TA-GPL using different pre-training strategies: edge prediction (Jin et al. 2020) (denoted as EdgePred), Deep Graph Infomax (Veličković et al. 2018) (denoted as InfoMax), Attribute Masking (Hu al.) 2020a) (Denoted as AttrMasking), Context Prediction (Hu et al. 2020a) (Denoted as ContextPred) and Graph Contrastive Learning (You et al.) 2020 (Denoted as GCL) methods to demonstrate our model's robustness. We report results in both full-shot and few-shot settings, utilizing the ROC-AUC score as the metric. The few-shol setting is tested because prompt learning with fewer parameters is naturally less susceptible to the risk of overfitting when given limited supervision. We perform five rounds of experiments and report the mean and standard deviation. GCN is adopted as our backbone model for a fair comparison with other basclines. For the basclines, based on the authors' code and default settings, we further tune their hyperparameters to optimize their performance.
To evaluate the performance of TA-GPT. in both in-domain and out-of-
8

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: 50-shot ROC-AUC (%) performance comparison on molecular prediction benchmarks using random split. Bold numbers represent the best results in the graph prompting field (shaded region) to which our method belongs. Underlined numbers represent the best results achieved by other methods.
Tuming Strategies
Meshods
DDBP
Tox21
ToxCast
SIDER
ClinTox
BACE
HIV
MUV
Avg.
$0.204:1.00
64.5541.14
50.77=2.32
12.11
52.68:44.2
69.14=1.17
02877:2552
92
60.56
Supervised
GCN
83.9720.80
61.6520.73 73
51,35=1.43
08.5120.75
59.22:4 61
11.91=1744
39.9121.06
50.85114.00
61.3
GraphSAGE:
80.72+1.91
63.91:1008
32.09=0.43
49:14:1.19
58.5742.
71.33=0.97
61.0611334
53.0815.38 38
61.36
Pre-träniigg+
Lioearr Probing
79.67=1.31
69.99100.20 277
61.74=0.48
02:61:00.50
70.33: 76
76.17-0.77
65:04:1.40
59.12:
66:83
Fize taning
Fine Timing
38.0033.0
09.2522.77
00.12-0.55
62.3220.10
12.0942.74
7..97-0.22
61.1220.90
5d.1742.11 11
26.95
All in Oue
19. 2913.32
52. 07.4522.50 23
50.33
51.2112 0G
57. 67.65:111111
7312
16.3127.80 50
51.52
Prompt
GPP
61.11=0.43
02.29400100
13.31+44008
76.54-1.76
63.21:0.53
10.14+1..02
67:24
Learing
CPF-plus
83.08+1.55 1.57
71.31++0.80 80.00
60.85-1.60 1.69
52 02.44400.00 80
70.85+ 15
76.02-0.50
64.40+1.18
69.93-+-088
67.74
IA-GPL
00.62
72.65-+0.40
61.63++0.40
62.86=0.84
74.50+0077
76.04:00.83
64.60=00.55
69.3241.13
68.46
Table 2: 50-shot ROC-AUC (%) performance comparison on molecular prediction benchmarks using scaffold split. Bold numbers represent the best results in the graph prompting field (shaded region) to which our method belongs. Underlined numbers represent the best results achieved by other methods.
Timing Strategies
Methods
DDPP
Toc21
ToxCase
SIDER
ClinTox
BACE
IIV
MUV
Arg.
GIN
56.92:2.50 54
00.83:::51
52.50=0.68
18.852: 16
50.0017.53
51.0822114
68.09133 89
19. 19.1112.0 45
52.92
Supervised
GCN
57.0525.50
47.4023.56
49.67=0.61
19:92
58.8115.54
618842211
62.211..5 50
42.4443340
53.87
GraphSAGG
59.1347.25 28
18.12:3.01
51.90=1.43
19.00:1.92
10.53:14.25
39.28_1.60
61.26::108
19.11:2.9 90
52.78
Pre Pre-trainiug+
Linesu Probing
52.5125777
01:0220.42 42
37.46-0.35
50 7.
02.51:11.29 26
59.75=1.35
61.891-410
63.0713.08
59.05
Fine-taning
Fine-turinng
48 88.8800088
60.95+1-46
35.73-0-43
61.30+-2:21
57.78+-4:03
61.27-6.10
62.20++4.00 05
64.75 03
57 85
All In One
63-46+7.50 98
16. 19+-4.96
35.35-2.12
61.51++2.8 82
48.91+16.08
52.90-7.00 7.50
30.00+6:08
51.17
Prompt
GPF
52 1.21
63. 03.48:0..1
37.60:0.119
61.07:1.08
65.16+1.76
38,78-5.04
63.50122.3 31
66.91133 91
60.09
Learning
GPF-plus
64.73+5.20
63:23+-0.55
37. 37.19-0.67
00.31+1.60 1.00
64. 14-144+2.55
55.87-7.40 7.
61.4+-4.30
67.11+22.00
50.25
IA-GPL
56.54=2.35
64.1410.44
58.11::0.38
53.18=1.18
63,28:33.5 52
61.95:2-4.00
66.52:22.0
69.03::3300 0202
61.59
Additional implementation details are provided in Appendix C For cach experimental sctting, we train and test using 5 different random seeds and report the average value and the deviation, and the results passed the significance test.
5.2 Performance Evaluation
Due to the page limit, we present the experimental results of 50-shot random split and scalfold split settings on molecular datasets using edge prediction pre-training strategy in Table [ and Table 2 The results of full-shot lcarning, nodc-level tasks, larger graph datascts and more pro-training strategics
results are provided in Appendix B
In-domain performance. Table I jillustrates the results for 50-shot graph classification under the in-domain setting (random split). We have the following observations: (1) Compared to the pro-training and finc-tuning approach, LA-GPL achieves competitive results despite employing significantly lower number of trainable parameters. This underscores the key advantage of prompt learning, particularly when confronted with limited supervision. (2) While fine-tuning outperforms IA-GPF on certain datasets, IA-GPF consistently surpasses other graph prompt learning methods as shown in the shaded area, highlighting the significance of employing instance-aware prompts. (3) Unexpectedly, the All-in-One approach lags behind other prompting methods, cxhibiting the highest variance. This discrepancy may be attributed to an unstable training process.
Out-of-domain performance. Table illustrates the results for 50-shot graph classification under the out-of-domain setting (scaffold split). We have the following observations: (1) Overall, IA-GPL attains optimal results across these eight datascts, underscoring its efficacy even when confronting the out-of-distribution (OOD) challenge. We altribute this success to the vector quantization process, which caplures the clustering property of molecules. The disentangled clustering information can enhance performance in the presence of OOD samples by facilitating the transfer of learned knowledge. (2) Across different datasots, the performance trends of supervised learning, pre-training and fine-tuning, and prompt learning paradigms vary lot. Training GCN, GIN, GraphSAGE in an end-to-end manner yields the highest performance in the BBBP
9

---

Published in Transactions on Machine Learning Research (02/2025)
Samples of Carbon
Samples of Carbon
Somples of Nitrogen
Samples Nitregen 2
0-
0
-2
-3
-4-
-4-
-5%
1.0
0.5
0.0
0.5
1.0
0.5
0.0
0.5
1.0
Samples of Carbon
Samples of Nitrogen
Codchook vectors
Figure 4: Codebook visualization.
dataset, whereas it performs less effectively in other datasets such as Tox21 and SIDER. These fluctuations in performance may be attributed to the distinctive intrinsic propertics characterizing each datasct.
When considering the broader context, several key observations emerge: (1) Comparing linear probing (T.P) and fine-tuning (FT), the performance trend differs between random and scaffold split. For random split, FT outperforms LP, whereas the reverse is observed for scaffold split. This observation confirms the negative transfer drawback associated with the 'pre-training and fine-tuning' paradigm: the gap bet ween pretext tasks and downstream tasks leads to suboptimal performance. (2) The performance gain achieved by TA-GPL is more pronounced in the out-of-domain scenario, emphasizing the importance of vector quantization within our model. (3) The overall performance for in-domain classification remains significantly better than that for out-of-domain classification, underscoring the need to design effective methods to address the OOD problem.
5.3 Model Analysis
Scalabilty on larger datascts. Beyond the results of the previous molecular datasets, here we show the performance comparison of large biological dataset, the PPI dataset, which has 88k graphs and 40 classes, to showcase the scalability of our method. We tested it using the edge prediction pre-training strategy under both few-shot and full-shot settings. The results are illustrated in Table 3 Clearly, TA-GPL outperforms all these basclines.
Table 3: 50-shot and full-shot performances on large PPI datasct.
Setting
GraphSAGE
GCN
GIN
Linear Probing
Fine Tuning
All-in-One
GPF
GPF-plus
IA-GPL
50-shot
37. 37.20
40.75
39.56
49.70
46.23
42.90
50.64
52.56
53.13
full-shot
77.43
79.90
78.86
70.94
72.41
48.67
75.43
75.06
77.70
Codebook visualization and interpretability. We conduct a visualization and interpretability analysis on the learned codebook using a molccule from the BACE dataset with the SMILES string example. The model is configured VQ space. For every node (atom), we sample 5 vectors using Equation [4 which are then averaged and used as quantized prompts. Figure presents the t-SNE (Van der Maaten & Hinton 2008) plots of the samples of two carbon atoms and two nitrogen atoms in this molecule. Two characteristics of the learned codebooks are observed: (1) Samples corresponding to different atoms manifest. substantial distinctions (iee., the regions of samples in the plot). However, samples corresponding to the same atomss tend to exhibit in proximate regions in the codebook vector space. This observation affirms that LA-GPL effectively generates instance-aware prompts. (2) Fach atom's samples also demonstrate clustering property. This phenomenon may be attributed to the disentanglement of reprosentations for individual instances within the prompt space, which potentially encompasses general information.
(=C1
have 50 codebook vectors in the
Ablation study. To assess the individual contributions of each component, we conduct an abla- tion study by comparing IA-GPL with two different variants: (1) w/o VQ: After getting P. through the PHM layers, we directly use as the final prompts without the vector quantization process.
10

---

Published in Transactions on Machine Learning Research (02/2025)
(2) w/o PHM: We replace PHM layers in the prompt generation model with the standard MLP layers. Note that to maintain a fair com- parison and ensure a roughly equivalent number of trainable parameters, we reduce the size of the hidden dimension to 1 of that of PHM lay-
70
Original
w/o PHM
w/o VQ
65
ROC-AUC (%)
60
n ers, as discussed in Section 1.2) We conduct the
55
ablation study under 50-shot learning with scaf- fold split and illustrate the results in Figure 5
50
45
We have the following observations: (1) Replac- ing PHM layers with MLP layers of the same parameter size adversely affects performance to varying degrees across datasots. This result highlights the advantage of PHM layers over MLP layers when training resources are limited.
40
BDBP
Tox21
ToxCast
SIDER
ClinTox
BACE
Figure 5: Ablation study,
(2) Without the VQ process, the results drop there is no constraint to prevent codebook vectors from collapsing which leads to inferior performance or unstable training process.
Efficiency and complexity analysis. We an- alyze IA-GPL's parameter efficiency and frain- ing efficiency in Table 4 In terms of parameter efficiency, compute the number of tunable parameters for different strategics. (excluding the lask-specific projection head). Specifically, fine-tuning demands the update of all param- eters, making the most time- and resource- consuming process. the prompt learning do- main, GPF the most efficient since requires only one universal prompt while GPF-plus incorporates multiple attentive prompts. All-in-One also utilizes more than prompt node to construct prompt graph. In our model, the parameter size is prodominantly dominated by the prompt generation ion model (i.e., PHM layers), which aligns with the scale of other graph prompt learning methods and is significantly smaller than the fine-tuning approach.
Table 4: Model efficiency analysis.
# #'Tuning parameters
Relative ratio
Traizing time per spoch
GPL memory consumptton
Models
Fine tunitt
1.86M
100%
~0.68s
796MB
GPF GPF-plus
0.K
0.02%
~0.81s
768MB
3-12K
0.16-0.65%
~0.82s
~700MB
All-in-One
3K
0.16%
IA-GPL (Ours)
L08%
780MB
In terms of training efficiency, we compute the training time per epoch and GPU mem- ory consumption on the ToxCast dataset using single Nvidia RTX 3090. We keep all hyper- parameters the same including batch size, di- mensions, etc. All-in-One is omitted due to its unsatisfactory performance and unstable train- ing process. Generally, prompt-based methods are slower than traditional fine-tuning due to additional procodures such as computing at- lention scores and sampling. Regarding GPU memory consumption, prompt-based methods occupy slightly less GPU space since they do not need to save the gradients and optimizer states like fine-tuning for the frozen GNN backbone. But all of them are roughly at the same level considering the dominant backbone models and overhead GPU consumption.
BBBP
BACE
90
90
ROC-AUC (96)
80
ROC-AUC (96)
80
70
70
60
60
50
40
40
20
30
10
20
30
Shot Number
Shot Number
GPF
All-in-One Fine tuning
Lincar Probing
IA-GPL
GPF piuusp
GraphSAGE
Figure 6: Impact of shot numbers.
Impacts of the shot number. We study the impact of the number of shots on the BBBP and BACE datasets in the few-shot random split setting. We vary the number of shots within
the range of [5,10,20,30] and results are illustrated in Figure D In general, method IA-GPL consistently surpasses altains comparable results with other graph prompt learning frameworks
11

---

Published in Transactions on Machine Learning Research (02/2025)
in most cases especially when given very limited labeled data. As the mumber of shots increases, the overall performance increases while conventional supervised methods become more competitive.
Impacts of the codebook hyperparame- ters. We investigate the impact of the number of codebook vectors and the number of samples in the vector quantization process. Specifically, vary the size of the codebook within the
ROCAUID (k)
30-
range of [5, 10, 20, 50, 100] and the sample size within [3, 5, 10, 15, 20), while keeping the remaining hyperparameters constant. Results We observe that for most of the datascts, our model achieves relatively stable performance with respect. to the hyperparameters, alleviating the need for meticulous and specific tuning.
46
40
20
10
samples
are illustrated in Figure 7
ClinTox
Toa21
Figure 7: Impact of VQ hyperparameters.
Impacts of the input-agnostic promopt. Besides the two ablation study settings we have (i.e., without PHM layers and without VQ process), we present another setting: without the input-agnostic prompt. We test our model using scaffold split under both few-shot and full-shot scenarios. The results are shown Table We can find that the aggut-ag prompt has a limited impact on the model performance.
Table 5: 50-shot and full-shot performance comparison w/ and w/o input-agonistic prompt.
Shot
Methods
BBBP
Tox21
ToxCast
SIDER
ClinTox
BACE
HIV
Avg.
50-shot
IA-GPL
56.54422.35
64.14-0.44
18.1180.38
53.1841.18
63.2843.52
61.95=4.00
66:32-2.10
69.03:+.02
61.59
shot
IA GPL w/o
56.31:1:2.4
64.37=0.45
37.8840.41
12.06:11.15
63.05:13.50
61.61
66.71:22.08
69.10:13.01
61.50
full-shot
IA-GPL
09. 29.25:00006
80.28=0.20
65.8720.66
66 62:11.23
71.9610.41
83.3820.94
78.86:21.38
83.26:11.777
74.93
full-shot
IA-GPL-W/o
89.80:0.007
80.01=0.21
65.8220.66
66.12:11.22
72.01:0.42
83.10=0.95
78.89=1.36
82.95::1.76
71.88
6 Broader Impacts and Limitations
Broader Impacts. Research that is focused on paramneder-eflcient fine-tuning methods (PEFT) including our approach, LA-GPL, can usually bring several broad positive societal impacts such as: (1) Accessibility and Sustainability. By reducing the computational, financial, and environmental resources needed for fine- tuning pre-trained models, PEFT methods make advanced AI technologies accessible to wider range of individuals, organizations, and communities, including those with limited resources. (2) Improved Quality of Al Applications. LA-GPL also enhances numerous graph-related tasks and applications. By incorporating node-level instance-aware prompts, IA-GPL particularly well-suited for complex and heterogencous graphs, such as molecules and social networks. Consequently, TA-GPT can positively impact areas like drug discovery, protein structure prodiction, fraud detection, and so on.
Limitations. (1) Larger porameter size. IA-GPL demands larger parameter size, primarily due to the РНM layers when generating prompts. As discussed in section 5.3 compared to the huge pre-trained GIN model, the increase in the number of parameters remains acceptable considering the performance improvement achieved. However, it does introduce additional trainable parameters. (2) More lyperpurumelers. In addition to the standard hyperparameters such as the number of layers and embedding size, LA-GPL introduces new hyperparametors: the number of codebook vectors, the number of samples, and the temperature factor. However, as analyzed in Appendix 5.3 IA-GPL exhibits limited sensitivity these hyperparameters across most datasets, obviating the need for meticulous and specific tuning.
12

---

Published in Transactions on Machine Learning Research (02/2025)
7 Conclusions
In this paper, we introduce novel graph prompting method named Instance-Aware Graph Prompt Learning (IA-GPL), which is designed to generate distinct and specific prompts for individual input instances within a downstream task. Specifically, we initially gemerate intermediate prompts corresponding to each instance using a parameter-effcienn bottleneck architecture. Subsequently, we quantize these prompts with a set trainable codebook vectors and employ the exponential moving average strategy to update the paramcters which ensures a stable training process. Extensive experimental evaluations under full-shot and few-shot learning settings showcase the superior performance of IA-GPL in both in-domain and out-of-domain scenarios.
Acknowledgments
This work was partially supported by the NSF under grants I18-2334193, I1S-2340346, CNS-2426514 and CMMI-2146076. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
References
Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. Extractive opinion summarization in quantized transformer spaces. Transaclions of the Associalion for Compulational Linguistics, 2021.
Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network-based frand detectors against camouflagod fraudsters. In CTKM, 2020.
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In WWW, 2019.
Yujic Fan, Mingxuan Ju, Chuxu Zhang, and Yanfang Yo. Heterogencous temporal graph neural network. In SDM, 2022.
Taoran Fang, Yunchao Zhang, Yang Yang, Chunping Wang, and Lei Chen. Universal prompt tuning for graph neural networks. In NewrIPS, 2023.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint 1577,3,01,,15783 2020.
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington. Chembl: A large-scale bioactivity database for drug discovery. Nucleic acids research, 2011.
RM GRAY. Quantization. IEEE Transactions Information Theory, 1998.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, 2016.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2 09. 04392, 2021.
Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. Few-shot graph learning for molecular property prediction. In WWW, 2021.
Zhiwei Guo and Heng Wang. A deep graph neural network-based mechanism for social recommendations. IEEE Transactions on Industrial Informatics, 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NewrIPS, 2017.
W Hu, B Liu, Gomes, M Zitnik, Liang, Pande, and J Leskovee. Strategies for pre-training graph neural networks. Tn ICI.R, 2020a.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. or Xiv preprinl arXiv:2055006887, 2020b.
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self-supervised learning on graphs: Deep insights and new direction. arXiv preprint arXi::2006.101 10144, 2020.
Wei Ju, Zheng Fang, Yiyang Gu, Zequn Liu, Qingqing Long, Ziyuc Qiao, Yifang Qin, Jianhao Shen, Fang Sun, Zhiping Xiao, et al. comprehensive survey on deep graph representation learning, arXiv preprinl NrXiv:2304.05055 2023.
Thomas N Kipl and Max Welling, Semi-supervised classification with graph convolutional networks. arxiv preprint Xr0: 029077, 2016.
Jiazheng Li, Chunhui Zhang, and Chuxu Zhang. Heterogeneous temporal graph neural network explainer. In CIKM, 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing contimuous prompts for generation. arXiv preprint arXiv:2101.001 10,,,,,22 2021.
Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang, One for all: Towards training one graph model for all classification tasks. In ICLR, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In ACL, 2022.
Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang, and Qing He. Pick and choose: A gun-based imbalanced learning approach for frand detection. In WWW, 2021.
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In WWW, 2023.
Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Joerg Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction on chembl. Chemical Science, 2018
Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology, 2021.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments vector quantized autocncoders. arXiv preprint srXiv:1805.11000 2018.
Timo Schick and Hinrich Schütze, Exploiting cloze questions for few shot text classification and natural language inference. anXiv preprint arXiv:201..06666, 2020.
Teague Sterling and John J. Irwin. Zinc 15- ligand discovery for everyone. Journal of Chemical Information and Modeling, 2015.
Mingehen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In KDD, 2022.
Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In KDD, 2023.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Chao Huang, Ronald Metoyer, and Nitesh V. Chawla. Reciperee: A heterogeneous graph learning model for recipe recommendation. In IJCAI, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008
14

---

Published in Transactions on Machine Learning Research (02/2025)
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. arXie preprint arXiv: 1700. 10903, 2017.
Petar Veličkovié, William Fedus, William L Hamilton, Pietro Lió, Yoshua Bengio, and Devon Hjelm. Deep graph infomax. In ICLR, 2018.
Zehong Wang, Zheyuan Zhang, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. Gft: Graph foundation model with transferable tree vocabulary, In NeurIPS, 2024a.
Zehong Wang, Zheyuan Zhang, Tianyi Ma, Nitesh V Chawla, Chuxu Zhang, and Yanfang Ye. Learning cross-task generalities across graphs via task-trees. arXiv preprint arXiv:2412.16441 2024b.
Qianlong Wen, Jiazheng Li, Chuxu Zhang, and Yanfang Ye. A multi-modality framework for drug-drug interaction prediction by harnessing multi-source data. In CIKM, 2023.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 2018.
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In WWW, 2022.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2018.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In ICML, 2016.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NewrIPS, 2020.
Scongjun Yun, Minbyul Jeong, Rachyun Kim, Jacwoo Kang, and Hyunwoo J Kim. Graph transformer networks. In NewrIPS, 2019.
Aston Zhang, Yi Tay, SHUAI Zhang, Alvin Chan, Anh Tuan Luu, Siu Hui, and Jie Fu. Beyond fully-comnected layers with quaternions: Parameterization of hypercomplex multiplications with 1/n parameters. In ICLR, 2020.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Hetcrogeneous graph neural network. In KDD, 2019.
Jianan Zhao, Qianlong Wen, Mingxuan Ju, Chuxu Zhang, and Yanfang Yo. Sclf-supervised graph structure refinement for graph neural networks. In WSDM, 2023.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2025)
A Dataset Details
We have two kinds of tasks and corresponding datasets: graph-level tasks: molecular datasets and node-level tasks: citation networks. The statistics of these datasets are illustrated in Table G
For molecular datasets, during the pre-training process, we sample million unlabeled molecules from the ZINC15 (Sterling & Irwin 2015) database, along with 256K labeled molecules from the preprocessed ChEMBL (Mayr et al. 2018| Gaulton et al., 2011) dataset. For downstream tasks, we the molecular datasets from MoleculeNet (Wu et al. 2018) encompassing molecular graphs spanning the domains of physical chemistry, biophysics, and physiology- Specifically, they involve 8 molecular datasets: BBBP, Tox21, ToxCast, SIDER, Clintox, BACE, HIV and MUV, All datasets come with additional node and edge features introduced by open graph benchmarks (Hu et al. 2020b).
For citation networks, we use commonly used datasets: Cora, citepSeer, and PubMed from Vang al. (2016). Nodes represent documents and edges represent citation links. Each document (node) in the graph is described by 0/1-valued word vector indicating the absence/ pponnccceeecccceee of the corresponding word from the dictionary. During the pre-training phase, In the fine-tuning stage, we convert the node-level task to the graph-level task following Sun et al. 20233 and process them in the same way as the molecular datasets.
them without labels in self-supervised learning approach.
Table Statistics of the datasets.
Tasks
Name
#graphs
#nodes
#edges
#features
#binary tasks classsss
BBBP
2,050
23.9
51.6
9
Tox21
7,831
~18.6
388.6
12
ToxCast
8,597
188.7
388.4
9
617
SIDER
1,427
333.
700.
9
27
Graph-level
ClinTox
1,484
26.1
555.5
BACE
1,513
34.1
~73.7
9
MUV
93,087
244.
52.6
9
17
HIV
41,127
25.5
554.9
Cora
2,708
10,556
1,433
Node-level
citepSeer
3,327
9,104
3,703
6
PubMed
19,717
88,648
500
B Additional Experimental Results
B.1 Results of Full-shot Learning
We present the experimental results using the full datasets to train the model in both scaffold split and random split scenarios in Table 7 and Table respectively.
In-domain performance. Table g illustrates the results for full-shot graph classification under the in- domain setting (random split). We have the following obscrvations: (1) Overall, fine-tuning cxhibits superior performance across all methods including supervised schemes and prompt learning frameworks which is not surprising. Given an ample amount labeled training data, fine-tuning can effectively adapt the pre-trained model that already encapsulates intrinsic graph properties, thereby contributing to optimal performance. (2) IA-GPL consistently attains the highest results in the realm of graph prompt learning, demonstrating its exceptional performance in this category and the importance of inst ansce-e-wwree prompts.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Table 7: Full-shot ROC-AUC (%) performance comparison on molecular prediction benchmarks using scaffold split. Bold numbers represent the best results in the graph prompting field (shaded region) to which our method belongs. Underlined numbers represent the best results achieved by other methods.
Thaing Strategies
Methods
BBBP
Tox21
ToxCass
SIDER
ClaToo
BACE
HIV
MUV
Avg.
GIN GCN
67,30+2.88
74 74:25:6.65
62.22::133
57.4351.24 24
48.83=3.03
72.78+2.49
75.82+2.80
74.7941
66.65
Supervised
62,1823.41
11:.8.0.555
02.7110.59
02.5121.06
50,5823.222
73.111166 61
78.2642.01
71.9822 31
67.77
GraLSAGE
67.91
74.1410.55
63.79:9.70
02.8011.11
58.01:5.68
09:27::2991
75.77:3300
71.9041 43
67.96
Pre-tniningg+
Linewr Probing Finc Tuning
69.46:00.88
79. 79.55:20.12
65.41:20411
06.30-20.79
67.41=1.77
83-10:0.44
76.87=11.6
80 80.42+1
73.57
Fincstrming
66.56+3.56
78.63+
06.29-0.45
64.35+0.78
600774461
80:90+0:32
79.79+2.76
61.76+1 80
73.42
CPPT
64.13=0.14
66.41+0:04
60.34=0.14
54.86+0.22 25
10.81 00.46
70.85+11 442
60.54+00554
63.05+0034 3.4
62.49
GPPT (w/o
69.43:10.18
78 78.9110.15
64 61.00.0.11
00:9410.01
62.15=0.69
70:3140.99
73.19:40.19
62.0640.35 50
70.23
CraphPrompt
69.29+0.15
08.09+0.19
60.54+0.21
58.71+0011
55.37=0.57
67. 07.70++1.200 .26
10.31+00533
62. 02.35++0044
62.67
Prompt Learnnng
All in One
58.01=4.880
52.38:1::46
55.07=7.22
53.334
50.91=9.33
55, 12.71..775
58.32:14.40 4.
54.84
GPF
68.87:10.57
79. 79.93.10.08
65.63.0.11
65.9310.64
60, 2.777
80.372:-.07
75.20:11.30
80.87.11.70
73.47
GPF-phs
68,16:0.78
79. 0:000000000
65, 03.22:0..32
66 06:08-55.85
71:23=33.0
82. 16:115111 64
76 70.99:2.01
81.05t 68
73.91
1A GPL
69.25.10.06
80.28::0.20
05.874:0.44
06.02:21.23
71.00:10.41
83.38:0..9
18.80::1.38
83:26:11.7
74.93
Table 8: Full-shot ROC-AUC (%) performance comparison on molecular prediction benchmarks using random spilt. Bold numbers represent the best results in the graph prompting field (shaded region) to which our method belongs. Underlined numbers represent the best results achieved by other methods.
Timing Strategies
Methods
BMBP
Tor21
ToxCase
SIDER
ClinTox
BACE
IIV
MUV
Arg.
GIN GCN
93.09.00.0 91
82. 82.47:10.68
70.71=0.45
67. 37.70:11:42
75.6113.57
87.90=1.19
81.961:
80.57.2.02
78.76
Supervised
02.5020.79
81.82:0.22 25
72.50=0.55
$7.10#0.95
80.4513326
88.00=0.60
83.06:0.45
19.18
79.95
GraphSAGE
91.9810.19
82.52:0.322
72.55_0.42
56.0541118
80,5742.02
88.05_1.50
83.03:1134
79.6142.9 93
79.42
Pre training+
Lineus Probing
88.2110.05
82 03.8610..3 12
74.55=0.25
B1.1610.51
85.5111.09 0.00
89.73_0.52
85.1210.008
89.5310 42
82.12
Vine-Unnnn
Fine Tuning
93.0620.35
45.46:0020 26
75.35-0.33
63 63.8920.00 69
87,22+1.1
90.83-0.55
86,8420.77 72
87:261: 76
83.76
All in One
62.88-99.08 60
12.38:3 46
8.53
48 78:4844117 17
44.86:188.8 81
51.8 31.82-4.01
54.78:1.70 76
51.53
Prompt
GPF
92.71=0038
83.001:0.20 22
03.58=0.35
61. 1..9611.08
90.65:2..33
86 06.83.0.336
85.63110.8 39
90 00.29110.1 14
83.08
Learning
GPF--puus IA-GPL
89.9120.22 22
83.04::0.70 7U
74.24=0.36
62 02.00:11.33
72:00.64
88.56-0.50 0.52
85 0.
11:+0 16
83.67
91.7720.30 40
61.15:10.29
75.64:10.44
62.6120.73
87.2710.97
90.14::.114
86.02:20.90
91.57:10.19
85.90
Oft-of-domain performance. Table illustrates the results for full-shot graph classificatinn under the out- of-domain setting (scaffold split). We have the following observations: (1) When addressing the out-of-domain problem, LA-GPL consistently showcases superior performance compared to other baselines, confirming the clustering benefit derived from the vector quantization process. (2) While supervised learning can yield acceptable results in the in-domain setting, it notably lags behind fine-tuning and prompt learning approaches when confronted with the out-of-domain challenge. This underscores the benefit in generalization gained from the graph pre-training phase when substantial amount of labeled and unlabeled data are available to equip the pre-trained model with prior knowledge.
B.2 Results of Node-level tasks
We present the experimental results using node-level datasets-Cone Cora, citepSeer and PubMed in Table 9 We unify the task into general graph-level task by generating local subgraphs for the nodes of interest and use the 100-shot setting following Sun et al.] (2023). We observe that (1) IA-GPL achieves the best performance on all three datasets, demonstrating its capacity in node-level tasks. (2) Supervised learning outperforms the approach by a large margin, showcasing the implicit negative transfer problem.
B.3 Results of More Pre-training Strategies
Besides the edge prediction (Jin et al. 2020) pre-training strategies, we also use Deep Graph Info-
max /Velickovi et al., 2018) (denoted as Inx) Attribute Masking (Hu et al.) 2020a) (Denoted as AttrMasking), Context Prodiction (Hu et al., 2020a) (Denoted as CondextPred) and Graph Contrastive Learning (You et al., 2020) (Denoted as GCL) methods to compare with IA-GPL demonstrate our model's robustaess. Note that we test under the full-shot scaffold split setting. Results are illustrated in Table 10 We
17

---

Published in Transactions on Machine Learning Research (02/2025)
Table 9: 100-shot test accuracy (%) performance on node-level citation network datasets. Bold numbers represent the best results in the graph prompting field (shaded region) which our method belongs. Underlined numbers represent the best results achieved by other methods.
Tuning Strategies
Methods
Cora
citepSeer
PubMed
Avg.
GCN GAT Crans
78.06+1.37
82.1111.02
74.33+1.44
78.17
Supervised
79.71=0.77
82.27=0.68
74.44+0068
78.81
78.50
82.66+0.36
75.00+1.24
78.72
Pre-training+ Fine-tuning
Linear Probing
60.53+4.07
82.05++0.00
70.22+1.25
70.93
Fine Tuning
55.16+3.87
80.33+0.40
60.11+0.10
65.20
All in One
6..9
80.38:10.20
58.3311.44
67.56
Prompt
GPF
70.13+1.58
77.67+1.24
58.6741.58
68.82
Learning
GPF-plus
71.43#1.04
78.6710.92
61.3311.29
70.48
IA-GPL
71.51
81.33+1.29
63.33+0.67
72.06
observe that IA-GPL achieves state-of-the-art results in 27 out of cases within the graph prompt learning area.
C Additional Implementation Details
Table 11 presents the hyperparameter settings used during the adaptation stage of pre-trained GNN models on downstream tasks in IA-GPL. For molecular datasets, adopt the widely used 5-layer GIN (Xu et al 2018 the underlying architecture for our models. For citation networks,
adopt 2-layer Graph Transformers (Yun) et al. 2019) as the underlying architecture. Grid search is used to find the best set of hyperparameters. You can also visit our code repository to oblain the specific commands for reproducing the experimental results. All the experiments are conducted using NVIDIA V100 graphic cards with 32 GB of memory and PyTorrh framework. For the details, please visit our code repository.
18

---

Published in Transactions on Machine Learning Research (02/2025)
Table 10: Full-shot ROC-AUC (%) performance comparison on molecular prediction benchmarks with Deep Graph Infomax, Attribute Masking, PontextPred and GCL as pre-training methods. Bold numbers represent the best results in the graph prompting field to which method belongs. Underlined numbers represent the best results achieved by other methods.
Pre-training Strategies
Tuning Strategies
Methods
BBBP
Tox21
ToxCast
SIDER
Clin Tox
BACE
HTV
MUV
Avg.
CraphSACE
60.12
74.17
62.65
63.
55.43
74.70
70.44
73.60
67. 92
Supervised
GCN
68.07
74.63
59.03
63.89
55.24
63.39
76.85
71.82
66.62
GIN
70.43
73.20
60.73
60.42
51.19
71.70
74.21
71.77
66.71
Pre-traiming+
Linear Probing
66.52
78.02
66.49
65.18
73.74
84.55
77.68
80.02
74.03
InfoMax
Fine-tuming
Fine Tuming
69.81
78.92
66.50
66.54
71.86
82.68
76.33
81.01
74.21
All-In-Oue
58.50
66.09
52.43
46.09
58.98
69.69
48.08
57.12
Prompt
GPF
67.33
77.53
65.91
65.46
83.27
74.89
79.96
73.49
Learning
GPF-Plus
67.61
79.67
65.78
64.96
71.68
78.61
72.74
TA-GPT.
78.95
66.16
75.34
GraphSACE
71.68
73.94
61.96
62.16
61.01
63.86
73.90
76.27
68.10 10
Supervised
GCN
67.02
74.47
60.83
61.88
56.21
70.82
75.55
73.20
67.50
GIN
66.43
73.69
60.98
60.29
56.65
79.63
73.48
72.75
67.99
Pre-training+
Linear Prohing
66.56
79.37
66.15
67.65
74.52
86.61
78.55
81.34
75.09
Attrassking
Fine-taming
Fine Tuming
67.51
78.66
67 07.33
65.16
74.68
80.73
78.31
77.22
73.70
All-In-Oue
49.79
52.78
68.26
49.57
41.69
53.46
34.97
50.07
Prompt
GPF
67.70
79.16
66.75
66.39
72.24
85.82
77.51
79.08
74.33
Learning
GPF-Plus
67.73
78.42
67.95
68.13
73.02
84.08
78.08
84.11
75.19
TA-GPI.
69.35
79.30
68.52
69.66
80.15
86.78
78.90
84.70
77.17
GraphSAGE
64.12
72.05
60.20
61.99
72.94
77.77
74.18
75.85
69.89
Supervised
GCN
63.58
71.40
62.98
57.65
70.60
79.84
78.09
75.61
70.21
61.88
75.42
64.92
61.39
69.46
80.74
75.79
77.64
70.91
Pre-training+
Linear Probing
65.78
80.62
59.33
65.55
70.87
78.10
76.58
$3.19
72.25
ContextPred
Fine-tuning
Fine Tuning
67 99
78.
63.71
63.88
73.20
$1.90
79.71
81.41
73.75
All-In-Oue
56.93
62.18
61.62
45.91
59.19
48.01
39.10
53.13
Prompt
GPF
67.
78.
68.98
63.25
70.78
83.32
78.60
82.60
74..4
Learning
GPF-Plus
08.05
77.17
68.57
64.95
75.83
81.06
76.34
85.12
74.63
IA-GPL
69.92
80.49
68.18
66.07
77.30
82.62
79.90
85.53
76.07
GraphSAGE
67.88
68.79
63.79
51.08
72.47
68.41
71.10
68.50
66.50
Supervised
GCN
65.20
66.88
61.50
55.54
74.72
65.86
74.90
73.09
67.21
GIN
67.56
68.65
67.14
51.17
71.06
69.07
70.69
67.73
Pre-traming+
Linear Probing
71.82
74.81
58.75
76.92
65.49
74.02
73.91
69.83
GCL
Fine-tuning
Fine Tuming
72.56
63.17
56.26
74.64
68.20
73.89
75.73
69.41
All-In-Oue
47.33
49.54
41.07
57.70
54.24
46.17
50.87
Prompt Learning
GPF
70.54
73.19
61.08
61.77
72.10
67.53
73.61
74.92
69.34
GPF-Plus
70.94
73.70
60.90
62.48
71.54
70.62
76.84
77.07
70.51
IA-GPL
72.58
76.08
60.86
64.22
75.07
71.24
75.59
77.33
71.62
19

---

Published in Transactions on Machine Learning Research (02/2025)
Table 11: The hyperparameter settings for 50-shot learning.
Dataset
split
Learning rate
# Cooebooo vectors
#Samples
#MLP layers (Proj. head)
BBBP
Scaffold
0.005
20
10
Tox21
Scaffold
0.0005
50
10
ToxCast
Scaffold
0.0001
50
10
4
SIDER
Scaffold
0.005
10
5
ClinTox
Scaffold
0.0001
50
10
4
BACE
Scaffold
0.0001
20
10
2
HIV
Scaffold 0.005
20
10
4
MUV
Scaffold
0.0005
20
10
BBBP
Random
0.001
20
50
Tox21
Random
0.001
20
50
ToxCast
Random
0.005
50
5
SIDER
Random
0.005
50
5
ClinTox
Random
0.001
20
10
BACE
Random
0.001
50
5
HIV
Random
0.005
50
10
MUV
Random
0.005
20
10
20


Reviews:
[{"review_id": "i8lBPZdn6u", "reviewer": "Reviewer_F4fU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The key contributions and new knowledge presented are:\n1. The paper proposes the Instance-Aware Graph Prompt Learning (IA-GPL) framework, which, for the first time, enables the generation of instance-specific prompts tailored to individual graph nodes or graphs. This approach addresses the limitation of existing methods that rely on static, task-specific prompts.\n2. IA-GPL incorporates a lightweight bottleneck architecture using parameterized hypercomplex multiplication (PHM) layers for transforming input features into prompt space, ensuring parameter efficiency.\n3. The framework introduces a vector quantization (VQ) mechanism to discretize intermediate prompt representations into a fixed set of codebook vectors. This approach reduces variance and enhances the stability of the training process.\n4. By employing an exponential moving average (EMA) strategy for updating the codebook vectors, IA-GPL effectively prevents the collapse of prompts into trivial solutions and ensures diverse and meaningful prompt generation.\n5. The paper demonstrates the superiority of IA-GPL across multiple datasets, including molecular and citation networks, under various learning scenarios (full-shot, few-shot, and out-of-domain tasks).\nIt outperforms state-of-the-art baselines in both in-domain and out-of-domain settings, showcasing its robustness and generalization capability.\n6. The framework achieves competitive performance while maintaining parameter efficiency compared to traditional fine-tuning approaches, making it more suitable for real-world scenarios with limited computational resources.\n\nWeakness:\n1. While the use of instance-aware prompts is novel, the interpretability of the learned prompts and their impact on decision-making is not well-explained. For practitioners or domain experts, understanding how these prompts influence the final predictions is critical, particularly in sensitive domains like healthcare or finance.\n2. The use of PHM layers and vector quantization (VQ) adds computational overhead, particularly during training, which may limit scalability for extremely large graphs or high-dimensional datasets. The additional steps in quantization and the reliance on codebook updates may pose challenges in real-time or large-scale applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel framework for improving graph representation learning by generating distinct, instance-specific prompts. \nIn summary, this submission advances the field by bridging the gap between pre-training and fine-tuning in graph neural networks through a prompt-based learning paradigm, while addressing the limitations of static, task-specific prompt generation methods. It paves the way for more adaptive and instance-specific approaches in graph representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would be better to add a complexity analysis on the PHM layers and vector quantization (VQ).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4PpKPmbuAB", "reviewer": "Reviewer_ANs3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The problem of Graph Prompt Learning is important and popular. The paper is well motivated as the proposed challenge of fixed task-specific prompts for all graph instances is critical.\n\n+ The proposed IA-GPL model is well-designed and reasonable. Generating intermediate prompts for each instance using a lightweight architecture is relatively new compared to the existing studies. \n\n+ Extensive experiments on multiple datasets are conducted. The proposed model has the best performance compared to many baseline methods. Additional analyses including ablation study, efficiency analysis, pre-training methods, and visualization, are sufficient. \n\n+ The paper writing and organization are overall good. The paper is easy to follow. \n\nWeaknesses\n\n- Compared to the existing graph prompt learning models (e.g., GPF and All-in-One), IA-GPL introduces more parameters due to instance-aware design, which sacrifices the efficiency of IA-GPL. \n\n- IA-GPL introduces node-level prompts and how they will affect the graph-level tasks conducted in the experiments, which requires clarification. In addition, it is better to provide more illustration on how the generated node prompts will impact the results of the GNN model. \n\n- The writing is overall good, while there are some typos/writing issues. It is necessary to proofread the whole paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel Instance-Aware Graph Prompt Learning (IA-GPL) model to mitigate the label cost issue in training graph neural networks. Unlike existing studies that concentrate on learning fixed task-specific prompts, IA-GPL generates distinct prompts tailored to different input instances and further improves the generalization capability across diverse instances. Extensive experiments on multiple graph datasets demonstrate the effectiveness of the IA-GPL for different graph classification tasks, compared with the SOTA baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness part.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Skxo2EtGKM", "reviewer": "Reviewer_FMQx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The authors propose a new setting for prompt learning, namely instance-aware graph prompt learning\n- The authors propose a new method that falls within instance-aware graph prompt learning\n\n# Weaknesses\n- The authors fail to justify the new instance-aware graph prompt learning setting convincingly. As far as I understand, Figure 1 is meant to act as a strong motivator for this setting. However, I fail to understand how this figure and the accompanying description are meant to motivate instance-aware prompting. More specifically, I do not understand the meaning of Figure 1.\n- The method IA-GPL, which the authors propose, uses many unusual and poorly (at least within this work) motivated components. The authors attempt to alleviate this with an ablation study. However, I would argue that this ablation study falls short of what should be expected here. Specifics follow in the next points.\n- The naive approach itself is not justified or motivated in this work. Specifically, the bottleneck component is not well justified. I would suspect it is being used to reduce the number of trainable parameters?\n- Further, the replacement of the FC layers in this bottleneck component is not well justified or put into context. The authors should ideally discuss specifics regarding PHM layers, i.e., what kind of advantages and disadvantages PHM layers have over FC layers.\n- The prompt quantization approach is not well justified. The authors don't convincingly argue why VQ is the best method of choice. For instance, a more traditional approach would be to learn the vector cookbook as proposed, but instead of probabilistically mapping the prompt vectors onto the cookbook vectors, use a direct projection into the cookbook vector space (that is, if E is the matrix containing all cookbook vectors as rows, then E^TE will be a projection of the prompt into the subspace spanned by the cookbook vectors).\n- In the prompt quantization section the authors claim that this cookbook of vectors will be beneficial because of its clustering property, however, the authors fail to mention what they mean specifically by this cluster property and why it would be beneficial for prompts to fall within clusters.\n- There are multiple small typos and grammar mistakes, and some sentences are not clearly expressed. For instance, what does the sentence on pages 6-7, \"Please note that the quantization process for each intermediate prompt p_c operates independently of other prompts.\" mean, and what is its purpose? Are there other aspects of the methodology that do not act independently of other prompts? I will add a list with at least some of the typos and grammar mistakes to the requested changes.\n- The authors claim that their method demonstrates a new state of the art for prompt learning. However, their experiments are not convincing.\n  - For one, the authors evaluate only the MoleculeNet and Cota, CiteSeer, and PubMed datasets. However, consider, for instance, the datasets present in \"All in One Multitask Prompting for Graph Neural Networks,\" which further include the Reddit and Amazon datasets as well as PROTEINS, ENZYMES, QM9, MovieLens, PersonalityCafe, and Facebook (the last here notably in the appendix). As additional suggestions for dataset diversity, one could also consider work like \"Graph Positional and Structural Encoder,\" which further includes ZINC, PCQM4Mv2, MolHIV, MolPCBA, Peptides-struct, Peptides-func, CIFAR10, MNIST, arXiv, (and CSL, and EXP for expressivity)\n  - In the considered experiments, the authors correctly mark the best method in bold; however, statements of state of the art should be accompanied by evaluations of significance, i.e., the authors should also mark all methods that are only insignificantly worse than the best method according to, for instance, a student T-test. Notably, it is quite obvious that for many, if not most, of the evaluations, the author's \"state of the art\" falls within one standard deviation and is thus unlikely significant to a standard 95% significance level.\n  - The baselines do not seem to be representative of reasonable baselines. Consider, for instance, \"Strategies for pre-training graph neural networks,\" which has a (in this case not pretrained or otherwise modified) GIN network that far outperforms all methods considered in Table 2 (which is equivalent due to its scaffold split)\n  - I would ask the authors to explain why the mentioned baselines are reasonable and why other performances like the ones in \"Strategies for pre-training graph neural networks\" are not considered.\n- One issue with their comparison with other prompt methods becomes evident from Table 3, namely, that their method has far more tuning parameters (notably still far less than fine-tuning). Why did the authors not attempt to evaluate their method using the same number of parameters?\n- In their ablation study the authors include their method having replaced PHM with an MLP with only very few parameters, while this is reasonable for fairness sake, for completeness sake the authors should also include a standard non bottlenecked MLP here even if its performance then eclipses the proposed method.\n\n\nCantürk et al., \"Graph Positional and Structural Encoder,\" ICML 2024\nHu et al., \"Strategies for pre-training graph neural networks,\" ICLR 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes instance-aware graph prompt learning as a relevant area of study, which leads to the proposal of an IA-GPL model that falls within this domain. The authors pre-train their method on domain-specific datasets (ZINC15 and ChEMBL for molecules and Cota, CiteSeer, and PubMed for citation networks). The authors evaluate their method on the MoleculeNet datasets and Cota, CiteSeer, and PubMed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors should either explain how they meant Figure 1 and convince me that their explanation is sufficient or modify their current justification of instance-aware graph prompt learning.\n- The authors should greatly expand on their motivations for each component of IA-GPL (if this does not fit into the 12-page limit, this can be added to the appendix). This includes:\n  - motivation the Naive approach, which is essentially motivating the bottle-neck architecture.\n  - motivating PHM.\n  - motivating VQ\n- The authors should expand on the benefits of clustering wherever mentioned or remove these passages.\n- The authors should expand the experimental evaluations. I believe it is unlikely that the authors can finish a significant amount of experiments within the discussion phase. However, the current rather small array of experiments is not convincing enough, especially in regards to:\n- The authors should add significance evaluations to their tables marking insignificantly different other methods (usually this is done by underlining them, however, the authors have already made use of underlining for non-prompt based methods. Perhaps italics could be used here.) As soon as this is done, the authors should also discuss the significance of their results.\n- The authors should further justify why their baselines are sufficiently representative, especially considering, for instance, \"Strategies for pre-training graph neural networks,\" which has far better-performing baselines.\n- The authors use the language of shots for full-shot and few-shot learning but do not detail exactly what this means in their context. I assume they are referring to the fact that their \"pre-training\" for each task is conducted on only 50 samples for 50-shot learning; however, how these 50 samples are obtained to ensure fair evaluations would still be important.\n- If possible, I would request that the authors evaluate their method using only 3K parameters as well for fairer comparison with other prompting methods.\n- I request that the authors include in their ablation:\n   - The above-mentioned low-rank projection (notably, this is essentially PCA on the prompt vectors) instead of VQ.\n   - An evaluation of their method without PHM and without VQ.\n   - An evaluation using a full MLP instead of a bottleneck MLP.\n   - An evaluation with and without the instance-agnostic prompt.\n   - for completeness, the authors should include one full sequence of adding one component after another from a reasonable baseline, where each addition improves performance. This should complete their ablation. (This can replace most of the requests above. However, I would still ask the authors to compare VQ with PCA)\n- In the following, I include a list of some typos, grammar mistakes, and other smaller changes I request. I would suggest that the authors generally take their entire manuscript and pass it through one or multiple grammar checkers (like Grammarly, DeepL, ChatGPT, ...):\n   - In the abstract, \"owing to their strong expressive power.\" This is a poor choice of words as compared to many other methods for graphs. GNNs are known to be limited by the 1-WL in the expressivity. Change this or completely rewrite this first sentence.\n   - In section 3, page 3, \"N(v) denotes the neighboring set of node v.\" This is not a proper description of the set of neighbors N(v). For concreteness, the authors should provide the exact formula for this set.\n   - In equation 1 within the set it should be h_u^(l-1), replace v with u here.\n   - In the paragraph after equation 2, h_v^0 corresponds to only some of the matrix X. Clarify this to be X_{i, \\cdot} or something similar, or change this sentence entirely.\n   - In the same spot, the output of the last layer is h_v^l for all v, not just the one vector. Clarify this.\n   - Define or at least describe what the ReadOut function is.\n   - After equation 8, \"By far, we have ...\" should be \"So far, we have ...\"\n   - In general, the authors should revise section 4. Some of this, like FCN and PHM, belongs in the preliminaries, and at the very least, the first paragraph should include a more detailed description of what follows. The issue arises because while reading the naive approach, it is not clear that this will be further modified later on and does not represent the actual method. If this approach is already present in some related work (which I am not aware of), then this should also be included in the preliminaries, cited, and put into proper context.\n   - After equation 1^/n^n should be (1/n)^n.\n   - In prompt quantization, specify what the high variance problem is and clarify what \"there is no explicit constraint in the PHM layers\" refers to, i.e., what constraints are the authors considering here?\n   - Also, in section 4.3, \"is utilized to discrete the intermediate ...\" should be \"is utilized to discretize the intermediate ...\"\n   - Also, in section 4.3, \"E=(e_1,\\dots, e_k)\\in\\mathbb R^{k\\tims d}\", the small k should be replaced with large K (if I understood this correctly).\n   - Towards the end of section 4.3, \"learnable instance-agonist prompt\" should be \"learnable instance-agnostic prompt\"\n   - The authors choose GCN as their backbone. The authors should argue why they made this choice, whether they tried other backbones, and which other backbones they tried. \n   - There are some other typos and grammar mistakes, which is why I would still suggest that the authors check their entire manuscript using some tool as mentioned above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers generally consider the graph prompt learning problem explored in this study significant and potentially appealing to a broader audience. The proposed IA-GPL model, which generates instance-aware prompts for different graph instances and employs a lightweight architecture for training, is a new approach compared to existing methods. The experiments, which include a set of comparisons, ablation studies, efficiency analysis, pre-training methods, and visualizations, are comprehensive and well-executed.  

Reviewers have also suggested improving the motivation discussion, adding significance tests, and providing more in-depth efficiency analysis and additional ablation studies. While all points are valid/valuable, I believe the pros outweigh the cons, and these points can be addressed in the revised version. I strongly hope to see the reviewer consider all these revisions before submitting the camera-ready.
Decision: Accept (poster)