Edge Prompt Tuning for Graph Neural Networks

Authors: Xingbo Fu, Yinhan He, Jundong Li
Primary area: learning on graphs and other geometries & topologies
Keywords: Graph Neural Networks, Prompt Learning

Abstract:
Pre-training powerful Graph Neural Networks (GNNs) with unlabeled graph data in a self-supervised manner has emerged as a prominent technique in recent years. However, inevitable objective gaps often exist between pre-training and downstream tasks. To bridge this gap, graph prompt tuning techniques design and learn graph prompts by manipulating input graphs or reframing downstream tasks as pre-training tasks without fine-tuning the pre-trained GNN models. While recent graph prompt tuning methods have proven effective in adapting pre-trained GNN models for downstream tasks, they overlook the crucial role of edges in graph prompt design, which can significantly affect the quality of graph representations for downstream tasks.
In this study, we propose EdgePrompt, a simple yet effective graph prompt tuning method from the perspective of edges. Unlike previous studies that design prompt vectors on node features, EdgePrompt manipulates input graphs by learning additional prompt vectors for edges and incorporates the edge prompts through message passing in the pre-trained GNN models to better embed graph structural information for downstream tasks. 
Our method is compatible with prevalent GNN architectures pre-trained under various pre-training strategies and is universal for different downstream tasks.
We provide comprehensive theoretical analyses of our method regarding its capability of handling node classification and graph classification as downstream tasks.
Extensive experiments on ten graph datasets under four pre-training strategies demonstrate the superiority of our proposed method against six baselines. Our code is available at https://github.com/xbfu/EdgePrompt.

Decision: accept (poster)
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730183056275

thread_type:
official_review

<thread_object>
creator: reviewer_t54u

<content>
summary:
The paper presents EdgePrompt, a method that enhances pre-trained GNNs for downstream tasks by using learnable prompt vectors on edges. EdgePrompt+ further customizes these vectors for individual edges. This approach improves graph structural representation and is compatible with various GNN architectures. Experiments on multiple datasets show its effectiveness over existing methods for node and graph classification tasks.

strengths:
1. The paper is well-organized, with clear points, and is easy to follow.
2. The effectiveness of EdgePrompt is theoretically guaranteed, and it performs excellently in downstream tasks.

weaknesses:
1. The motivation for constructing EdgePrompt is insufficient. Why is it necessary to design EdgePrompt under graph prompt tuning? What core problem does EdgePrompt address compared to existing graph prompt tuning methods? What are its advantages?
2. Compared to ALL-in-one and GPF, EdgePrompt and EdgePrompt+ set different prompt vectors $p^{(l)}$ for each layer. What are the benefits of this design? Both All-in-one and GPF only add prompt vectors in the first layer to reduce dependency on the specific structure of the model. EdgePrompt lacks such advantages, and the paper does not explore the reasoning behind this design. Furthermore, the experimental section does not include relevant comparisons to demonstrate the necessity of setting different prompt vectors for each layer.
3. The datasets included in the experimental section do not contain initial edge features, which raises doubts about the effectiveness of EdgePrompt on graphs that inherently have edge features. If the original graph already contains edge features, how should EdgePrompt be integrated with these edge features? What would its performance be like in that case?
4. The downstream tasks involved in the experiments are limited to node classification and graph classification, with other graph tasks such as link prediction and node regression not being included.

questions:
Please refer to the points I mentioned in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730273947347

thread_type:
official_review

<thread_object>
creator: reviewer_7gly

<content>
summary:
The paper proposes EdgePrompt, a graph prompt tuning method that enhances GNNs by learning prompt vectors for edges, improving graph representations. EdgePrompt integrates these edge prompts through message passing, outperforming existing methods across ten datasets under four pre-training strategies.

strengths:
1. The paper is well-motivated. It's important to integrate structural knowledge in prompt learning.
2. The authors conducted extensive experiments, demonstrating the effectiveness of the proposed methods.
3. The authors provide theoretical analysis, further proving the effectiveness of the proposed methods.
4. The paper is well written and easy to follow.

weaknesses:
1. **Inaccurate statement**: GraphPrompt [1] is not based on a specific pre-training strategy. As shown in GraphPrompt+ [2], all contrastive learning pre-training methods can be unified as subgraph similarity calculations. The link prediction used in [1] can be replaced by other methods.
2. **Missing related work**: GraphPrompt+ [1] also adds prompt vectors to each layer of the pre-trained graph encoder, which should be discussed and compared.
3. **Unclear explanation of anchor prompts in EdgePrompt+**: It is unclear what the anchor prompts in EdgePrompt+ represent. In my opinion, anchor prompts are introduced to address the overfitting problem caused by directly learning edge-specific prompts for different edges, but there lacks a explanation for the meaning of the anchor prompts. A more reasonable and effective solution could be conditional prompting [3,4], which I highly recommend the authors explore in future work.


[1] Liu et al. "Graphprompt: Unifying pre-training and downstream tasks for graph neural networks." Proceedings of the ACM Web Conference 2023. 2023.\
[2] Yu et al. "Generalized graph prompt: Toward a unification of pre-training and downstream tasks on graphs." IEEE Transactions on Knowledge and Data Engineering (2024).\
[3] Zhou et al. "Conditional prompt learning for vision-language models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\
[4] Yu  et al. "Non-Homophilic Graph Pre-Training and Prompt Learning." arXiv preprint arXiv:2408.12594 (2024).

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730519113192

thread_type:
official_review

<thread_object>
creator: reviewer_wrhj

<content>
summary:
This paper introduces EdgePrompt, a new graph prompt tuning method that improves graph representation for downstream tasks by learning edge-specific prompts, enhancing the performance of pre-trained GNNs. Extensive experiments show EdgePrompt’s effectiveness across various datasets and pre-training strategies, outperforming several baseline methods.

strengths:
1. EdgePrompt improves the adaptation of pre-trained GNN models for downstream tasks by introducing edge-level prompts, which helps bridge the objective gap between pre-training and downstream tasks..
2. Extensive experiments on multiple datasets and pre-training strategies demonstrate the method’s effectiveness, showing better performance compared to existing graph prompt tuning approaches.

weaknesses:
1. EdgePrompt uses shared prompt vectors, which may not capture the different relationships between edges well. This can limit the model’s ability to use all the information in the graph.
2. EdgePrompt+ adds multiple anchor prompts and score calculations, which can make the model more complex. This can lead to higher computational costs, making it harder to use in larger graphs.
3. The method struggles with few-shot learning because most edges lack supervision. This can reduce the model’s performance in real-world tasks where labeled data is limited.

questions:
How can the performance of EdgePrompt be improved in scenarios with limited labeled data to enhance its effectiveness in node classification tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720015765

thread_type:
official_review

<thread_object>
creator: reviewer_d2xh

<content>
summary:
Recent graph prompt tuning methods have proven effective in adapting pre-trained GNNs to downstream tasks. However, they often overlook the crucial role of edges in graph prompt design. To address this research gap, this submission introduces a new graph prompt tuning method focused on edges, called EdgePrompt. Nevertheless, despite emphasizing the importance of edges in graphs, the authors make an overly strong assumption by considering only a single type of edge. Additionally, the paper does not address edge-related tasks, which significantly undermines the overall contribution and impact of the work.

strengths:
S1. Clear motivation and presentation.

S2. The proposed method can be integrated with existing pre-trained GNNs.

weaknesses:
**Weakness**

W1. The unclear statements regarding the edge-level aspect weaken the paper’s contributions.

W2. The authors need to further elaborate on the technical contributions.

W3. More experiments are needed to better support the superiority of the proposed method.

**Concerns**

C1. As a study focused on edge-level prompt tuning, the assumption that there is only one type of edge could significantly undermine the contributions and claims of this paper. In line 154, the modeling of the adjacency matrix, $\mathbf{A} \in \{0,1\}^{N \times N}$, implies that the paper does not target multi-relational graphs. However, compared to other node-level graph prompting systems, the proposed edge-level graph prompting method could be more suitable for graphs with multiple edge types. The authors may need to clarify this in the submission.

C2. Since this work emphasizes edge-level prompt tuning, it would be beneficial for the authors to explore edge-related tasks, such as edge classification and link prediction, to further expand the scope of the paper.

C2-1. In many real-world scenarios, studying edge-level tasks is highly relevant because the space of edge types can evolve over time. For example, in a social network, a newly introduced user interaction feature might require predicting new edge types using a trained GNN.

C2-2. If the research on edge-level tasks is beyond the scope of current pre-trained GNNs (i.e., no existing pre-trained GNNs focus on edge-level tasks), the authors should clarify this limitation in the submission.

C3. The core Equation (4) in EdgePrompt+ appears overly similar to existing work, which may diminish the paper’s technical contribution. In CompGCN [1], the operation of weighting relation embeddings based on relation base embeddings has already been shown to be simple and parameter-efficient. Therefore, the authors should elaborate on the unique technical contributions of their method.

Minor Concerns:

C4. More classic and promising pre-trained GNNs, such as Infomax, EdgePred, AttrMasking, MGSSL, GraphMAE, and Mole-BERT, could be included in the experimental section. At the very least, the authors should discuss these models and explain why they are excluded from comparison.

C5. Figure 2 presents convergence speeds in terms of the number of epochs. The authors should also analyze the efficiency of the proposed method using learning curves or running time comparisons.


**Reference**

[1] COMPOSITION-BASED MULTI-RELATIONAL GRAPH CONVOLUTIONAL NETWORKS, ICLR 2020.

questions:
Please focus on answering concerns C1-C3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732075139942

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal Revision

comment:
We sincerely thank all the reviewers for their efforts to review our work. In response to the valuable feedback, we have made several major updates to our manuscript, as outlined below:
1. We have corrected the pre-training compatibility of GraphPrompt and added GraphPrompt+ for comparison (see the $\color{blue}{\text{blue}}$ part in Table 1);
2. We have added new results on model efficiency, results on graph data with edge features, and results with edge prompts at the first layer (see the $\color{red}{\text{red}}$ part in Appendix D); 
3. We have added future works about experiments under more pre-training strategies, other designs for edge prompt (e.g., conditional prompting), adaptation for heterogeneous graphs (see the $\color{purple}{\text{purple}}$ part in Appendix E).

We hope that the revised manuscript can help address the concerns and resolve the issues raised by the reviewers.

Best,  
Authors of Submission 4905
</content>
</thread_object>

</thread>
<thread>
creation date:
1732376807598

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
Dear reviewers,

We sincerely appreciate your time and effort to review our paper.
We are happy to see the reviewers' recognition of our paper's strengths, including ***clear motivation*** (Reviewer D2XH, 7GLy), ***theoretical analysis*** (Reviewer 7GLy, t54u), ***comprehensive experimental evaluations*** (Reviewer WRHJ, 7GLy, t54u), and ***good presentation*** (Reviewer D2XH, 7GLy, t54u). 

You insightful suggestions are important to our paper. We have provided point-by-point responses to reviewers' comments and updated corresponding sections in our PDF. We think our responses have fully addressed your concerns  — in light of this, **we hope you consider raising your score**. Please let us know in case there are any other concerns, and if so, we would be happy to respond.

Best,  
Authors of Submission 4905
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes EdgePrompt, a graph prompt tuning method that enhances GNNs by learning prompt vectors for edges, improving graph representations. The reviewers agree that is well-organized, with clear points, and is easy to follow. The effectiveness of EdgePrompt is theoretically guaranteed, and it performs excellently in downstream tasks. Although, some of the reviewers noted that since this work emphasizes edge-level prompt tuning, it would be beneficial for the authors to explore edge-related tasks, such as edge classification and link prediction, to further expand the scope of the paper.