Multi-modal Prompt Learning Empowers Graph Neural Networks with Semantic Knowledge

Authors: Zihao Li, Lecheng Zheng, Bowen Jin, Dongqi Fu, Baoyu Jing, Yikun Ban, Jingrui He, Jiawei Han
Primary area: foundation or frontier models, including LLMs
Keywords: Graph Foundation Model, Multi-modal Prompt Learning, Graph Neural Network, Language Models, Contrastive Learning

Abstract:
While great success has been achieved in building generalizable language models, three fundamental issues hinder GNN-based graph foundation models: the scarcity of labeled data, different levels of downstream tasks, and the conceptual gaps between domains. In depth, though the labels of real graphs are associated with semantic information, most graph learning frameworks ignore it by turning semantic labels into numerical labels. In this work, to address these issues, we present a new paradigm that leverages the text modality to align downstream tasks and data with any pre-trained GNN given only a few semantically labeled samples. Our paradigm embeds the graphs directly in the same space as the LLM by learning both graph prompts and text prompts simultaneously. To accomplish this, we improve state-of-the-art graph prompt method based on our theoretical findings. Then, we propose the first multi-modal prompt learning approach for exploiting the knowledge in pre-trained models. Notably, in our paradigm, the pre-trained GNN and the LLM are kept frozen, so the number of learnable parameters is much smaller than fine-tuning any pre-trained model. Through extensive experiments on real-world datasets, we demonstrate the superior performance of our paradigm in few-shot, multi-task-level, and cross-domain settings. Moreover, we build the first zero-shot classification prototype that can generalize GNNs to unseen classes. The code is provided in the supplementary materials.

Decision: reject
Reviewer scores: 5, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730448627575

thread_type:
official_review

<thread_object>
creator: reviewer_kgke

<content>
summary:
This paper introduces a multimodal prompt learning method called Morpher that uses both graph and text prompts to align graph representations with LLM’s semantic embeddings. This method is able to do zero-shot classification for unseen graph classes. In their paradigm, the pre-trained GNN and the LLM are kept frozen, making it much more efficient than finetuning. They also introduce an improved graph prompting method that cross-connections between the prompt graph and the input graph share the same density with the input graph. By using a projection layer and training with contrastive loss, they align the graph and text prompt embeddings in the same space, allowing for effective adaptation to various tasks with limited downstream data.

strengths:
1. Morpher is able to adapt to various downstream tasks with limited labels. It is also able to do zero-shot classification on unseen classes.
2. They introduce an improved method of graph prompting without overwhelming or limiting the input graph's features.
3. The use of a contrastive loss function to align the graph and text embeddings helps bridge the gap between graph structure and semantic text features, enhancing the model's flexibility across modalities.

weaknesses:
1. The effectiveness of Morpher relies on the availability and quality of semantic labels in the text associated with graph data. When these labels are noisy, incomplete, or inconsistent, the alignment might be weakened and the performance of downstream tasks might be reduced.
2. Freezing GNN and LLM reduces the number of trainable parameters, but it also limits adaptation flexibility. Certain tasks might require fine-tuning the GNN or LLM to capture task-specific information.
3. The independently pre-trained GNN and LLM may cause huge representation gaps between the two modalities, causing difficulty in the alignment process.
4. Applying the graph prompting might be computationally expensive on large and complex graphs.

questions:
1. How does different text labels affect the model performance? Does randomizing the class text label lead to comparable performance in few-shot setting? How fine-grained text can the model take? Will adding explanations to the text label help improve the performance?
2. Why does the test accuracy of zero-shot class decrease as the training epoch increases?
3. What’s the distribution of graph and text representation? Does there still exist domain shift after the alignment? Can you visualize the distributions of graph and text representations before and after alignment, using t-SNE or UMAP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730485889379

thread_type:
official_review

<thread_object>
creator: reviewer_m9xe

<content>
summary:
This paper proposes a new framework for integrating pre-trained GNNs with LLMs, leveraging the reasoning capabilities of LLMs alongside the structural representation advantages of GNNs. Specifically, the authors prompt both the LLM and the GNN with trainable parameters while keeping the parameters of both base modules frozen. This prompting, along with a cross-model projector, assists in aligning the embeddings produced by the GNN and the LLM for an input instance, such as a graph. Additionally, they introduce a variant of a current GNN prompting method that addresses its shortcomings through an attention mechanism.

strengths:
- The proposed method does not involve training the base GNN or LLM, which reduces the number of parameters needed for training and enhances complexity and versatility.
- The authors reasonably justify the shortcomings of one current state-of-the-art model and propose modifications to mitigate its issues, empirically demonstrating the improvements achieved by these modifications.
- The main part of the prompting method, Morpher, is explained clearly with formulations, making it easy to understand.
- The authors provide sufficient and clear visualizations for their proposed methods.

weaknesses:
- The authors do not compare their work with state-of-the-art approaches that combine LLMs and GNNs, such as [1]
- While the authors claim to provide a theoretical analysis that "in many cases, state-of-the-art graph prompt is unable to learn good representations," this analysis is limited to Lemma 3.1. This lemma presents a trivial conclusion that does not significantly contribute to the development of the method or elaborate on the effectiveness of current prompting methods. It may be beneficial to remove this lemma and its proof, as it has been widely discussed in the literature on GNNs, and cite related works [2], which would help streamline the paper.
- In the Background section, few-shot prompt learning is only addressed in the context of methods that learn trainable parameters. It would be valuable to also discuss methods that do not train parameters, particularly regarding the intrinsic in-context learning of these methods.
- Some definitions and formulations appear extraneous. For instance, Equation 2 addresses concepts that are reiterated in Equation 6 and relate to matching dimensionality. The term $\tilde{h}$ does not appear in subsequent equations, and similarly, the introduction of $e^t$ in Equation 5 is later replaced by $e^t_{norm,i}$ without further context. Streamlining these sections could reduce unnecessary complexity.
- The authors do not discuss the inference process of their method at test time. It would be helpful to clarify whether the same model must be used at inference time and whether the LLM or the GNN makes the final prediction.
- The claim that this method is the first to address zero-shot classification for GNNs to unseen classes may overlook existing approaches. Referencing works such as [1] and [3] and comparing with them would provide necessary context.
- While the authors mention generating textual features from numerical features of the original datasets, it appears that for node classification datasets, only the labels of the nodes are converted to text, as indicated in Appendix B.2, rather than the node features themselves. Moreover, converting high-dimensional feature vectors to text features is not trivial and could exceed the acceptable input length for LLMs. This oversight may limit the model's generalizability to node/edge-level tasks due to the neglect of node features.


[1] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, & Muhan Zhang (2024). One For All: Towards Training One Graph Model For All Classification Tasks. In The Twelfth International Conference on Learning Representations.

[2] Keyulu Xu, Weihua Hu, Jure Leskovec, & Stefanie Jegelka (2019). How Powerful are Graph Neural Networks?. In International Conference on Learning Representations.

[3] Chen, J., Mi, R., Wang, H., Wu, H., Mo, J., Guo, J., Lai, Z., Zhang, L., & Leung, V. (2024). A Review of Few-Shot and Zero-Shot Learning for Node Classification in Social Networks. IEEE Transactions on Computational Social Systems, 1-15.

questions:
- In Section 3, it is stated that the number of cross-connections is controlled to $n_e$ by thresholding. How are the best edges selected? Is this selection random, or is there a specific criterion?
- The objective function used for training Morpher does not incorporate labels or the task but focuses solely on aligning the prompting parameters for the GNN and LLM. This raises concerns about potential misalignment with pre-trained task-specific information, as merely aligning representations without consideration of the task could lead to forgetting previously learned knowledge. What is the rationale behind this objective function, and how does it mitigate this issue?
- Given that the objective function does not involve labels and the GNN and LLM have frozen parameters, how is the model evaluated in a few-shot setting with labels? Does this imply that the base GNN and LLM are pre-trained, with prompting applied afterward without using labels? If so, the contributions to domain transfer and zero-shot learning may be limited, as the fundamental learning from one domain to another occurs prior to prompting. The proposed method may primarily align representations of the LLM and GNN, which has already been explored in few-shot settings within a single domain.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621240927

thread_type:
official_review

<thread_object>
creator: reviewer_9hu4

<content>
summary:
This paper introduces a new paradigm for enhancing Graph Neural Networks (GNNs) through multi-modal prompt learning, which leverages text data to align downstream tasks with pre-trained GNNs using only a few semantically labeled samples. The approach embeds graphs in the same semantic space as large language models (LLMs) by jointly learning graphs and text prompts. Building on theoretical insights, the authors improve existing graph prompt methods and propose a multi-modal framework that fully utilizes knowledge from pre-trained models without requiring fine-tuning of either the GNN or LLM. This setup significantly reduces the number of learnable parameters. Extensive experiments on real-world datasets demonstrate the paradigm's effectiveness across few-shot, multi-task, and cross-domain settings. Additionally, the framework includes a zero-shot classification prototype that enables GNNs to generalize to previously unseen classes.

strengths:
S1. The manuscript is well-organized and clearly written, with structured and detailed mathematical derivations. For instance, in the "Improved Graph Prompt Design" section, the paper rigorously derives the formulation for balancing cross-connections between input nodes and prompt nodes. 
S2. The proposed sparse cross-connection mechanism in Morpher prevents prompt features from overwhelming graph features, offering a balanced integration for improved feature alignment.
S3. Comprehensive experiments across diverse datasets, including Cora, CiteSeer, and synthetic zero-shot datasets, effectively demonstrate Morpher's adaptability and robustness.
S4. Results show that Morpher achieves superior performance over baselines in few-shot and zero-shot tasks, validating its cross-domain generalization capabilities.

weaknesses:
W1. This paper proposes a sparse cross-connection mechanism for Morpher to prevent prompt features from overloading graph features, but could this also limit information exchange between graph and text modalities, weakening alignment quality? For complex tasks requiring richer feature fusion, this limited interaction may be insufficient. 

W2. Morpher’s reliance on graph-text fusion assumes complementary information from both modalities, but in tasks with weaker associations, might this integration introduce noise and reduce performance?

W3. The paper does not analyze how differences in data distribution between pre-training and target datasets influence transfer performance, which is crucial for domain adaptation. Quantifying these differences (e.g., via JS or KL divergence) could clarify Morpher’s sensitivity to dataset similarity.

W4. The domain transfer experiment only tests on limited datasets (MUTAG and PubMed), which share similar characteristics in molecular biology. It is unclear if the findings generalize to more diverse domains like social networks.

W5. In the "Zero-Shot Classification Prototype" section, the zero-shot experiment relies on synthetic datasets (ZERO-Cora, ZERO-CiteSeer), which are generated by modifying features and labels and may fail to capture the complexity of natural data distributions. I wonder about Morpher’s generalization performance on real-world datasets.

W6. The author mentions "Figure 5" in the text, though no such figure exists; it seems they meant to refer to "Table 4" instead.

W7. Some relevant works are missing, e.g., Killing Two Birds with One Stone: Cross-modal Reinforced Prompting for Graph and Language Tasks, KDD 2024.
Natural Language Is All a Graph Needs. arxiv. 2023

questions:
Please refer to the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641628505

thread_type:
official_review

<thread_object>
creator: reviewer_3fnq

<content>
summary:
This paper proposes Morpher, which leverages the text modality to align downstream tasks and data with any pre-trained GNN given only a few semantically labeled samples. The key idea is to embed the graphs directly in the same space as the LLM by multi-modal prompt learning. Experiments demonstrate the superior performance of Morpher in few-shot, multi-task-level, and cross-domain settings.

strengths:
1.  The proposed Morpher can generalize GNN to unseen classes.
2.  The authors provide codes for reproducibility.
3.  This paper is easy to follow.

weaknesses:
1. The authors only show that AIO [76] is unable to learn good representations of the downstream data. In my opinion, some state-of-the-art graph prompt methods such as PRODIGY [Ref1] have addressed this issue.
2. From Appendix D.3 in [Ref1], PRODIGY can also apply to the zero-shot classification. Therefore, the comparison of PRODIGY and Morpher is necessary.
3. Some references such as [Ref2, Ref3] are missing.



[Ref1] PRODIGY: Enabling In-context Learning Over Graphs. NeurIPS 2023.

[Ref2] Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning. ICLR 2024.

[Ref3] Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Bias. TPAMI 2024.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732932074982

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers. Unfortunately, we have decided to withdraw this submission. Nonetheless, we have addressed the concerns raised in each review to provide clarification and ensure public transparency.
</content>
</thread_object>

</thread>


Metareview: