Query-Aware Learnable Graph Pooling Tokens as Prompt for Large Language Models

Authors: Wooyoung Kim, Byungyoon Park, Wooju Kim
Primary area: learning on graphs and other geometries & topologies
Keywords: Graph Neural Network, Large Language Model, Continuous Prompting, Sf

Abstract:
Graph-structured data plays a vital role in numerous domains, such as social networks, citation networks, commonsense reasoning graphs and knowledge graphs. While graph neural networks have been employed for graph processing, recent advancements have explored integrating large language models for graph-based tasks. In this paper, we propose a novel approach named Learnable Graph Pooling Token (LGPT), which addresses the limitations of the scalability issues in node-level projection and information loss in graph-level projection. LGPT enables flexible and efficient graph representation by introducing learnable parameters that act as tokens in large language models, balancing fine-grained and global graph information. Additionally, we investigate an Early Query Fusion technique, which fuses query context before constructing the graph representation, leading to more effective graph embeddings. Our method achieves a 4.13\% performance improvement on the GraphQA benchmark without training the large language model, demonstrating significant gains in handling complex textual-attributed graph data.

Decision: reject
Reviewer scores: 3, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730208956689

thread_type:
official_review

<thread_object>
creator: reviewer_m81s

<content>
summary:
- This paper proposes a learnable graph pooling module to enhance LLM-based GraphQA.

strengths:
- The combination of LLM and GNN is an important research topic.
- The design of this paper is reasonable.

weaknesses:
- The novelty seems to be limited in this paper because authors only made a new incremental design in the graph encoder. The core paradigm of graph QA is preserved compared with other baselines.
- Some important GNN+LLM baselines are missing in the experiments. For example, GNP [1].
- The training/inference efficiency of the method should be compared with other baselines.
- The detailed information about the graphs in each dataset is not reported.
- The original dataset and README instructions are not provided in the code, making it difficult to reproduce the performance.




[1] Graph neural prompting with large language models

questions:
- What is the meaning of Sf in the author keywords?
- See weaknesses and make some revisions to the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730616241864

thread_type:
official_review

<thread_object>
creator: reviewer_enae

<content>
summary:
The paper addresses the problem of Textual-Attributed Graph QA, divided into two main steps: sub-graph retrieval and answer generation. For answer generation, their approach transforms the sub-graph into textual embeddings through a prompt, generates embeddings, and then uses a graph encoder with learnable parameters to process them. The paper highlights scalability issues in node-level prompting (where each node is treated as a separate token in the language model) and information loss in graph-level projection (where the entire graph is compressed into a single vector). To address this, the authors propose Learnable Graph Pooling Tokens (LGPT), a pooling method that introduces learnable parameters (tokens) that connect to all nodes and perform message passing. This method allows for flexible, efficient graph representation that balances fine-grained and global information, achieving improved performance on Graph QA tasks.

strengths:
The paper is easy to read and understand. Extensive experiments and analysis have been shown to prove the proposed method.

weaknesses:
The idea of “early fusion” by forming an external node and fully connecting to other nodes in the graph is not novel to the field. The LGPT idea seems intuitive that increasing the number would increase the performance but would like to see more analysis here.

questions:
1. “ However, the key difference from these methods is that, instead of pooling into a single graph embedding, our approach uses multiple learnable tokens for pooling, thereby reducing information loss” - Is there a pattern in the information loss. Is there a way to quantify this loss other than looking at the accuracy? What kind of data samples perform better when we increase the number of LGPT? 

2. How does the number of LGPT performance vary with the different datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679926053

thread_type:
official_review

<thread_object>
creator: reviewer_75jm

<content>
summary:
This paper leverages graph neural networks and large language models for task of knowledge graph question answering. Based on recent proposed techniques include graph soft prompt and query-aware graph prompting. The author proposed query-aware graph pooling to overcome the limitations of node-level and graph-level representations. In experiments, it shows competitive performance on recent proposed graph QA benchmarks in different domains.

strengths:
1. The paper identifies a critical disadvantage of graph pooling method; the granularity control is either graph-level or node-level. 
2. On this pain point, the proposed multiple tunrable prompt (LGPT) effecvtively imrpove the performance on benchmarks.

weaknesses:
1. The novelty of the paper is questionable. As the author mentioned, recent work such as G-Retriever;Graph Token and GNP (Graph Neural Prompting) has covered most of the techniques used in the paper except the graph prompt paramters. However, the learnable graph prompt is proposed in multiple related work including [1] and supernodes (connect every node to a virtual node for pooling) in graph pooling [2] literature.

2. The proposed work re-uses most of the component of G-Retriever, which also causes my concern on cherry-picking hyperparameters given the performance improvements over G-retriever is subtle.




[1] Universal prompt tuning for graph neural networks, Neurips 2023
[2] Understanding Pooling in Graph Neural Networks,

questions:
1. What's the perfomance of LGPT in figure 1 without GNN and fine-tune lanaguage model (i.e. GraphToken with LLM fine-tuning)? It would be interesting to see whether design of graph pooling is still neccessary when LLM is tunable given that GNN introduces additional parameters.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692997114

thread_type:
official_review

<thread_object>
creator: reviewer_uwj1

<content>
summary:
The paper presents a novel approach for integrating graph representations with large language models (LLMs), addressing the critical challenge of efficient graph-text interaction. The primary contributions are twofold: (1) an early fusion mechanism that performs message passing between sub-graph node representations and query text embeddings, and (2) a learnable pooling strategy utilizing dedicated tokens (LGPT) that act as information aggregators within the graph structure.
The early fusion mechanism is particularly noteworthy as it enables direct interaction between textual and structural information at the embedding level, potentially capturing more nuanced relationships compared to traditional late fusion approaches. The authors implement this through message passing operations that allow bidirectional information flow between the sub-graph nodes and query text representations.
The learnable pooling strategy introduces fully-connected LGPT tokens that serve as dynamic information hubs within the graph. These tokens effectively aggregate information from all nodes through message passing, potentially creating a more comprehensive and adaptable graph representation. This approach appears to offer more flexibility than static pooling methods.

strengths:
1. The paper introduces an innovative early fusion mechanism that addresses a fundamental challenge in graph-language modeling: the seamless integration of structural and textual information; The learnable pooling tokens (LGPT) provide a flexible and adaptive approach to graph representation, offering advantages over traditional static pooling methods. 

2.The authors conduct extensive experiments across three diverse graph QA datasets, demonstrating the robustness and generalizability of their approach. The method achieves competitive performance compared to state-of-the-art baselines, while potentially offering improved computational efficiency.

weaknesses:
1. The paper's scalability argument lacks sufficient comparative analysis against existing methods like G-retriever and GraphToken; The authors do not provide a detailed complexity analysis or empirical benchmarks to substantiate their efficiency claims; While the authors assert improved efficiency compared to Tian et al. 2024 (Line 210), this claim requires further scrutiny since: a). The dominant computational cost typically lies in the LLM inference; b). The relative improvement in message passing efficiency may be marginal in the overall computational pipeline; c) No concrete timing or memory usage comparisons are provided. 
2. The evaluation is primarily confined to GraphQA tasks, leaving several important questions about generalization unexplored: a). The method's effectiveness on standard graph learning tasks (node classification, link prediction) remains unvalidated; b) The paper lacks a theoretical or empirical bridge between GraphQA performance and the claimed improvements in node-level and graph-level information integration. A broader evaluation across diverse graph-based tasks would strengthen the paper's contributions. 
3. The hyperparameter analysis in Section 4.4 shows significant gaps in the experimental design: The LGPT token count investigation only examines extreme values (8 and 32), omitting crucial intermediate points; The impact of other critical hyperparameters (e.g., message passing steps, fusion layer configurations) is not thoroughly explored. 
4. The paper should improve the methodological clarity from a). a more rigorous theoretical justification for the chosen LGPT architecture; b). Clear computational complexity analysis compared to baseline methods.

questions:
1. How sensitive is the model's performance to the choice of text encoder in Equation 7?
2. Have the authors experimented with different text encoders (e.g., BERT variants, RoBERTa, T5) and observed any significant variations in performance?
3. Regarding Equation 5, how does the choice of graph encoder architecture impact the model's performance?
4. Can the authors provide case studies or visualization analysis demonstrating how LGPT addresses information loss compared to baseline methods?
5. In Equation 9, please clarify the definition and dimensionality of $S_g$
6. For Equation 10, please provide a detailed explanation of $S_p$ and its role in the architectur
</content>
</thread_object>

</thread>
<thread>
creation date:
1732604381140

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
## Dear Reviewers

Thank you very much for your valuable reviews, which have greatly contributed to improving the quality of our research. We sincerely appreciate your thoughtful feedback.  
After thoroughly reviewing your comments, we have made the following major revisions:  

1. **Section 3.3**  
   We have explicitly addressed the issue of **Time Complexity**, which received significant attention, by incorporating the necessary details into the main text.

2. **Table 1**  
   We have added more detailed and comprehensive descriptions of the data.  

3. **Section 3.2**  
   We apologize for the incorrect use of the word "efficiency" in the original text (as non-native speakers, we may have lacked precision in our word choice, and we kindly ask for your understanding).  
   What we intended to convey was that embedding a small amount of information about $I_{query}$ is more **effective** than embedding all information about $I_{total}$.  
   Accordingly, we have revised the wording of this section.  

Thank you for your attention.

P.S.  Feel free to let us know if you have any further questions; we will happily answer them
</content>
</thread_object>

</thread>


Metareview:
metareview:
Scientific Claims and Findings: This paper presents a novel approach for integrating graph representations with large language models (LLMs) for graph question-answering tasks. The key contributions include A Learnable Graph Pooling Token (LGPT) mechanism that aims to bridge the gap between node-level and graph-level projections by using learnable parameters as tokens that connect to all nodes in the graph and an Early Query Fusion technique that incorporates query context before constructing graph representations, rather than after encoding the graph.

The paper identifies and addresses a real limitation in current graph-LLM integration approaches - the tradeoff between fine-grained node-level information and efficient graph-level representations. The proposed approach shows consistent performance improvements across different experimental settings, including both frozen and fine-tuned LLM scenarios.

There is limited novelty: the core ideas build heavily on existing work: The learnable graph prompt concept appears in prior work on universal prompt tuning for GNNs. The early fusion approach using virtual nodes has precedent in the graph pooling literature. Many components are borrowed directly from G-Retriever. 

The paper lacks a thorough complexity analysis comparing computational costs with baseline methods. There's limited investigation of why/how LGPT helps beyond empirical results. The hyperparameter analysis is sparse, particularly regarding LGPT token counts.

While the paper presents a well-executed study with clear empirical improvements, the primary concerns around novelty and technical depth suggest it falls slightly below the acceptance threshold. The core ideas are largely incremental combinations of existing techniques rather than fundamental advances. The missing analyses and comparisons also leave important questions unanswered about the method's efficiency and advantages. While the experimental results are positive, stronger theoretical justification and more comprehensive analysis would be needed to make this work more compelling for acceptance. The paper would benefit from addressing these gaps and potentially finding ways to differentiate its contributions more clearly from existing approaches.