GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model

Authors: Haotong Yang, Xiyuan Wang, Qian Tao, Shuxian Hu, Zhouchen Lin, Muhan Zhang
Primary area: learning on graphs and other geometries & topologies
Keywords: GNN, LLM

Abstract:
Recent research on integrating Large Language Models (LLMs) with Graph Neural Networks (GNNs) typically follows two approaches: LLM-centered models, which convert graph data into tokens for LLM processing, and GNN-centered models, which use LLMs to encode text features into node and edge representations for GNN input. LLM-centered models often struggle to capture graph structures effectively, while GNN-centered models compress variable-length textual data into fixed-size vectors, limiting their ability to understand complex semantics. Additionally, GNN-centered approaches require converting tasks into a uniform, manually-designed format, restricting them to classification tasks and preventing language output. To address these limitations, we introduce a new architecture that deeply integrates GNN with LLM, featuring three key innovations: (1) Structure-Aware Transformers, which incorporate GNN’s message-passing capabilities directly into LLM’s transformer layers, allowing simultaneous processing of textual and structural information and generating outputs from both GNN and LLM; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges, ensuring complete semantic integration; and (3) GNN-LLM Twin Predictor, enabling LLM’s flexible autoregressive generation alongside GNN’s scalable one-pass prediction. GL-Fusion achieves outstand performance on various tasks. Notably, it achieves state-of-the-art performance on OGBN-Arxiv and OGBG-Code2,

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729860703131

thread_type:
official_review

<thread_object>
creator: reviewer_yare

<content>
summary:
The paper proposes a new architecture to address the limitations of LLM-centered models in capturing structural information and GNN-centered models in capturing textual semantics. This architecture includes three components: (1) Structure-Aware transformer, which integrates GNN and LLM into the same transformer layer, (2) Graph-Text Cross-Attention, allowing direct aggregation of node textual information, and (3) GNN-LLM Twin Predictor, which adds an additional layer to the output of LLMs to adapt to downstream tasks.

strengths:
- The motivation is interesting.

weaknesses:
1. The paper is not well written. 
- The use of notations presents a significant problem so that I could not understand the paper. In the preliminaries section, $L_n$ occurs only once. I think that $L$ describing the length of the node text in Line 129 should always be $L_n$. In section 3.2, Equation 3 and Equation 4 are very unclear. None of $T_i, W_q, W_k, X_v$ is defined. It is not clear to me whether $T_i$ in this context refers to the T_i of Equation 2. In line 238, the authors also admits to abusing the notations, and I don't understand why the authors didn't make further corrections.
- The order of Table 1 and Table 2 is also confusing.
- I am suspicious of the time complexity of the cross attention layer because of the confusion of the notations.
- The data format of the table is not aligned.
- There is a large amount of blank data in Table 3 with no further explanation as to why.

2. Lack of important ablation studies and hyperparameter sensitivity analysis. 

The paper proposes three modules, but in the experiments, only 'w/o cross-attention' is included in the graph property prediction task. In general, aggregators we will only use one way, such as max or sum. I'm not sure what the motivation was for designing all three and how they affected the final experimental results in the experiment? I would like to understand the difference between them and the type of information captured, especially the standard deviation. Moreover, the experiments did not indicate the need for a gating mechanism in Equation 2.

3. Detailed descriptions of the dataset and baseline methods are missing.

This paper covers a large number of experimental tasks, but in the appendix, detailed information on the datasets as well as information on the baseline methods are not listed.

4. Lack of important baseline methods. The baseline methods should be divided into three parts, (1) LLM-centered models, (2) GNN-centered models, and (3) combined LLMs and GNNs methods.

- [1] Huang, Qian, et al. "Prodigy: Enabling in-context learning over graphs." Advances in Neural Information Processing Systems 36 (2024).
- [2] Tan, Yanchao, et al. "MuseGraph: Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining." arXiv preprint arXiv:2403.04780 (2024).
- [3] Chien, Eli, et al. "Node feature extraction by self-supervised multi-scale neighborhood prediction." arXiv preprint arXiv:2111.00064 (2021).
- [4] Huang, Xuanwen, et al. "Prompt-based node feature extractor for few-shot learning on text-attributed graphs." arXiv preprint arXiv:2309.02848 (2023).

5. I also doubt the reasonability of the proposed model. Consider that we want to do node classification task only. Given limited number of labeled nodes, the model has significant number of trainable parameters than standard GNNs. However, I do not think the proposed model can outperform GNNs under small labeled datasets. So is the model really needed? I do not think so.

questions:
Please see the comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570660005

thread_type:
official_review

<thread_object>
creator: reviewer_xxby

<content>
summary:
The manuscript proposes a method that deeply integrates Large Language Models (LLMs) and Graph Neural Networks. The method features structure-aware transformers, graph-text cross attention, and GNN-LLM twin predictor, enabling it to perform tasks of various types related to graphs and language.

strengths:
1. The method integrates LLMs and GNNs deeper than previous methods by introducing 3 key components.
2. The manuscript has conducted experiments on downstream tasks of various types, including traditional node classification, knowledge-related tasks and text generation, demonstrating the generalizability of the proposed method.

weaknesses:
1. Experiments are not comprehensive. Baselines and benchmarks are missing. For example, for the node classification tasks, there are only two datasets considered. In Table 3, many entries in the table are missing, which makes the results less convincing. I suggest the author refer to [1] for benchmarking models on text-attributed graphs.

2. The manuscript proposes a twin predictor for node classification. However, edge-level and graph-level tasks should also be considered given their importance in graph-related tasks. Regression tasks should also be considered.



[1] Yan, Hao, et al. "A comprehensive study on text-attributed graphs: Benchmarking and rethinking." Advances in Neural Information Processing Systems 36 (2023): 17238-17264.

questions:
In the Graph-text attention layer, the language tokens do not attend edges. Would the loss of the information affect performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695074676

thread_type:
official_review

<thread_object>
creator: reviewer_b4va

<content>
summary:
This paper presents GL-Fusion, an architecture integrating the capabilities of graph neural networks (GNNs) and large language models (LLMs). The proposed architecture attempts to address limitations in existing methods for GNN-LLM fusion by carefully embedding GNN structure within LLMs to leverage both structural graph data and rich textual information. Components of the architecure include structure-aware Transformer layers, cross-attention between graph and text, and GNN-LLM integrated predictor, which together tries to enable simultaneous handling of textual and structural features while providing flexibility in output. GL-Fusion shows improvement across various tasks, outperforming baseline GNN-centered, LLM-centered, and hybrid models from the recent literature.

strengths:
- the design of the architecture builds upon the recent research on LLMs for text attributed graphs and addresses some limitations, especially of the existing approaches which only rely on either GNN or LLM as the main model.
- the permutation invariance of graph node tokens is carefully modeled in the structure-aware transformer layer.
- both kinds of gnn based and text based tasks can be conceptually handled using the proposed method.

weaknesses:
Limitations and questions:
1. if there is a common understanding that self attention is a form of message passing, why is there a need to incorporate a separate MPNN in the structure aware Transformer layer? This aspect is unclear and has not been empirically studied as well. 
2. in page 5, the positional encoding which would be ideal for both text and graph tokens is discussed. I am wondering if it is possible to have a unified structure based positional encoding which perceives both sequential structure and graph structure as 'arbitrary graph structure'? This is in context of recent works in graph transformer literature which show laplacian eigenvectors can generalize the text positional encoding of Transformers.
3. although the proposed  aims to reduce complexity, the integration of graph structures into LLM layers may still encounter scalability issues with large graphs or highly interconnected datasets. How does GL-Fusion handle memory usage and processing time for very large graph structures?

questions:
included with weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730966092374

thread_type:
official_review

<thread_object>
creator: reviewer_yrc1

<content>
summary:
This paper proposed a method to combine GNN and transformer architecture to learn the graph tasks. Specifically, the graph tokens and text tokens are learned together in a sequence, while the graph tokens are only attending to other graph tokens in the same graph. The graph structures are learned through integrating GNN into prediction layer. The graph tokens are learned by cross attention module over the text tokens.

strengths:
1. The proposed architecture provides a feasible solution to combine transformer architecture and graph neural networks to learn on graph tasks.

2. The paper is well written and easy to follow.

3. The reported experimental results are good.

weaknesses:
1. The proposed method is complex which includes three components: text and graph token transformers, text to graph token attention module, and GNN prediction module. The optimization process on three components seem non-trivial. However, the authors did not discuss any details on it. 

2. The method needs to train a new transformer architecture, which seems not easy to combine with existing pre-trained LLMs. Can authors describe if it is possible to include the power of pre-trained language models?

3. The included datasets for node classification is limited (only two datasets). The reported results in link prediction table lacks most of the comparison results.

questions:
For optimization process, is this method using back propagation to directly optimize all parameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732540317061

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Stage is Running Out

comment:
Dear reviewers:

Thank you for your thoughtful and constructive feedback on our work. We have carefully revised our paper to address your comments and made the following improvements:


1. Add more node classification datasets. 

We have incorporated five more node classification datasets from [1]. The results are as follows:

|            |           | PLM-Based |        |  GNN-Based |        |         |         |  Co-Training Based |          |
|------------|-----------|-----------|--------|------------|--------|---------|---------|--------------------|----------|
| Model      | GL-Fusion | Tiny      |   Base |  T-GCN     | B-GCN  | T-SAGE  |  B-SAGE |  GCN(T)            | SAGE(T)  |
| Children   | **61.19**     | 49.85     | 59.91  | 57.07      | 58.11  | 57.57   | 58.74   | 54.75              | 59.7     |
| History    | **86.16**     | 83.06     | 86.09  | 84.52      | 85.04  | 84.79   | 85.12   | 83.52              | 85.09    |
| Photo      | 82.89     | 73.75     | 77.53  | 82.42      | 82.7   | 83.25   | 83.27   | 83.32              | **86.64**    |
| Computers  | **88.91**     | 58.32     | 60.4   | 87.43      | 87.86  | 87.9    | 88.3    | 83.93              | 86.04    |
| Sports     | **93.33**     | 81.47     | 86.02  | 84.93      | 86.16  | 87.06   | 87.34   | 85.06              | 85.87    |

Our model, **GL-Fusion, shows significant improvements on the Children, Computers, and Sports datasets**. For the Photo dataset, GL-Fusion outperforms most models except those based on GraphSage.

2. Complete link prediction baseline metrics.

In the link prediction table, baselines had previously conducted experiments on only some metrics or subsets of datasets, leading to incomplete comparison. We have added baseline results for GraiL, NBFNet, and UniLP. **Our GL-Fusion still outperforms all baselines across all metrics and datasets.**

3. Add more baseline models.

We have added [2, 3, 4] in revision. Our GL-Fusion outperforms them significantly.

[2] is a LLM-centered model. On ogbn-arxiv dataset, our model outperforms it significantly, achieving 77.80 micro-F1 and 61.40 macro-F1, while [2] only achieves 69.20 micro-F1 and 42.45 macro-F1. 

[3] combines LLMs and GNNs.   It achieves 76.12 accuracy on ogbn-arxiv, while our GL-Fusion achieves 78.20 accuracy.

[4] is a GNN-centered models.  It achieves 52.91 and 62.12 ogbn-arxiv test accuracy with 10-shot and 100-shot training example per class, while our GL-Fusion achieves  54.66 and 68.18, respectively. The results are shown in the following Table.

|  #shots per class    | 10    | 100    |
|--------------------|-------|--------|
| GL-Fusion          | **56.44** | **68.18**  |
|  OGB-Feature       | 45.76 | 58.75  |
| PLM+GAE            | 50.16 | 58.1   |
|  PLM+GAE+prompt    | 51.89 | 60.63  |
|  GIANT             | 50.5  | 60.81  |
|  GIANT+prompt      | 51.4  | 61.26  |
|  PLM-cls           | 46.97 | 58.69  |
|  PLM-Prompt-dense  | 51.17 | 58.65  |
|  PLM-Prompt-sparse | 52.01 | 60.85  |
|  G-Prompt          | 52.48 | 61.67  |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732733715560

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Eagerly Awaiting Your Feedback

comment:
Dear Reviewers,

Thank you once again for dedicating your time to reviewing our work and for providing such detailed comments and concerns!

During the rebuttal process, we have carefully addressed each of your points and hope that our responses have resolved your concerns. If there is anything that remains unclear or if you have additional questions, we would be very happy to discuss further.

As deadline for revision is approaching, we kindly hope you to review our responses and let us know if they satisfactorily address your concerns.

We greatly appreciate your time and consideration. Wishing you a wonderful day!

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) The paper proposes GL-Fusion, a new architecture integrating GNN and LLM to address limitations of existing methods. It includes Structure-Aware Transformers, Graph-Text Cross-Attention, and GNN-LLM Twin Predictor. Claims to outperform baselines on various tasks.

(b) Strengths: Novel architecture with potential to handle text and graph data jointly. Some reviewers noted good experimental results and well-written presentation.

(c) Weaknesses: Reviewers criticized the paper for complexity with unclear optimization, lack of details on handling large graphs, limited datasets initially, and notation issues. Some questioned the need for the proposed model compared to simpler GNN or LLM models.

(d) Reasons for rejection: Despite the authors' efforts in the rebuttal to address concerns like adding more datasets and improving notation, significant weaknesses remain. The model's complexity and training cost compared to simpler alternatives are major drawbacks. The need for such a complex combination of GNN and LLM for tasks that might be adequately handled by simpler models is not convincingly justified. There are also remaining concerns about the paper's clarity and the practicality of the proposed method.