Curriculum GNN-LLM Alignment for Text-Attributed Graphs

Authors: Haibo Chen, Xin Wang, Zeyang Zhang, Haoyang Li, Weigao Wen, Ling Feng, Wenwu Zhu
Primary area: learning on graphs and other geometries & topologies
Keywords: Graph neural networks, Large language models, Text-attributed graphs, Curriculum learning

Abstract:
Aligning Graph Neural Networks (GNNs) and Large Language Models (LLMs) benefits in leveraging both textual and structural knowledge for Text-attributed Graphs (TAGs) learning, which has attracted an increasing amount of attention in the research community. Most existing literature assumes a uniformly identical level of learning difficulties across texts and structures in TAGs, however, we discover the $\textit{text-structure imbalance}$ problem in real-world TAGs, $\textit{i.e.}$, nodes exhibit various levels of difficulties when learning different textual and structural information. Existing works ignoring these different difficulties may result in under-optimized GNNs and LLMs with over-reliance on either simplistic text or structure, thus failing to conduct node classifications that involve simultaneously learning complex text and structural information for nodes in TAGs. To address this problem, we propose a novel Curriculum GNN-LLM Alignment ($\textbf{CurGL}$) method, which strategically balances the learning difficulties of textual and structural information on a node-by-node basis to enhance the alignment between GNNs and LLMs. Specifically, we first propose a text-structure difficulty measurer to estimate the learning difficulty of both text and structure in a node-wise manner. Then, we propose a class-based node selection strategy to balance the training process via gradually scheduling more nodes. Finally, we propose the curriculum co-play alignment by iteratively promoting useful information from GNNs and LLMs, to progressively enhance both components with balanced textual and structural information. Extensive experiments on real-world datasets demonstrate that our proposed $\textbf{CurGL}$ method is able to outperform state-of-the-art GraphLLM, curriculum learning, as well as GNN baselines. To the best of our knowledge, this is the first study of curriculum alignment on TAGs.

Decision: reject
Reviewer scores: 5, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1729291575417

thread_type:
official_review

<thread_object>
creator: reviewer_s8ol

<content>
summary:
This paper presents CurGL, a curriculum-based alignment method for Text-Attributed Graphs that introduces a text-structure difficulty measurer, a class-based node selection strategy, and curriculum co-play alignment to iteratively enhance Graph Neural Networks and Large Language Models. The experiments demonstrate the superiority of the method.

strengths:
- The writing is fluent and easily comprehensible.
- It is one of the earliest works to apply Curriculum Learning in the field of TAGs.
- The authors propose a class-based hard node selection strategy and a text-structure difficulty measurer, which serve as an inspiration for the community.

weaknesses:
- There is a lack of explanation and mathematical proof as to why the EM algorithm is used instead of updating the model parameters of LLM and GNN simultaneously.
- The experiments in this work are insufficient, as they only include node classification experiments without covering graph classification or link prediction. Additionally, I have some concerns about the current experimental results, see below.

questions:
1. Could the authors provide more insights into the selection of the $\lambda$ and $\alpha$ values? For different datasets (Cora and Citeseer in Figure 4), they seem to require different values.
2. In section 4.4, why does the accuracy of pseudo-labeled nodes selected by CurGL decrease as the number of steps increases?
3. In Section 4.5, why do both of the two components, LLM and GNN, exhibit opposite trends across different datasets?
4. In the main results, it seems that a simple GCN outperforms models like TAPE and ENGINE.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605411326

thread_type:
official_review

<thread_object>
creator: reviewer_zpuw

<content>
summary:
This paper tackles the text-structure imbalance problem in representation learning on text-attributed graphs. The authors propose a curriculum GNN-LLM alignment method to balance the learning difficulties of textual and structural information.

strengths:
The paper is clearly written and easy to follow.

The proposed method demonstrates improved performance.

weaknesses:
The novelty of this paper is incremental. It is evident that the ideas of difficulty measurement and curriculum learning are borrowed from existing works, such as [1,2]. Additionally, using loss as a measure of learning difficulty is a well-adopted approach.

The authors emphasize the text-structure imbalance problem, i.e., the learning difficulties of different nodes vary depending on textual attributes and topological structures. However, they provide only an intuitive example in figure 1 to illustrate this issue, without presenting real experimental results to substantiate the existence of this problem.

In Table 1, it is unusual that SOTA methods perform worse than the baseline GCN. An explanation for this unexpected phenomenon is needed.

What is the computational efficiency of the proposed model? Calculating node difficulty involves computing shortest paths between nodes, which increases computational complexity and may limit the model’s applicability to large-scale datasets.

The integration of curriculum learning into the training stage is unclear. According to Algorithm 2, the training process appears closely similar to that of the existing method [2].

The mathematical representation of equations and loss functions lacks clarity. For example:

   a. In Eq. (1), what does "MLP" represent? Is it a large language model or a multi-layer perceptron?  
   b. What is the relationship between $\hat{y}$ and $f_\theta(S)$? Are they equal?  
   c. In Eq. (6), there is no $y_v$. What does $1$ represent? Is it a column vector or a scalar? Additionally, if $D_t$ is larger, does this imply that the loss is also larger?

[1] Mitigating label noise on graph via topological sample selection. 

[2] Learning on large-scale text-attributed graphs via variational inference.

questions:
Does selecting nodes with lower difficulty improve classification accuracy on nodes with higher difficulty?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653472223

thread_type:
official_review

<thread_object>
creator: reviewer_s9p5

<content>
summary:
The paper introduces a method called Curriculum GNN-LLM Alignment designed for Text-Attributed Graphs (TAGs). The goal is to address the text-structure imbalance problem in TAGs, where nodes have varying levels of difficulty in learning textual and structural information. The CurGL approach progressively balances these difficulties through three key modules.

strengths:
1.	The integration of node difficulty into node classification is clearly motivated and engaging.
2.	The presentation is well-organized, and the ablation study clearly demonstrates the impact of each technique.

weaknesses:
1.	My primary concern is whether this method is effective for heterophilic graphs, as the proposed difficulty measurer seems tailored to homophilous graphs.

2.	The class-based node selection assumes balanced class sampling within each subgraph. However, this assumption may be problematic for highly imbalanced datasets, potentially leading to biased or suboptimal node selection.

3.	Related to the above, it’s unclear how the authors handle edges between selected nodes in the class-based node selection. In each subgraph, nodes from different classes are sampled proportionally, which does not guarantee connectivity among them.

4.	Another major concern is computational overhead. This method involves additional steps beyond GLEM, which already has considerable running time[1]. Efficiency analysis is essential and appears missing from the experimental section.

5.	The performance appears insufficiently strong; for instance, the results on Arxiv are weaker than those of methods like LEADING[2]. Larger datasets, such as ogbn-products and ogbn-papers100M should also be included.

6.	The paper’s claim of using LLMs seems overstated, as small language models are used. To solve this obvious mismatch and align claims with experiments , it would be more appropriate to use models like LLAMA or GPT.

7.	It is unclear if this approach generalizes to tasks beyond node classification, such as link prediction and graph classification. Focusing solely on node classification may limit the overall contribution.

8. I am curious why GLEM achieves strong performance on Cora and Pubmed, which have very few labels. The results seem inconsistent with findings from existing works[1].

9.  Similar to GLEM, this method relies heavily on the quality of pseudo-labels. Although an ablation study is provided, it does not fully address potential issues that could arise with low labeling ratios. 

[1] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs

[2] Efficient End-to-end Language Model Fine-tuning on Graphs

questions:
1. How does the implementation achieve sampling of simpler nodes first?
2. Are the pretrained language models in the third block in Table 1 fine-tuned with labels?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723535443

thread_type:
official_review

<thread_object>
creator: reviewer_t915

<content>
summary:
The paper introduces curriculum GNN-LLM alignment to tackle the text-structure imbalance problem in learning on text-attributed graphs. It proposes a node-wise difficulty measurer, a class-based node selection strategy, and a curriculum co-play alignment to balance and enhance learning between GNNs and LLMs. Extensive experiments show CurGL outperforms selective existing models on the classic node classification task across five well-known datasets.

strengths:
- The motivation to balance the role of LLMs and GNNs in a fine-grained sample-wise manner for TAG learning is straightforward and convincing.
- Writing is clear and easy to follow. Pseudo code is present to assist better understanding of the mechanisms designed.

weaknesses:
- Novelty. The E-M training between LLMs and GNNs has been explored by a handful of pioneer works, e.g., SimTEG and GELM. The proposed CurGL enhances such framework by adding additional strategies for balancing the LLM and the GNN.
- Unclear and potentially unfair experimental setting for baseline llm-as-predictor models. Is is unclear that how are the llm-as-predictor models testified, especially in Table 1. Is is based on a zero-shot manner as in Glbench, or are they also trained with the same training set? In terms of training, is it training-from-scratch or fine-tuning?
- Lack of discussion of a very relevant work, GraphGPT: Graph Instruction Tuning for Large Language Models.
- The proposed CurGL relies heavily on the availability of training data, and shows poor potential in transferring learning and zero-shot generalisation. Although it leverages an LLM for textual-based learning, the design of node-wise difficulty measurer and class-based selection strategy requires high-quality labels for the model to learn and converge. While it is quite reasonable for a graph learning model, the trained CurGL cannot be adapted to unseen data (unlike LLaGA, OFA and others), making its potential application quite narrow.
- Can CurGL generalise beyond node classification, to link prediction or graph classification?

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732500837398

thread_type:
official_comment

<thread_object>
creator: area_chair_l3qt

<content>
title:
Please reply to the authors' response.

comment:
Dear reviewers,

The ICLR author discussion phase is ending soon. Could you please review the authors' responses and take the necessary actions? Feel free to ask additional questions during the discussion. If the authors address your concerns, kindly acknowledge their response and update your assessment as appropriate.


Best,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733300348162

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers(1/2)

comment:
We sincerely appreciate all reviewers' insightful and valuable suggestions in reviewing our paper.    We are truly grateful for your recognition of our contributions: 

| **Aspect**                     | **Comment**                                                  | **Reviewer**  |
| ------------------------------ | ------------------------------------------------------------ | ------------- |
| **Motivation**                 | "The motivation to balance the role of LLMs and GNNs in a fine-grained sample-wise manner for TAG learning is straightforward and convincing." | Reviewer t915 |
|                                | "The integration of node difficulty into node classification is clearly motivated and engaging." | Reviewer S9p5 |
| **Presentation**               | "Writing is clear and easy to follow. Pseudo code is present to assist better understanding of the mechanisms designed." | Reviewer t915 |
|                                | "The presentation is well-organized, and the ablation study clearly demonstrates the impact of each technique." | Reviewer S9p5 |
|                                | "The paper is clearly written and easy to follow."           | Reviewer ZPUW |
|                                | "The writing is fluent and easily comprehensible."           | Reviewer s8oL |
| **Novelty**                    | "It is one of the earliest works to apply Curriculum Learning in the field of TAGs." | Reviewer s8oL |
|                                | "The authors propose a class-based hard node selection strategy and a text-structure difficulty measurer, which serve as an inspiration for the community." | Reviewer s8oL |
| **Experimental Effectiveness** | "The proposed method demonstrates improved performance."     | Reviewer ZPUW |

***

The reviewers also raised some concerns about our work. Here, we aim to address a few common concerns and will provide detailed responses to others individually.

> R1: The differences between the **EM training** in this paper and previous works.

The previous GraphLLM(e.g. GLEM[1]) paper employs the EM algorithm for jointly training LLM and GNN. However, our motivation lies in addressing the issue of text-structure imbalance in node representation learning. While the EM algorithm serves as a classical approach, our Curriculum Co-play Alignment method introduces the following unique aspects:

1. **Curriculum strategies tailored to text-structure imbalance**: To tackle the imbalance issue, we design different curriculum strategies for the GNN and LLM within the EM framework. Specifically, the GNN processes structure-difficulty nodes progressively, from easy to hard, while the LLM focuses on text-difficulty nodes. This ensures both components undergo comprehensive training and avoid shortcuts.
2. **Iterative enhancement with confident pseudo-labels**: A key objective is to iteratively integrate valuable information between the LLM and GNN, progressively enriching both components with a balance of textual and structural insights. Unlike methods that rely on random pseudo-label sampling often introducing significant label noise, our curriculum strategy emphasizes the selection of confident pseudo-labels to facilitate mutual learning between the components. As shown in Figures 5(a) and 5(b), our method effectively selects high-confidence pseudo-labels.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the text-structure imbalance problem in Text-Attributed Graphs (TAGs) by aligning Graph Neural Networks (GNNs) and Large Language Models (LLMs) using a curriculum learning approach. The proposed Curriculum GNN-LLM Alignment (CurGL) method balances the learning difficulties of textual and structural information on a node-by-node basis, enhancing the alignment between GNNs and LLMs and improving performance on node classification tasks.

The review highlighted the paper’s strengths, including clear writing, an innovative approach, and improved performance. However, concerns were raised about incremental novelty, computational complexity, and the limited scope of experiments, which focused only on node classification. Reviewers also had questions about the selection of hyperparameters, explanation of trends in results, and computational efficiency. The rebuttal seems to partially address the concerns. Most reviewers did not change their ratings. Overall comments suggest a weak rejection.