When Do LLMs Help With Node Classification? A Comprehensive Analysis
Authors: Xixi Wu
Yifei Shen
Fangzhou Ge
Caihua Shan
Yizhu Jiao
Xiangguo Sun
Hong Cheng
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Graph Neural Networks
Node Classification
Primary area: deep_learning->graph_neural_networks
TL;DR: We release a testbed for LLM-based node classification algorithms, and provide 8 novel takeaways based on extensive experiments on this testbed.

Abstract: Node classification is a fundamental task in graph analysis, with broad applications across various fields. Recent breakthroughs in Large Language Models (LLMs) have enabled LLM-based approaches for this task. Although many studies demonstrate the impressive performance of LLM-based methods, the lack of clear design guidelines may hinder their practical application.  In this work, we aim to establish such guidelines through a fair and systematic comparison of these algorithms. As a first step, we developed LLMNodeBed, a comprehensive codebase and testbed for node classification using LLMs. It includes 10 homophilic datasets, 4 heterophilic datasets, 8 LLM-based algorithms, 8 classic baselines, and 3 learning paradigms. Subsequently, we conducted extensive experiments, training and evaluating over 2,700 models, to determine the key settings (e.g., learning paradigms and homophily) and components (e.g., model size and prompt) that affect performance. Our findings uncover 8 insights, e.g., (1) LLM-based methods can significantly outperform traditional methods in a semi-supervised setting, while the advantage is marginal in a supervised setting; (2) Graph Foundation Models can beat open-source LLMs but still fall short of strong LLMs like GPT-4o in a zero-shot setting. We hope that the release of LLMNodeBed, along with our insights, will facilitate reproducible research and inspire future studies in this field. Codes and datasets are released at https://llmnodebed.github.io/.

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Xixi Wu Yifei Shen 2 Fangzhou Ge'
Caihua Shan 2
Yizhu Jiao
Xiangguo Sun Hong Cheng'
Abstract
1. Introduction
Node classification a fundamental task graph analysis, with a wide range of applications such as item tagging (Mao et al., 2020), user profiling (Yan et al., 2021), and financial fraud detection (Zhang et al., 2022a). Developing effective algorithms for node classification is crucial, they can significantly impact commercial success. For instance, US banks lost 6 billion USD to fraudsters in 2016. Therefore, even a marginal improvement in fraud detection accuracy could result in substantial financial savings.
Node classification is fundamental task in graph analysis, with broad applications across various fields. Recent breakthroughs in Large Language Models (LLMs) have enabled LLM-based ap- proaches for this task. Although many stud- ies demonstrate the impressive performance of LLM- methods, the lack of clear design guidelines may hinder their practical application. In this work, we aim to establish such guide- lines through fair and systematic comparison of these algorithms. As a first step, we developed Bed,, a comprehensive codebase and testbed for node classification using LLMs. It includes 10 homophilic datasets, 4 heterophilic datasets, 8 LLM-based algorithms, classic base- lines, and 3 learning paradigms. Subsequently, we conducted extensive experiments, training and evaluating over 2,700 models, to determine the key settings (c.g., learning paradigms and ho- mophily) and components (e.g., model size and prompt) that affect performance. Our findings uncover 8 insights, e.g., (1) LLM-based methods can significantly outperform traditional methods in semi-supervised setting, while the advantage is marginal in a supervised setting; (2) Graph Foundation Models can beat open-source LLMs but still fall short of strong LLMs like GPT-4o in a zero-shot setting. We hope that the release of Led,odeBed, along with our insights, will facil- itate reproducible research and inspire future stud- ies in this field. Codes and datasets are released github.... io/.
Given its practical importance, node classification has been long-stane standdign research focus in both academia and industry. The earliest attempts to address this task adopted techniques such as Laplacian regularization (Belkin et al., 2006), graph embeddings (Yang et al., 2016), and label propagation (Zhu et al., 2003). Over the past decade, GNN-based methods have been developed and have quickly become prominent due their superior performance, demonstrated by works such Kipf & Welling (2017). Veličković et al. (2018), and Hamilton et al. (2017). Additionally, the incorporation of encoded textual information has been shown to further com- plement GNNs' node features, enhancing their effectiveness (Jin et al., 2023b; Zhao et al., 2023a).
Inspired by the recent success of LLMs, there has been a surge of interest in leveraging LLMS for node classifica- tion (Li al., 2023b). LLMs, pre-trained extensive text corpora, possess context-aware knowledge and superior se- mantic comprehension, overcoming the limitations of the non-contextualized shallow embeddings used by traditional GNNs. Typically, supervised methods fall into three cate- gories: Encoder, Explainer, and Predictor. the Encoder paradigm, L.LMs employ their vast parameters to encode nodes' textual information, producing more expressive fea- tures that surpass shallow embeddings (Zhu et al., 2024), The Explainer approach utilizes LLMs' generative capabil- ity to enhance node attributes and the task descriptions with more detailed text (Chen et al., 2024b; He et al., 2024). This generated text augments the nodes' original informa- tion, thereby enriching their attributes. Lastly, the Predictor role involves LLMs integrating graph context through graph encoders, enabling direct text-based predictions (Chen et al., 2024a; Tang et al., 2023; Chai et al., 2023; Huang et al., 2024). For zero-shot learning with LLMs, methods can
at htttt::///lnnooeeeee
'Department Systems Engincering and Engineering Manage- ment, and Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong "Microsoft Research Asia Universityy of Illinois Urbana-Champaign. Correspondence to: com>, Hong Cheng
Yifei Shen www.ffessheiiimcm <hchenge so. cuhk.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
be categorized into two types: Direct Inference and Graph Foundation Models (GFMs). Direct Inference involves guid- ing LLMs to directly perform classification tasks via crafted prompts (Huang et al., 2023). In contrast, GFMs entail pre-training on extensive graph corpora before applying the model to target graphs, thereby equipping the model with specialized graph intelligence (Li et al., 2024b). An illustration of these methods is shown in Figure
Datasets
Academic
Wiebi mk
Social +
F-Commerce d
HewemphitciC
Common Modules
Backhomes
Utils
GNNa GDN, SAGE: GAT, LMs SweBElt T--01, RoMti
Datn
Narmalosde)
Groph codier
Esgelanpuliion
Owen Nistral LLaWA
Supported Methods
Semi-superissd Supervised
LLM-as- L---ue-
LLM Ex-plaaaee
LLM-as Prediccor
EN
TAPE
Instrction Turinn
Graph Tssuaco
GNNLusere
LLaGAruua
Despite tremendous efforts and promising results, the design principles for LLM-based node classification algorithms remain elusive. Given the significant training and infer- ence costs associated with LLMs, practitioners may opt to deploy these algorithms only when they provide substan- tial performance enhancements compared to costs. This study, therefore, seeks to identify (1) the most suitable settings for each algorithm category, and (2) the sce- marios where LLMs surpass traditional LMs such as BERT. While recent work like GL (Li et al., 2024c) has evaluated various methods using consistent data splits in supe-supe and zero-shot settings, differences in backbone architectures and implementation codebases still hinder fair comparisons and rigorous conclusions. To ad- dress these limitations, introduce new benchmark that further standardizes backbones and codebases. Addition- ally, we extend GLBench by incorporating 3 E-Commerce datasets and 4 heterophilic datasets, while also expanding the evaluation settings. Specifically, we assess the impact of supervision signals (e.g., supervised, semi-supervised), different language model backbones (e.g., RoBERTa, Mis- tral, LLaMA, Qwen, GPT-4o), and various prompt types (e.g., COT, ToT, ReAct). These enhancements enable more detailed and reliable analysis of LLM-based node classifica- tion methods. In summary, our contributions to the field of LLMs for graph analysis are as follows:
Zero-shot
Direct Inference with Prompt Trmplates @ 99-94
T Graph Foundation Model
Direct
CoT
ToT
ReACT
w. Neighbor
ZeroB
Five-tuned LLMe
Evaluation
Accurary
Macro F1-Score
Hallacinadioe
Figure I: Overview of LLMNodeBed.
best and the key components for achieving this per- formance. Our work provides intuitive explanations, practical tips, and insights about the strengths and lim- itations of each algorithm category.
2. Preliminaries on Node Classification
To leverage the language abilities of LLMs, we study the node classification task within the context of text-attributed graphs (TAGs) (Ma & Tang, 2021). TAGs are represented as g = (V) where V represents the set of nodes, & the set of edges, and the collection of textual descriptions associated with each node v E V. Some of the nodes are associated with labels, represented as Vi C V. The remain- ing nodes do not have labels, and are denoted as Vu. The goal of node classification is to train a neural network based on the graph G and the labels of Vi, which can predict the labels of unlabeled nodes in V..
Traditionally, the textual attributes of nodes can be encoded into shallow embeddings as X = [a....... 2 mpy TR. Vixd using naive methods like bag-of-worde or TF-IDF (Salton & Buckley, 1988), where d represents the dimensionality of the embeddings. Such transformation is adopted in most GNN papers. Instead, the input for LLM-based approaches is the raw text and one may expect that the pre-trained knowledge in LLMs can improve performance.
A Testbed:
We release LLMNodeBed, a PyG-based testbed designed to facilitate reproducible and rigor- ous research in LLM-based node classification algo- rithms. The initial release includes 14 datasets, 8 LLM-based algorithms, and learning configurations. I.LMNodeBed allows for easy addition of new algo- rithms or datasets, and a single command to run all experiments, and to automatically generate all tables included in this work.
3. LLMNNoeBBd: A Testbed for LLM-based Node Classification
Comprehensive Experiments: By training and evalu- ating over 2,700 models, we analyzed how the learning paradigm, homophily, language model type and size, and prompt design impact the performance of each algorithm category.
In this section, we present the datasets, baselines, and learn ing paradigms within LLMNodeBed (Figure 1).
3.1. Datasets
Insights and Tips: Detailed experiments were con- ducted to analyze each influencing factor. We identified the settings where each algorithm category performs
To provide guidelines for applying algorithms across di- verse real-world applications, the selection of datasets in
2

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Table I: Statistics of supported datasets in LLMNode
Academic Puhmed
Web WikiCS
Social
E-Commerce Photo
Heterophilic
Statisties
Cora
Citeseer
arXiv
Instagram
Reddit
Books
Computer
Corelll
Texas
Wisconsin
Washington
Classes
40
12
10
5
# Nodes
2,708
3,186
19.717
169.343
11,701
11,339
33,434
41.551
48.362
87,229
191
187
265
229
Edges
5,429
4,277
44,338
1,166,243
215,363
144,010
198,448
358,574
500,928
721.081
292
310
510
394
Avg. Taken
1834
210.0
446.5
239,8
629.9
56.2
197.3
337.0
201.5
123.1
594.6
453.2
639,1
469.0
Homophily (%6
82.52
72.93
79.24
63.53
68.67
63.35
55.52
78.05
78.50
85.28
11.55
6.09
16.27
17.07
Advanced Prompts
L
LLM
L GNN
Label
"Let's diink saep by steps
LLM Encoder
LLM
Label
Enriched Prompts
"Neightorr Jicass
Direct Inference
4 LM
LLM
Label
4 GNN
LLM as Explainer
5
GFM
Optional
Graph Foundation
Source dalasets
3
& GNN
Model
Label
Task Instraction
Target dataset
GFM
Label
LLM Predictor
(a) Semi- / Supervised Settings
(b) Zero-shot Setting
Figure 2: Illustrations of LLM-based node classification algorithms under supervised and zero-shot settings.
LLMNodeBed considers several key factors: (1) Multi- domain Diversity to reflect different contexts, (2) Varying Scales to examine algorithm scalability and the associated costs of leveraging LLMs, and (3) Diverse Levels of Ho- mophily to understand its impact on performance. There- fore, LLd comprises 14 datasets spanning various domains. These datasets vary significantly in scale, ranging from thousands of nodes to millions of edges, and exhibit differing levels of homophily. Such diversity in domain, scale, and homophily enables the assessment of algorithms across a wide range of contexts.
criteria: (1) Diverse Roles of LLMss thoroughly evaluate their effectiveness, (2) Straightforwar Design to facili- tate clear comparisons by avoiding complex and intertwined architectures, and (3) Representativeness to ensure bench- mark relevance by including widely recognized methods. Therefore, the LLM-based baselines include:
We include ENGINE (Zhu et al., 2024) and introduce GNN ENGINE aggregates hidden embeddings from each LLM layer to create comprehensive node representations. In contrast, GNNLLMEmb initializes node embeddings using the LLM's last hidden layer before feeding them into GNNs for classification.
For datasets where raw text has been preprocessed into vec- tor embeddings using bag-of-words or TF-IDF techniques, we utilize collected versions including Cora and Pubmed (He al., 2024), Citeseer (Chen et al., 2024b), and Wi- kiCS (Liu et al., 2024). The remaining datasets already include text attributes in their official releases, including arXiv (Hu et al., 2020), Instagram and Reddit (Huang et al., 2024), Books, Computer, and Photo (Yan et al., 2023). Cor- nell, Texas, Wisconsin, and Washington (Wang et al., 2025). Detailed statisties and information for these datasets are provided in Table and Table 10 in the Appendix.
W..M-as-Explaineee We select TAPE (He et al., 2024), a representative LLM-as-Explainer method. TAPE prompts the LLM to think over nodes by generating predictions along with explanations, thereby enriching the node's 's text attributes and enhancing classification performance.
Both Encoder and Explainer methods require processing the entire dataset, either by encoding each node's text or gener- ating explanations for nodes, which introduces additional processing time before the actual model training begins.
LLM-as-Prediche We select GraphGPT (Tang al., 2023). LLaGA (Chen et al., 2024a), and implement LLM Instruction Tuning (LLMMr).. GraphGPT employs a multi- stage pre-training and instruction tuning process to clas- sify nodes based on both text and graph context. LLaGA
3.2. Baselines
The initial release of LLMMNodeBed includes eight LLM- based baseline algorithms alongside classic methods. We
selected these LLM-based algorithms based three key
3

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
integrates tokenized task instructions and graph context into LLMMs generate predictions. We implement LLM to evaluate whether LLMs alone can function as effec- tive predictors. This involves fine-tuning the LLM us- ing task prompts and ground-truth labels formatted as (Question, Answer) pairs.
Supervised Learning: A larger subset of nodes Vi with known labels is provided, assessing the model's s performance with abundant supervision. Specifically, we use a 60% training, 20% validation, and 20% testing split for most datasets. This consistent split facilitates fair comparisons across baselines. Detailed data splits are provided in Table 10 in the Appendix.
This category refers to LLMs generating prediction labels directly from a node's text with- out additional training or labels. We employ two types of prompt templates: (1) Advanced Prompts that improve LLMs' reasoning abilities such as Dhain-of-Thoughh (CoT) (Wei et al., 2022) and Tree-of-Though (ToT) (Yao et al., 2023a), and (2) Enriched Prompts that incorporate neigh- boring node information to provide structural context.
LLM Direct Inference:
No labeled data is provided for training. The model predicts labels solely based on node textual descriptions and the graph structure, as- sessing its ability to generalize to new, unseen data. For test samples, we follow existing literature (Zhu et al., 2025) by selecting one smaller dataset from each domain and using 20% of its nodes as test samples.
Zero-shot Learning:
GFMs are foundation models trained on large- scale source graph datasets to acquire general classification knowledge, which can then be seamlessly applied to target graphs. We include ZeroG (Li et al., 2024b) as a repre- sentative GFM due to its superior performance in zero-shot settings. Additionally, LLM-as-Preedee Predictor methods trained with extensive graph corpora are also considered within this category for zero-shot applications.
GFMs:
4. Comparisons among Algorithm Categories
In this section, we present empirical results among algo- rithm categories, along with key insights derived from them.
4.1. Semi-supervised and Supervised Performance
Settings: To ensure a fair comparison of baseline algo- rithms, all methods are implemented with consistent compo- nents: GCN (Kipf& & Welling, 2017) for GNNs, RoBERTa- 355M (Liu et al., 2019) for LMs, and Mistral-7B (Jiang et al., 2023) for LLM components where applicable. This uniformity guarantees that performance differences are at- tributable to model designs rather than underlying architec- tures. Each experiment was conducted over 4 runs. Based on the results of Accuracy (Table 2) and Macro-Fl (Table 12 in Appendix), we summarize the following takeaways:
Besides LLM-based methods, LLMNodeBed also integrates classic algorithms, including MLPs, GNNs, and LMs. MLPs generate predicted label matrices from node em- beddings, while GNNs combine shallow embeddings with graph structures for label prediction. LMs process a node's s text through hidden layers and use classification head to produce label distributions. Furthermore, include GLEM (Zhao et al., 2023a), hybrid LM+GNN method that combines graph and text modalities. Further discussions of existing algorithms are provided in Appendix A.
Takeaway 1: Appropriately incorporating LLMs con- sistently improves the performance. According to the table, the best performance is often achieved by LL.M-based methods compared to classic methods. It suggests that us- ing I.LM to exploit the textual information is useful. This conclusion extends to large-scale graphs as well, c.g., ogbn- product dataset with millions of nodes. Detailed results and analysis for this dataset are provided in Appendix D.
Prompt templates for LLM-as-Predietto and Direct Infer- ence are listed in Appendix 1.1 and B.2, respectively. Ap- pendix C.2 describes the implementation details, backbone selections, and hyperparameter search spaces for all al- gorithms. Additionally, we highlight the distinctions of L.dMNodeBed in Appendix C.3.
3.3. Learning Paradigms
Takeaway 2: LLM-based methods provide greater im-
provements in semi-supervised settings than in super- vised settings. By comparing the tables, we observe that performance gains are more significant in semi-super scenarios. From an infformation-theoree node classification task with cross cross-entropy loss aims to max- imize the mutual information between the graph and the provided labels, denoted as 1()) If we consider graph G as a joint distribution of node attributes X and structure E, we have:
We evaluate the baselines under three learning configura- tions: Semi-supervised Supervised, and Zero-shot. These configurations are defined as follows:
perspective, the
Semi-supervised Learning: A small subset of nodes Vi C V with known labels 3/ is provided. This setting assesses the model's ability to effectively utilize lim- ited labeled data, reflecting real-world scenarios where labeling is scarce. For experimental datasets, we adopt the official splits designed for semi- supervised settings to ensure standardized evaluation.
I(G: v.) = (;0)) = ((8)))) + (()))N)). (1)
The first term represents the information encoded in the
4

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Table 2: Performance comparison under semi-supervised and supervised settings with Accuracy (%) reported.
The best and second-best results are highlighted. Results of Macro-F1 are shown in Table 12 in the Appendix. LLMr on the arXiv dataset requires 30+ hours per run, preventing multiple executions.
Semi-supervised
Cora
Citeseer
Pubmed
WikiCS
Instagram
Reddit
Books
Photo
Computer
Avg.
GCNsstentre
82.30
=0.55
78. 28294
20.6
20.5
61.44 61.4420.3
29:
69.25 69.25
71.44-119
71.79
SAGEasdeetr
20.39
69.56_0us
77.88
79.67
63. 63.57
56.6 26.65
78.50 78.50 a2ss
81,43=022
73.50
GATaaaatii
81 20.30
69. 89.94_074
78.49
29.9
63. 63.564004
60.60=1.7
74.35=035
80.40
83. 83.39_022
74.67
Classic
SenBERT-66M
66.66 66.661142
60.5221.22
36.044292
77.77
59 21.0021.7
56.05
83. 83.682019
73.88 73.89
70. 70.76_..15
64.93
RoBERTa-355M
72. 72.24:2114
66. 66.68.22 2828
42.32
76.81 21:04
63. 43.52.044
59.27.004
84, 94.62:0.6
74.79 74.79
72.31 12.31
68,06
GLEM
11.3
68.80_2.6
81.70...97
76.43 po.ss
60.25 13:6
35.13.1441
83.28
76.93
80.46_1
73.81
Encoder
GCNuLMas
83.33
71.39
78.71
80.94
67.49:...3
68.65
83.03
84, 24.84
88 22.112
78.51
ENGINE
84.22
72.14_074
77.84
80.94
67. 17.44
69.67
69.67:0016
82.89
84.3346.57
86, 86.42:022
78,40
Explainer
TAPE
84.04
71.87-035
78.
81.9
66.07
62.43
84.92.n36
86.46
89.52
78.43
LL
17.00
54.26
80,99
75.02
41
$4.09_1mp
80.92
71.28
66. 26.99..
65.76
Predictor
GraphGPT aLaG
64.72.11.0
64.58g1.55
70.34
70.71
62. 12188
58.25
g1.13
77.48 77.48
80. 10
70.02
78.94
62.61 136s
65.91 1209
76.47
15784
10.10.0..
g344
14.441000
75.07
Supervised
Cora
Citeseer
Pubmed
arXiv
WikiCS
Instagram
Reddit
Books
Photo
Computer
Avg.
87,41
75,74
89.0
71.39
83.67
63.942061
65.07 65.07
76 26.94
73.34
77. 77.16
76.37
SAGEaunwtre
87.44
74.96
90,474925
18
84,862091
64.14
61,52
79. 49
84.59
87.774034
78.64
Classic
88,25
71.57
83,94
64.93
64.16g1m is
14.84
88.32
78.70
SenBERT-66M
79.61 20.00
74.06
94,472035
72.664924
86.51
58. 58.70ga.5
85, 85.99
77.724035
74.22
76.40
RoBFFRTa-355M
83.17
75.90
94.84
74.1220.1
87.47
63.75
60.61 2014
86.68
79 79.454037
75, 75.76440.0
78. 18.17
GLEM
86.81 23.19
73.24
93.98
73.55
79.81 gu48
67.39
53.11
83. 33.98
78. 78.16
81 81.63
77.17
MCN
2179
16
88.38
34
84.78
68.27
20.65
84.23
26.07
89 52
81.09
Encoder
ENGINE
87.00=210
75.82+1.5
90.0840
34.694036
85,442053
68, 68,8740
71.21
84.0940..0
86 26.98.
89 89.052001
81.32
Explainer
TAPE
88.05
76.45
93.00
74.96,014
87.10
68.11
66.22
85.95
87.72
90.46
81.80
LLM
=1,93
60.97
94,16ga
76.08
80.6
44.20
58,30
84,80
78.27au34
14.51
72.38
Predictor
GrapbOPT
12.29.oa6
74.67
13.54
75.15
82,54
67.00
60.72
85.38
84.46
86.78
79.25
LLaGA
87.55
76.73
90.28
34.49
84.03gr 10
69.16
21.06
85.56
87.62ga30 30
90.41 13
81.69
graph structure, utilized by classic GNNs, while the second term represents information from node features, leveraged by LLMs. In semi- sunirvuiee settings, the mutual informa- tion between structure and labels is relatively low, allowing LLMs to contribute more significantly to performance.
ios, Predictrr methods enhance performance across most datasets. Especially, the LLaGA method achieves su- perior results among 5 of 10 datasets. Conversely, in semi- supervised settings, HLLM-as--rPdicce methods exhibit un- stable performance, evidenced by low Macro-Fl scores, and imbalanced output distributions (detailed discussion in Ap- pendix F.3). These findings indicate that methods most effective when ample supervision is avail- able, with LLaGA being especially strong choice. Fur- thermore, within the predictor methods, LLM Instruction Tuning typically falls behind the other two methods and in- curs substantial time costs (Table 19 in the Appendix). This shows that standalone LLMs are weak predictors and incor- porating graph context is essential for achieving satisfactory performance.
Takeaway 3: Lxx-aieE methods are highly
effective when labels heavily depend on text. TAPE achieves top runner-up performance on academic and web link datasets like Cora and WikiCS, where structural information is less relevant to labels (Zhang et al., 2022b). However, TAPE struggles with social networks that require deeper structural understanding, such as predicting popular users (high-degree nodes) on Reddit.
Takeaway 4: L.MMdds-nnooder methods balance compu- fational cost and accuracy effectively. LLM-as-Encoder methods perform satisfactorily across all datasets. Further experiments in Section 5.1 reveal that LLM-as-Encon methods are more effective than their LM counter- parts when graphs are less informative. Regarding cost- effectiveness, -xxpaannxe should generate long ex- planatory text, which is far more time-consuming than en- coding texts in LLLM-ss-Encolee (see Appendix G). There- fore,
4.2. Zero-shot Performance
Settings: For LLM Direct Inference, we utilize both closed- source and open-source LLMss, including GPT-4o (OpenAI, 2024), DeepSeek-V3 (DeepSeek-Al, 2024), LLMMA3.1-8B (Dubey et al., 2024), and Mistral-7B. The prompt templates include Direct, CoT, ToT, and ReAct (Yao et al., 2023b). Additionally, we incorporate a node's 's neighboring informa- tion into extended prompts, referred to as "w. Neighbor", and have the LLMs first reason over neighbors to generate a summary that facilitates the subsequent classification task, referred to as "w. Summary". Prompt templates are listed
methods strike balance between computational efficiency and accuracy.
Takeaway 5: L.LM-ssPredictto methods are more effec- tive when labeled data is abundant. In supervised scenar-
5

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Table 3: Performance comparison under zero-shot setting with Accuracy (%) and Macro-F1 (%) reported. Numbers in brackets represent the dataset's homophily ratio (%). Results of other LLMs are shown in Table 13 in Appendix.
Cora (82.52)
WikiCS (68.67)
Instagram (63,35)
Photo (78.50)
Avg.
Type & LLM
Method
Acc
Macro-F1
Acc
Macro-F1
Acc
Macro-F1
Acc
Macro-F1
Acc
Macro-Fl
Direct
68.08
69.25
68.59
63.21
44.53
42.77
63.99
61.09
61.30
59.08
CoT
68.89
69.86
70.75
66.23
47.87
47.57
61.61
60.62
62.28
61.07
LLM GPT-40
ToT
68. 68.29
69.13
70.78
65.69
44.16
42.68
60.84
59.16
61.02
59.16
ReAct
68.21
69.28
69.45
66.03
44.49
43.16
63.63
60.82
61.44
59.82
Neighbor
70.30
71.44
99.69
64,51
42.42
39,79
69.93
68.55
63.09
61.07
Summary
71.40
72.13
70.90
65.42
45.02
44.62
72.63
70.84
64.99
63,25
Direct
62.64
63.02
$6.77
$3.04
37.58
29.70
41.23
44,26
49.56
47.50
CoT
62.04
62.61
58.88
56.00
42.00
39.06
44,22
47.13
$1.78
$1,20
LLM
ToT
34,06
33,30
40.35
41.15
45.33
45.27
31.31
34.00
37.76
38.43
LLaMA-8B
ReAct
36.55
38.04
22.40
25.76
44.67
44.42
27.03
28.96
32.66
34.30
w. Neighbor
64.55
64,41
59.43
54.16
36.98
28.32
45.49
50.44
51.61
49.33
W. Summary
64.69
64.62
62.69
56,40
37.59
30,91
48.11
$2.20
53.27
51.03
ZeroG
62.55
$7.56
62.71
$7.87
50.71
50.43
46.27
$1.52
$3.56
$4.35
GFM
LLMr
52.58
51.89
60.83
53,59
41,58
26.26
49.23
44.85
51.06
44.15
LLaGA
18.82
8.49
8,20
8.29
47.93
47.70
39.18
4.71
28.53
17.30
in Appendix B.2. Besides, we assess the transferability of GFMs by evaluating ZeroG (Li et al., 2024b), LLM Instruction Tuning, and LLaGA. GFMs are applied fol- lowing the aotra-domain manner: each model is pre-trained on a larger dataset within the same domain (e.g., arXiv from the academic domain) before being evaluated on the target dataset. Results for Accuracy and Macro-FI are shown in Table 3 and Table 13 in the Appendix, where we have the following takeaways:
Summary", demonstrate performance enhancements across LLMs. The performance gains are particularly evident on homophilic datasets such as Cora and Photo (3%- 10611 where neighboring nodes are likely to share the same labels as the central node. High homophily means that informa- tion from neighboring nodes provides crucial clues about a central node's label, thereby improving classification perfor- mance. Among these enriched prompts, "w. Summary" is especially effective as it not only provides structural context but also leverages the self-reflection abilities of LLMs to further utilize structural information.
Takeaway 6: GFMs can outperform open-source LLMS but still fall short of strong LLMs like GPT-40. ZeroG outperforms LLaMA-SB in most cases, achieving up 6% average improvement in accuracy. However, it still falls short of GPT-4o and DeepSeek-V3. Among GFMs, LLaGA performs poorly because it uses a projector to align the source graph's 's tokens with LLM input tokens. This projec- tor may be dataset-specific, leading to reduced performance on different datasets, as also observed in Zhu et al. (2025). These findings highlight the need for further research to im- prove the generalization of GFMs to match the performance of more powerful LLMs.
4.3. Computational Cost Analysis
Time Cost: various methods in supervised settings. Detailed training times are provided in Table 19 in the Appendix, while inference times are presented in Table 20. All measure- ments were conducted on single NVIDIA H100-80G GPU to ensure consistency.
We evaluate the training and inference times of
Based on the results, we can conclude that classic methods are highly efficient, with GNNs typically converging within seconds (c.g.. 5.2 seconds for GCN on Pubmed) and LMs fine-tuning completed within minutes. In contrast, LLM-as L.MMaa-Explainer approaches are the most time- consuminn (e.g., 5.9 hours for TAPE on Pubmed) because they re- quire generating explanatory text for each node and sub- sequently processing this augmented text through both an I.M and GNN. This three-stage computational process significantly extends the overall computation time. LLM-as- Encoder methods are the most efficient among LLM-based approaches (e.g., 13.4 minutes for GCNL Pubmed), utilizing LLMs solely for feature encoding, which allows GNN training to remain efficient and complete within min- utes. Although LLM Predictor methods are more effi-
Takeaway 7: LLM direct inference can be improved by appropriately incorporating struetural information. Our results reveal that advanced prompt templates such as CoT, ToT, and ReAct, offer only minor performance improvements. Specifically, models like LLaMA exhibit limited instruction-followiin abilities, often producing unex- pected and over-length outputs when encountering complex prompts such as ReAct. This makes parsing classification results challenging and leads to suboptimal performance. The advanced prompts are generally designed for broad reasoning tasks and lack graph- or ciaifficationsspeccin knowledge, thereby limiting their benefits for the node clas- sification task. In contrast, enriched prompts that incor- porate structural information, i.e., "w. Neighbor" and "w.
LL-M-as-rredictor
6

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Table 4: Comparison of LLM- and AM-as-Encoded with Accuracy (%) reported under semi-supervised setting.
The best encoder within each method on a dataset is highlighted. Results in supervised settings are shown in Table 14 in Appendix.
Method
Encoder
Computer
Cora
Pubmed
Photo
Books
Citeseer
WikiCS
Instagram
Reddit
Homophily Ratio (%)
85.28
82.52
79.24
78,50
78.05
72.93
68.67
63.35
55.52
SenBERT
69,57 10.18
64.6
74.67
72.2810.6
81.93
66.83
71.48
64,981038
1052
RoBERTa
69.42..010
73.84
73.21 10.78
12.9510.34
81.71 10.14
70.59
75.8210.1
66.391004
59.66105s
MLP
Qwen-3B
27
7..0
70.30
72.72+ma28
g1660
68. 68.26g20.7
78.64+0.77
66.53+a037
60.4920017
Mistral-7B
69.37 .0228
23.90
1..00
74.16
81.91 1025
69.66_0.34
79.56
66.68 1024
61.911021
SenBERT
88.92::0119
81,76
78. 2.006
88.18
83,47
70.97
80.41
65.7840.14
64.97
RoBERTa
88. 88.90
84,56 1.44
78.08
15.19
73.52 83.22.0099
80,97
66.641021
65.691101
GCN
Qwen-3B
17
83.62200.4
28.50
84,264003
22.83
71.50=0.02
81.02.0.33
66.69
69.40
Mistral-7B
18
83,33
78.71=0.45
84.84+047
83.0
71.3920.0
80,94
67.49
68,65
SenBERT
89.08
20.45
77.2920045
85.5440.16
83,93
69.4222142
80.02.aa24
65.34.10.4
61.652017
RoBERTa
88,9 =9.97
84.06.20.52
75. 75.82.20.9
85.57
83.74.022
72.58
80.77.1029
66.53
63,65
SAGE
Qwen-3B
86.24_.022
83.31 33.31.000 10.63
76.76.20.5
84.2810.2
82.84
71,11
80.85
66.73
63.82.1038
Mistral-7B
88.48g0.20
82.73
77.64=1.73
85,50
83,32
71.42:0047
81.47
67. 67.444006
65.02
cient than Explainer approaches, they still require hours for
size) influence the effectiveness of LLM-as-Explaing and .LM--s--redictto
effective model training. Among predictor methods, LLaGA is the most efficient (e.g., 25.6 minutes on Pubmed) as it encodes both the node's textual and structural information into embeddings instead of processing raw text.
5.1. LLM--a-Ennodern Compared with LMs
Motivation and Settings: Both LLMs and small-scale LMs can encode nodes' associated texts. This raises the question: When do LLMs surpass LMs as encoders? To address this, we evaluate various methods using node features derived from LLMs and LMs, observing the resulting performance differences. For LMs, we select SenBERT-66M. (Reimers & Gurevych, 2019) and RoBERTa-355M (Liu et al., 2019). For LLMs, we choose Qwen2.5-3B (Qwen al., 2025) and Mistral-7B (Jiang et al., 2023). The evaluated datasets include both homophilic and heterophilic graphs to ensure generality. For heterophilic graphs, we use the released datasets collected by Wang et al. (2025)' with processing details explained in Appendix C.I. The considered meth- ods include: (1) MLP, which solely utilizes node features as input predict labels without incorporating any graph information, (2) GCN, (3) GraphSAGE, and (4) H&GCN (Zhu et al., 2020). a representative GNN explicitly designed for heterophilic settings, included specifically to accom- modate heterophilic graphs. For each method, we initialize node features using embeddings derived from various LM or LLM backbones, ensuring that all other components remain consistent across evaluations.
During inference, significant efficiency gap remains be- tween LLM--based and classic methods. Classic methods can complete the entire inference process for thousands of cases within milliseconds, making them suitable for in- dustrial deployments that demand real-time responses. In contrast, LLM--baed methods are limited to processing one case within the same timeframe, highlighting the urgent need improve their efficiency.
We also analyze the memory costs of differ- ent methods during both the training and inference stages to assess their resource usage. We select three datasets, Cora, WikiCS, and arXiv, spanning a range from 2,708 to 169,343 nodes. The results, presented in Table 21 in the Appendix, were measured using single H100-80GB GPU to ensure consistency and comparability. The results show that LLLM-as-Encoder methods demonstrate memory costs comparable to traditional GNNs, making them practical choices for deployment. In contrast, methods involving methods are sig- fine-tuning an LM or intensive, largely due to the high nificantly more memory memory-intensive, parameterization of language models. These findings under- score the importance of addressing efficiency challenges to enable the practical deployment of L.L--based methods.
Memory Cost:
From the results shown in Table 4 (homophilic graphs) and Table 5 (heterophilic graphs), we can observe that:
Takeaway 8: LLM-as-Encoon significantly outperforms LMs when graph structure less informative about the labels, e.g., heterophilic ones.
5. Fine-grained Analysis Within Each Category
In this section, we present empirical results within each category. For LLM-as-Encoded under which LLMs outperform traditional LMs. Addition- ally, we examine how key components (e.g., model type and
For homophilic graphs, the performance gap between LLMs
explore the conditions
httss//uuggnngffccoooolattsssr/e Attributed AttributedGrraps
7

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Table 5: Comparison of LLM- and LM-as-Encoder with Accuracy (%) reported on heterophilic graphs. The best encoder within each method on a dataset is highlighted.
Semi-supervised
Supervised
Method
Encoder
Cornell
Texas
Wisconsin
Washington
Cornell
Texas
Wisconsin
Washington
Homophily Ratio (%)
11.55
6.69
16.27
17.07
11.55
6.69
16.27
17.07
SenBERT
50.59 13.14
56.67
71.98
63.261289
16.15
76.32.3.72
17.0
70.44.8.65
RoBERTa
59.08 12:57
17247
73.87 11.82
65.43
66.67 16.671..88 18.88
74.21 16009
10.0019.88
76, 76.96.17.48
MLP
Qwen-3B
57.78
76.27
82.36
75.11 1.92
L4.76
88. 88,95 1.007
88.68 16.61
83.48
Mistral-7B
59.87=6.72
76.27=11.08
83,30
74.24
78.46-4.17
90.53
89.43
83,91
SenBERT
46. 42.33
54.93
58.30
52.61 11.35
50.77
59.47 15.16
61.13
61.30
RoBERTa
47.06
55,20
54.91
54.89
51.79
58.42
59.24
61.31
GCN
Qwen-3B
53.5942..7
56.80
63.02
64,56
58.46
64.74
65.28 16.82
67.83
Mistral-7B
5..64
58.67 33,00
62.08
61.52
59.49 1..9
65.79 1.66
64.90
66.96
SenBERT
12.55
61.73
70.47
65.54
68. 08.772+497
80,00
83.02+66.1
76.9644488
RoBERTa
55.55
64.26
73.59
66.08
70.26
80.53
81.89
74.35
SAGE
Qwen-3B
57.13
7..5
83.21
72. 23.66
74.87
89.47=1677
91.32
83.48
Mistral-7B
16.86
76.53.224
83,96
73.91 =3.91
77.44
91.05
89.4444424
81.74444.48
SenBERT
56.34
66.67
73.40+1166
70.55
73.85
84.21
86.42.t6.m
77.83
RoBERTa
60.00
68. 13
75.6642.12
71.52
74.87
83.16
84.53
79. 13.45.4..
H₂GCN
Qwen-3B
61.57
10.13
84.53
74.67
76.41
92.11
89.81
85 85.22+1.99
Mistral-7B
59.2224454
72.93g8.1
81.89
68.59+4.46
75.89
89.47=3.2
89,43
86.09
Table 6: Performance (%) of TAPE with different LLM backbones under semi-supervised setting.
Metrics
LLM
Cora
Citeseer
Pubmed
WikiCS
Instagram
Reddit
Books
Photo
Computer
Avg.
Mistral GPT-4o
84.04
10.87
78.61 -1.21
81.941016
66.07
62.43
84.92.0.0
86.461012
89.52
78.43
Ace
84.30.036
73.75
82.70
81.93
66.25 1035
62,22
85.08
2.17
89.62
79.17
Mistral
81.89 21.89
66. 20.30
78.46g1.13
80.03
20.01
61.23
47. $7.124330
82.3 19
84.90
70.31
F1
GFT-4o
82.62
67.41 10.82
82.45
80.27
51.16
61.11
47.51
82.54_a18
84.28
71.04
Table 7: Performance (%) of TAPE with different LLM backbones under supervised setting.
Metrics
LLM
Cora
Citeseer
Pubmed
arXiv
WikiCS
Instagram
Reddit
Books
Photo
Computer
Avg.
Mistral
88.05
76.45
93.00
74.96gu1
87.11
68.11
66.22
85 85.95
87.72.m2x
90.46
81.80
Acc
GPT-44
88.24_121
76.41
94.124a05
75.
87.10
67. 2151
66.33
86.19_am
87.65
90.56.4a1
81.97
Mistral
23.3
92. 2
57.79
86.03-1.4
58.31
65.91 2001
54.07=201
83.41
86. 36.78
74.52
F1
GPT-4o
87.34_LM Lm6
73.17
93.58
57.69
25,93
57.49
66.09:4400
54.32
83,40
$6.91
74.59
and LMs is relatively small and becomes noticeable only in graphs with lower homophily, such as WikiCS and Reddit. For instance, on Reddit, LLM-based encoders achieve 4% higher accuracy than their LM counterparts. The perfor- mance gap is substantially larger for heterophilic graphs, with differences reaching up 10% across methods in both semi-supe sumivisueeei and supervised settings. For example, on the Texas dataset, LLM-based encoders achieve a 12% im- provement over their LM counterparts for the GCN, SAGE, and HCGCN methods in the semi-supervised setting. We can also explain these results using mutual information in Equation (1). In homophilic graphs, edges often connect nodes with the same labels. Consequently, the first term in (1) dominates, leaving limited room for improved encoders like LLMs to enhance performance. In contrast, this prop- erty does not hold for iessiinformativ heterophilic graphs, where neighboring nodes are less likely to share the same labels, making the choice of encoder significantly more
impactful.
5.2. LLMMas-Explainerr Impact of LLM Capabilities
Motivation and Settings: In the LLM-as LLM-as-Explainee Explainer paradigm, as the language model should generate explana- tory texts, the adopted language models should be auto- regressive and the model size should be large. To investigate how the advanced reasoning capabilities of LLMs influ- overall performance, we replace the default Mistral-7B model in the TAPE method with the more powerful GPT-4o model, keeping all other components unchanged.
Results and Analysis: As presented in Table 6 (semi- supervised settings) and Table 7 (supervised settings), the effectiveness of L.MMa--Expplinerr methods positively cor- relates with the strength of the underlying LLMs. semi- supervised settings, TAPE utilizing GPT-4o consistently outperforms its Mistral-7B counterpart, achieving perfor-
8

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Table 8: Accuracy (%) of LLaGA to different LLM backbones under supervised settings.
The best LLM backbone within each series and at similar scales is highlighted. Semi-supervised performance is shown in Table 15.
LLM
Cora
Citeseer
Pubmed
arXiv
WikiCS
Instagram
Reddit
Books
Photo
Computer
Avg.
Qwen-3B
24.91 19
24.83
71.821197
82.23
62.49
67.96
83.56_18
89,37
79.10
Same
series
Qwen-7B
15.33
70.75
90.53
71.60
82.572167
63,8642776
68.62.285
-4.23_a51
23,55
87.21
78.82
Qwen-14BB
87.25 1143
75.49.20 200
73.15 1a74
12.26
63.88 12.00
67.60 17.60
83.94 ann
84.83 1077
87.06 10:0
79.54
Qwen-32B
85 35.93
75.39
89.97
72.84gas7
83,49
64.33
68.47.um
84. 18.29
84,77
28.49.0..
79.79
Similur scales
Mistral-7R
87.55
76.73_1.70 1.70
18
74,49_nxx
84,03
69.161872
71.06.4.
85.56 a.30
87.62.8.3
90.41 ja.z
$1.69
Qwen-7B
85.33
70.75
90,53
20.47
82.57
63,86
68. 68.622.05
84.23_n51
83,55
87.21
78.71
LLaMA-8B
85.77
14.84
89.57
72.72.026
82.25
11.12
67.70
84.05 ...26
85.57 1941
89:42 10
79.30
Table 9: Macro-F1(% of LLaGA to different LLM backbones under supervised settings.
LLM
Cora
Citeseer
Pubmed
arXiv
WikiCs
Instagram
Reddit
Books
Photo
Computer
Avg.
Qwen-3B
77.92
66,5225.0
28.88_1043
51,30=0.3
78.8
50.93
65.77
19.87=1.3
77.51
80,77
67.83
Same-
series
Qwen-7B
82. 82.50g4412
64.03
=0.29=0.33
$1.97=0.3
77.35
56.50=1.s
68.55
46.21
75.76
78. 78.86=500
69.20
Qwen: 14B
85.64_1
69.92;3.55
89 20.6
53.32:0034
29.
27.58
67. 67.10
44.26
26.09
80.17
70.29
Qwen-32B
g2.85
68.11
52.52_o0s
77.31
57.28 jasr
68.22 tomr
11.2
19.51
77.15 1775
70.08
Sinilar Stales
Mistral-7B
84.97 2av7
72.59 11.70
58.08 g000
82.37 1178
57.96..24
62. 14.14
54.89.129
83.56 1..40
86,97 10.54
73.35
Qwen-7B
82, 82.5004412
64.0724486
90.29
45, 75777
77.35+426
26.50=11s
68.55
46.21
75.76
78. 78.86=500
69.09
LLaMA 8B
81.40
69.87
89. 23
g5.23
80.14
54.58 24
67. 40
51.65
78.87
85. $5.54_.0 59
71.40
mance gains of up to 4% on the Pubmed dataset. However, in supervised scenarios, the performance gap between GPT- 40 and Mistral-7B narrows. This reduction is attributed to the abundance of labeled data, which increases the mutual information (8;; Di) in (I). Consequently, the dependency on node attributes decreases, thereby diminishing the rela- tive advantages of more powerful LLMs.
performance generally improves with larger model sizes, However, beyond Qwen-7B and Qwen- 14B, the perfor- mance gains become marginal while training and inference times increase significantly. For instance, Qwen-32B takes over 200 milliseconds per sample for inference, which is five times longer than Qwen-7B. Therefore, Qwen-7B or Qwen-14B are recommended as practical choices balanc- ing performance and efficiency. (2) Model selection at similar scales: When comparing models of similar sizes (Tables 8 and 9, and Table 15 in the Appendix), Mistral-7B outperforms other LLMs of comparable scale. Its superior performance makes Mistral-7B the recommended backbone LLM for node classification tasks.
Based on these findings, we recommend that when abundant supervision is available, practitioners may opt for open- source LLMs instead of more powerful and costlier models for the LLLMas-Explaine method. This practice can achieve comparable performance without incurring additional costs.
5.3. ALLM-as-Predictor: Sensitivity to LLM Backbones
6. Conclusion
Motivation and Settings: For most LLM-as-Predicee methods, only open-source LLMs are compatible. Given the diverse choices and varying scales of these models, we aim to investigate the sensitivity of performance to differ- ent LLM backbones. This examination seeks to identify potential scaling laws and determine which L.MSs excel at the node classification task. Therefore, we choose the best predictor method LLaGA as the baseline, include models of different sizes within the same series, i.e., Qwen2.5-series (Qwen et al., 2025). Additionally, we consider similar- scaled models to identify the most suitable for this task, including Qwen2.5-7B, Mistral-7B, and LLaMA3.1-8B All experiments maintain consistency by only varying the back- bone LLMs while keeping other components, training con- figurations, and hyperparameters unchanged.
This paper provides guidelines for leveraging LLMs to en- hance node classification tasks across diverse real-world applications. We introduce LLM,odeBed, a codebase and testbed for systematic comparisons, featuring 14 datasets, 8 LLMed algorithms, classic algorithms, and learning paradigms. Through extensive experiments involving 2,700 models, we uncover key insights: In supervised settings, each category offers unique advantages, but LLM- based ap- proaches deliver marginal improvements over classic meth- ods when ample supervision is available. In zero-shot sce- narios, directing powerful LLMs to perform inference with integrated structural context yields the best performance.
Our findings offer practical guidance for practitioners ap- plying LLMS to node classification tasks and highlight re- search gaps, e.g., the limited exploration of LLMs on het- erophilic graphs and the scareity of such text-rich datasets. We hope that LLM will inspire and serve as a valuable toolkit for further research.
Results and Analysis: (1) Scaling within the same series: Comparing Qwen-3B to Qwen-32B (performance shown in Tables 8 and 9, efficiency in Table 16, and performance trends in Figures and 4 in the Appendix), we observe that
9

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Acknowledgments
DeepSeek-AI. Deepseek-v3 technical report, 2024.
This research supported in part by project #MMT-p2- 23 of the Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong, by grants from the Research Grants Council of the Hong Kong SAR, China (No. CUHK 14217622). The authors would like express their gratitude to the reviewers for their feedback, which has improved the clarity and contribution of the paper.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and et al, F. The llama 3 herd of models. ArXiv, abs/2407.21783 2024.
Fang, Fan, D., Zha, D., and Tan, Q. Gaugllm: Improving graph contrastive learning for text-attributed graphs with large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2024.
Impact Statement
In this paper, we did not use any non-public data, unautho- rized software, APIs, and there are privacy or other related ethical concerns associated with our work. Similar to other machine learning models designed for node clas- sification, our L.LM-based algorithms have the potential to be misused for malicious purposes, such as unauthorized surveillance, manipulation of social networks, or exploita- tion of financial systems. We pledge to restrict the usage of models exclusively to research settings to prevent such misuse and encourage the responsible development and de- ployment of these technologies in real-world upplications.
Giles, C. L., Bollacker, K. D., and Lawrence, S. Citeseer: an automatic citation indexing system. In Digital library, 1998.
Grover, A. and Leskovec, J. Node2vee: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining, 2016.
Hamilton, W. L., Ying, R., and Leskovec, Inductive representation learning on large graphs. In NIPS, 2017.
He, X., Bresson, X., Laurent, T., Perold, A., LeCun, Y., and Hooi, B. Harnessing explanations: LLM-1o-LM inter- preter for enhanced text-attributed graph representation learning. In International Conference on Learning Rep- resentations, 2024.
References
Belkin, M., Niyogi. P., and Sindhwani, V. Manifold regular- ization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research, 7(11), 2006.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Waang S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Chai, Z., Zhang, T., Wu, L., Han, K., Hu, X., Huang, X., and Yang, Y. GraphL.MM: Boosting graph reasoning ability of large language model. arXiv preprint arXiv:2310.05845, 2023.
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 18-22133, 2020.
Chen, R., Zhao, T., Jaiswal, A. K., Shah, N., and Wang, Z. LLaGA: Large language and graph assistant. In Proceed- ings of the 41st International Conference on Machine Learning, pp. 7809-7823, 2024a.
Huang, J., Zhang, X., Mei, Q. and Ma, J. Can Ilms ef- fectively leverage graph structural information through prompts, and why? Trans. Mach. Learn. Res., 2024, 2023.
Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., Wang, S., Yin, D., Fan, W., Liu, H., et al. Exploring the po- tential of large language models (Ilms) in learning on graphs. ACM SIGKDD Explorations Newsletter, 25(2): 42-61, 2024b.
Huang, X., Han, K., Yang, Y., Bao, D., Tao, Q., Chai, Z., and Zhu, Q. Can gnn be good adapter for llms? Proceedings of the ACM on Web Conference 2024, 2024.
Chen, Z., Mao, H., Wen, H., Han, H., Jin, W., Zhang, H., Liu, H., and Tang, J. Label-free node classification on graphs with large language models (Ilms), 2024c.
Ji, Y., Liu, C., Chen, X., Ding, Y., Luo, D., Li, M., Lin, W., and Lu, H. Nt-llm: A novel node tokenizer for integrat- ing graph structure into large language models. ArXiv, abs/2410.11 10743, 2024.
Dai, X., Qu, H., Shen, Y., Zhang, B., Wen, Q., Fan, W., Li, D., Tang, J., and Shan, C. How do large language models understand graph patterns? a benchmark for graph pattern comprehension. In International Conference on Learning Representations, 2025.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, S., de Las Casas, D., Bressand, F,, Lengyel,
10

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
G., Lample, G., Saulnier, L.,, Lavaud, L.. R., Lachaux, M.- A., Stock, P., Seao, T. L.,, Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7b. ArXiv, abs/2310.06825, 2023.
Luo, L., Li, Y.-F,, Haffari, G., and Pan, S. Reasoning on graphs: Faithful and interpretable large language model reasoning. In International Conference Learning Rep- resentations, 2024a.
Jin, B., Liu, G., Han, C., Jiang, M. Ji, H., and Han, J. Large language models on graphs: A comprehensive survey. arXiv preprint arXiv:2312.02786 2023a.
Luo, Y., Shi, L.,, and Wu, X.-M. Classic GNNs are strong baselines: Reassessing GNNs for node classification. In The Thirty-eight Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2024b.
Jin, B., Zhang, W., Zhang, Y., Meng, Y., Zhang, X., Zhu,
Q. and Han, J. Patton: Language model pretraining on text-rich networks. In 61st Annual Meeting of the Association for Computational Linguisties, ACL 2023, pp- 7005-7020. Association for Computational Linguisties (ACL), 2023b.
Ma, Y. and Tang, Deep Learning on Graphs. Cambridge University Press, 2021.
Mao, K., Xiao, X., Zhu, J., Lu, B., Tang, R., and He, X. Item tagging for information retrieval: A tripartite graph neural network based approach. In SIGIR, 2020.
Kipf, T. N. and Welling, M. Semi-supervised classifica- tion with graph convolutional networks. In International Conference on Learning Representations, 2017.
Mernyei, P. and Cangea, C. Wiki-es: A wikipedia- based benchmark for graph neural networks. ArXiv, abs/2007.02900 2020.
Kong. L., Feng, J., Liu, H., Huang, C., Huang, J., Chen, Y., and Zhang, M. Gofa: A generative one-for-all model for joint graph language modeling. In International Confer- ence on Learning Representations, 2025.
Ni, J., Li, J., and McAuley, J. Justifying recommendations using distantly- reviews and fine- fine-grained aspects. In Conference on Empirical Methods in Natural Lan- guage Processing, 2019.
Li, X., Chen, W., Chu, Q.. Li, H., Sun, Z., Li, R., Qian, C.,, Wei, Y., Liu, Z., Shi, C., Sun, M., and Yang, C. Can large language models analyze graphs like professionals? benchmark, datasets and models, 2024a.
OpenAI. Gpt-4 technical report, 2024.
Perozzi, B., Fatemi, B., Zelle, D., Tsitsulin, A., Kazemi, M., Al-Rfou, R., and Halcrow, J. Let your graph do the talking: Encoding structured data for llms, 2024.
Li, Y., Ding, K., and Lee, K. Grenade: Graph-centric lan- guage model for self-supervised representation learning text-attributed graphs, 2023a.
Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang. J., Tu, J., Zhang, J., and et al. Qwen2.5 technical report, 2025.
Li, Y., Li, Z., Wang, P., Li, J., Sun, X., Cheng, H., and Yu, J. X. A survey of graph meets large language model: Progress and future directions. arXiv preprint arXiv:2311.12399 2023b.
Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 11 2019.
Li, Y., Wang, P., Li, Z., Yu, J. X., and Li, J. Zerog: Inves- tigating cross-dataset zero-shot transferability in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2024b.
Salton, G. and Buckley, C. Tern-weighting approaches in automatic text retrieval. Inf. Process. Manag., 24:513- 523, 1988.
Li, Y., Wang, P., Zhu, X., Chen, A., Jiang, H., Cai, D., Chan, V. W. K., and Li, J. Glbench: A comprehensive benchmark for graph with large language models. In Proceedings of Neural Information Processing Systems, 2024c.
Sen, P., Namata, G., Bilgic, M., Getoor, L., Gallagher, B., and Eliassi-Ran Rad, T. Collective classification in network data. In The Al Magazine, 2008
Shi, Y., Huang. Z., Wang, W., Zhong, H., Feng, S., and Sun, Y. Masked label prediction: Unified massage passing model for semi semi-supervised classification. In Proceed- ings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCA1-21), 2021.
Liu, H., Feng. J., Kong, L., Liang, N., Tao, D., Chen, Y., and Zhang, M. One for all: Towards training one graph model for all classification tasks. In International Conference on Learning Representations, 2024.
Liu, Y., Ott, M., Goyal, N., Du J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs//a90.1111 1162,, 2019.
Tang, J., Yang. Y., Wei, W., Shi, L., Su, L., Cheng, S., Yin, D., and Huang, C. GraphGPT: Graph instruc- tion tuning for large language models. arXiv preprint arXiv:2310. 13023, 2023.

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Tang, J., Zhang, Q., Li, Y., and Li, J. Grapharena: Bench- marking large language models on graph computational problems. In International Conference on Learning Rep- resentations, 2025.
Yang. Z., Cohen, W., and Salakhudinov, R. Revisiting semi-supervised learning with graph embeddings. In International conference machine learning, pp. 40-48. PMLR, 2016.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. Interna- tional Conference on Learning Representations, 2018.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of Thoughts: Deliberate problem solving with large language models, 2023a.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and act- ing in language models. In International Conference on Learning Representations, 2023b.
Wang, H., Feng, S., He, T., Tan, Z.., Han, X., and Tsvetkov, Y. Can language models solve graph problems in natu- ral language? In s'hirty-seennth Conference on Neural Information Processing Systems, 2023.
Zhang, G., Li, Z.,, Huang, J., Wu, J., Zhou, C., Yang, J., and Gao, efraudcom: An e-commerce fraud detection system via competitive graph neural networks. ACM Transactions on Information Systems (TOIS), 40:1 29,, 2022a.
Wang, H., Liu, S., Wei, R., and Li, P. Model generalization on text attribute graphs: Principles with large language models, 2025.
Wang, L., Yang, N., Huang. X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. Text embeddings by weakly-supervisee contrastive pre-training, ArXiv, abs/2211..0533,, 2022.
Zhang, S., Liu, Y., Sun, Y., and Shah, N. Graph-less neural networks: Teaching old mips new tricks via distillation. In International Conference on Learning Representations, 2022b.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Chi, E. H., Xia, F., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Proceed- ings of Neural Information Processing Systems, 2022.
Zhang, Z., Wang, X., Zhang, Z., Li, H., Qin, Y., and Zhu, W. Llm4dyg: Can large language models solve spatial- temporal problems on dynamic graphs? In SIGKDD, 2024.
Wen, Z. and Fang, Y. Augmenting low-resource text classi- fication with graph-grounded pre-training and prompting. In SIGIR, 2023.
Zhao, J., Qu, M., C., Yan, H., Liu, Q., R., Xie, X., and Tang, J. Learning on large-scale text-attributed graphs via variational inference. In International Conference on Learning Representations, 2023a.
Wu, F., Zhang, T., de Souza, A. H., Fifty, C., Yu, T., and Weinberger, K. Q. Simplifying graph convolutional net- works. In International Conference on Machine Learning, 2019.
Zhao, J., Zhuo, L., Shen, Y., Qu, M., Liu, K., Bronstein, M. M., Zhu, Z., and Tang. J. Graphtext: Graph reasoning in text space. ArXiv, abs/2310,01189, 2023b.
Wu. X., Shen, Y., Shan, C., Song, K., Wang, S., Zhang, B., Feng, J., Cheng, H., Chen, W., Xiong, Y., and Li, D. Can graph learning improve planning in Ilm-based agents? In Proceedings of Neural Information Processing Systems, 2024.
Zhu, J., Yan, Y., Zhao, L., Heimann, M., Akoglu, L., and Koutra, D. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.
Zhu, X., Ghahramani, Z., and Lafferty, J. D. Semi- supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International con- ference on Machine learning, pp. 912 919, 2003.
Xu, K., Hu, W., Leskovee, J., and Jegelka, S. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
Yan, H., Li, C., Long, R., Yan, C., Zhao, J., Zhuang, W., Yin, J., Zhang, P., Han, W., Sun, H., et al. A comprehen- sive study on text-attributed graphs: Benchmarking and rethinking. Advances in Neural Information Processing Systems, 36:17238-1728 2023.
Zhu, Y., Wang, Y., Shi, H., and Tang, S. Efficient tuning and inference for large language models on textual graphs. In Proceedings of the Thirtieth International Joint Confer- ence on Artificial Intelligence (IJCA), 2024.
Zhu, Y., Shi, H., Wang, X., Liu, Y., Wang, Y., Peng, B., Hong, C., and Tang, S. Graphelip: Enhancing trans- ferability in graph foundation models for text-attributed graphs. Proceedings of the ACM on Web Conference 2025, 2025.
Yan, Q. Zhang, Y., Liu, Q. Wu, S., and Wang, L. Relation- aware heterogeneous graph for user profiling. In Pro- ceedings of the 30th ACM International Conference on Information& & Knowledge Management, 2021.
12

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
A. Related Works and Discussion
In this section, we present a comprehensive taxonomy of node classification methods, ranging from classic approaches to those leveraging LLMs.
A.1. Classic Methods
Early approaches for node classification tasks relied on structural techniques such as Laplacian regularization (Belkin et al., 2006), graph embeddings (Yang et al., 2016), and label propagation (Zhu et al., 2003). These methods infer node labels by leveraging the connectivity and similarity among nodes within the graph.
Over the past decade, GNNs have emerged as the dominant paradigm for node classification, demonstrating superior performance across various benchmarks (Kipf & Welling, 2017; Veličković et al., 2018; Hamilton et al., 2017: Xu et al., 2019). GNNs enhance node representations by aggregating and transforming feature information from their local neighborhoods. Formally, given a graph's 's feature matrix X and structure E, GNN produces the predicted label matrix as Y = GNNe X, E) E IR vixc, where the number of classes, and represents the learned parameters.
Beyond structural methods, node classification can also be approached using LMs by treating each node as text entity, Fine-tuning LMs (Liu et al., 2019; Reimers & Gurevych, 2019; Wang al., 2022) allows these models to map textual information directly to node labels, leveraging their strong language understanding capabilities to predict labels. To harness the complementary strengths of GNNs and LMs, i.e., structural and textual information, hybrid LM+GNN architectures have been developed (Jin et al., 2023b; Zhao et al., 2023a; Wen & Fang, 2023; Li et al., 2023a). These models integrate LM- encoded textual features with GNN- proceseed structural features, enhancing node classification performance by combining both modalities.
A.2. LLM-based Methods
LLMs possess an extensive number of parameters, enabling them to generate highly expressive representations. These representations can replace shallow node embeddings, promising to enhance expressiveness and improve the performance of downstream task. A notable approach is ENGINE (Zhu et al., 2024), which utilizes hidden embeddings from LLMs construct node embeddings. ENGINE integrates these embeddings with GNNs propagate and update representations. Specifically, it aggregates hidden embeddings from each LLM layer that processes node's 's text and incorporates them into a cascaded GNN structure.
LLM as Encoder:
LLM as Explainer: A notable advantage of LLMs lies in their generative capabilities, enabling them to intelligently perform a wide range of downstream tasks, from creative generation to reasoning and planning (Luo et al., 2024a; Wu et al., 2024). Motivated by this strength, He et al. (2024) introduced TAPE, a method that leverages L.IM as Explainer for node classification. Specifically,, TAPE prompts the LLM to generate predictions along with chain-of-thought reasoning process that includes explanations, denoted as sap LLM( .p). where p denotes the textual prompt, and o and seu represent the original and generated texts for node v, respectively. Both the original and generated texts are processed by an LM to produce embeddings as Ze orig = LM(s) orig) and exp LM(s.*). which are subsequently processed by GNNs for the classification task as:
Y = Ensemble(GNNe,, (x)r)eg))
E), GNNe, (X E)))
Subsequent works have enhanced the reliability of LLn-ggeerated explanations (Fang et al., 2024). For example, KEA (Chen et al., 2024b) prompts LLMs to extract and explain specific technical terms from a node's original text instead of making direct predictions, thereby mitigating potential misguidance. Overall, the LLM-as-Explainng paradigm harnesses the reasoning and generative capabilities of L.MMs to produce reliable explanations, thereby enriching the original graph data and enhancing downstream tasks.
LLMs' strong reasoning abilities make them effective for direct downstream classification tasks. the L.M-as- Preeddcctrr paradigm, node's textual and structural information, along with task-specific instructions, tokenized and input into LLM for prediction. A notable method in this category is LLaGA (Chen et al., 2024a). Firstly, the original text of node v is encoded via LM as z.LM = LM(s.). Then, a frremeter-free GNN, i.e., SGC (Wu et al., 2019), updates the node embeddings based on the graph structure, initializing with h L. The embeddings from each SGC layer
LLM as Predictor:
13

---

When Do LLMS Help With Node Classification? A Comprehensive Analysis
are concatenated into H, 1 1 and further projected into the LLM'' dimensionality using a projection layer Φg. These projected embeddings are then combined with tokenized instructions T and input into the LLM to generate the predicted label as:
&u = [LM(( (H.) T)).
In the LLaGA framework, only the parameters of the projection layer are tuned, utilizing the next-token -eekkkettrriiiioon loss based on ground-truth labels and generated outputs. GraphGPT employs more complex framework with three distinct pre-training and instruction tuning stages. Other ALM-as-Predictes methods (Chai et al., 2023; Perozzi al., 2024; Kong et al., 2025; Huang et al., 2024; Zhao et al., 2023b; Ji et al., 2024) share similar frameworks with LLaGA but vary in integration approaches, training objectives, and tackled tasks.
A.3. Zero-shot Learning with LLMs
Supervised learning approaches, which rely labeled data, often struggle keep pace with the rapid evolution of worldworld graph data. Zero-shot learning methods address this limitation by enabling models to generalize to unseen data without requiring explicit labels. These methods can be broadly categorized into two approaches: LLM Direct Inference and GFMs.
LLM Direct Inference involves using LLMs to make predictions directly on the node's information through various prompt engineering techniques. Advanced prompt templates for reasoning tasks include Chain-of-Thouge (Wei et al., 2022), ReAct (Yao et al., 2023b), and Tree-of- Thooghtt (Yao et al., 2023a). Besides, structural information can also be integrated into extended prompts (Tang al., 2023; Wang al., 2023; Huang al., 2023), enriching the input provided LLMs and facilitating more accurate predictions.
On the other hand, GFMs are foundation models pre-trained on extensive graph corpora to achieve general graph intelligence. Approaches such as ZeroG (Li et al., 2024b) and OFA (Liu et al., 2024) fine-tune LMs or GNNs on multiple graphs, enabling these models to generalize to unseen graph datasets without extensive retraining. There also exist other zero-shot learning methods utilizing LLMs. For example, Chen et al. (2024c) leverage LLMs as annotators to generate pseudo-labels for GNN training, enabling classification tasks. These approaches are not considered in this work, as our focus is primarily on LLMs directly solving node classification tasks in zero-shot scenarios.
A.4. Benchmarks of LLMs for Graphs
In addition to developments in node classification algorithms, we discuss existing benchmarks that leverage LLMs for graph-related tasks. These benchmarks can be categorized based on the type of tasks they address.
The first category primarily utilizes LLMs for basic graph reasoning tasks, e.g., shortest path and connectivity. For instance, NLGraph (Wang et al., 2023) is a pioneering benchmark that encompasses eight different graph reasoning tasks presented in natural language. LLM4DyG (Zhang et al., 2024) further extends these reasoning tasks to dynamic graph settings. GraphArena (Tang et al., 2025) deals with more complex graph computational problems, with the complexity of tasks ranging from polynomial to NP-Complete like the Traveling Salesman Problem. ProGraph (Li et al., 2024a) evaluates the scalability of LLMs by handling large graphs with up to 10th nodes, necessitating the use of Python APIs for graph analysis rather than relying solely on direct reasoning of LLMss. Additionally, Dai et al. (2025) investigates whether LLMS can recognize graph patterns, e.g., triangles squares, based on terminological or topological descriptions.
The second category focuses on the potential of LLMs for node classification tasks. While numerous surveys discuss the progress in this area (Li et al., 2023b; Jin et al., 2023a), benchmarks that systematically evaluate L.LM node classification methods remain limited. In the preliminary work by Chen et al. (2024b), the exploration is confined to a narrow range of LLM-as as as-Encoder and LLM-as- -----laine approaches. primarily focusing on a limited set of language models. GLBench (Li et al., 2024c) emerges as the first comprehensive benchmark for LLM-based node classification, offering consistent data splits evaluate representative methods in both supervise and zero-shot settings. However, variations in backbone models and implemented codebases impede fair and rigorous comparisons.
Our benchmark, LLMNodeBed, distinguishes itself from existing benchmarks like GLBench by standardizing implementa- tions of baselines, extending learning paradigms and datasets to encompass more real-world contexts, and incorporating influential factors like model type and size, homophily, and prompt design. This comprehensive approach provides more practical guidelines for effectively leveraging L.MSs to enhance node classification tasks.
14

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
B. Prompts
B.1. Prompt in
Methods
For GraphGPT (Tang et al., 2023) and LLaGA (Chen et al., 2024a), we utilize the prompt templates provided in their original papers for several datasets, including Cora and arXiv. For datasets not originally addressed, such as Photo, we adapt their prompt designs to create similarly formatted prompts. For LLM Instruction Tuning, carefully craft prompt templates tailored to each dataset directly guide the LLMS in performing classification tasks. Below is summary of these prompt templates using Cora an example: (labels) denotes the dataset-specific label space (Table 10), (graph) represents the tokenized graph context, and (raw text) refers to the node's original raw text.
Illustration of Prompts Utilized by ALLM-s-Predictoo Methods on the Cora Dataset
LLM Instruction Tuning: Given a node-centered graph with centric node description: (raw text)x)))....... each node represents paper. we need to classify the center node into classes: (labels), please tell me which class the center node belongs to?
GraphGPT: Given a citation graph: (graph), where the 0-th node is the target paper, with the following information: (raw .eet).. Question: Which of the following specific rescarch does this paper belong to: (labels). Directly give the full name of the most likely
category of this paper.
LLaGA: Given a node-centered graph: (graph), each node represents a paper, we need classify the center node into classes:
(labels), please tell me which class the center node belongs to?
B.2. Prompt in Zero-shot Scenarios
For LLM Direct Inference, we consider prompt templates including Direct, Chain-of-Thoughh (Wei et al., 2022), Tree-of- Thought (Yao et al., 2023a), and ReACT (Yao et al., 2023b). The latter three methods are effective in enhancing LLM reasoning abilities across different tasks. Therefore, we adopt these advanced prompts for node classification to assess their continued effectiveness. Illustrations of these prompts on the Cora dataset are shown below:
Illustration of Advanced Prompts Utilized by LLM Inference the Cora Dataset
Given the information of the node: (raw text). Question: Which of the following categories does this paper belong to? Here are the categories: (labels). Reply only with category that you think this paper might belong to. Only reply with the category name without any other words.
Direct:
Chain-of- Chain-of-Thought: Given the information of the node: (raw.text). Question: Which of the following types does this paper belong to? Here are the 7 categories: (labels). Let's think about it step by step. Analyze the content of the node and choose one appropriate category. Output format: (reason: ), (classification::
Tree-of-Thoughh Given the information of the node: (raw_text). Imagine three different experts answering this question. All experts will write down step of their thinking. and then share it with the group. Then all experts will go on to next step, etc. If any expert realizes they't ro wrong at any point then they leave. Question: Based on this information, which the following categories does this paper belong to? Here are categories: (labels). Let's think through this using tree of thought approach. Output format: (discussion: (classification: ). The classification should only consist of one of the category names listed.
ReACT: Given the information of the node: (raw text). Your task is determine which of the following categories this paper belongs to. Here are the categories: (labels). Solve this question by interleaving the Thought, Action, and Observation steps. Thought can reason about the current situation, and Action can be one of the following: (1) Search[entityl, which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not. it will return some similar entities to search. (2) Lookup/keyword|, which returns the next sentence containing the keyword in the current passage. (3) Finish ansser), which returns the answer and finishes the task. The output format must be (process: ), (classification: ). The classification should only consist of one the category names listed.
Additionally, incorporate the central node's structural information into extended prompts evaluate performance. We propose two variants for integrating neighbor information: (1) "w. Neighbor": we concatenate the texts of all 1-hop neighbors of the central node as enriched context, and (2) "w. Summary": we first provide all neighbors' information to LLMs to generate a summary that highlights the common points among the neighbors. Then, we feed both the generated summary and the node's 's text to LLMs to facilitate prediction. The prompt templates for these methods on the Cora dataset are illustrated below:
15

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Illustration of Structure-enriched Prompts Utillized by LLM Inference on the Cora Dataset
Given the information of the node: (raw text). Given the information of its neighbors (raw text). Here give you the content
Neighbor:
node itself and the information 'citation'. Question: Based on this information, which the following sub-categories Al does this paper belong to? Here are the 7 categories: (labels). Reply only one category that you think this paper might belong to. Only reply with the category name without any other words.
1-hop neighbors. The relation between node and its neighbors
w. Summary (Step 1): The following list records papers related to the current one, with the relationship being 'citation': (raw_text), Please summarize the information above with a short paragraph, and find some common points that can reflect the category of the paper.
Summary (Step 2): Given the information of the node: (raw text), (summary). Here give you the content of the node itself and the summary information of its hhoop neighbors. The relation between the node and its neighbors is 'citation'. Question: Based this information, which the following categories only one category that you think this paper might belong to. Only reply with the category name without any other words.
this paper belong Here are categories: (labels). Reply
16

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Table 10: Details of datasets: label space and training data percentages with supervision.
Domain
Dataset
Label Space
Rule Learning Neural Networks, Case Based, Genetic. Algorithms, Theory, Reinforcement Learning. Probabilistic Methods Agents, ML (Machine Leamiig), IR (Information Retrieval), DB (Databases),
Cora
Citeseer
HCI Computtrr Interaction), AI (Artificial Intelligence) Experimentally induced diabetes, Type diabetes, Type 2 diabetes cs.NA, cs cs.MM, cs cs. cs.CY, cs.CR, cs. DC, cs. cs. CE, cs.Nl, cs.CC. cs.AI, cs.MA, cs. GL., NE,NE, cs.SC, AR, cs.CV, es. Gs.RR, cs.ET, es.SY, cs. Cs,CG, cs. OH, es.PL, es.SE, cs. es. cs.Sl, RO, es.IT, cs.PF. Cs.CL., cs.RR, cs.MS, cs. cs.DS, cs. cs.TT, CB cs. cs.
Pubmed
Academic
arXiv
Computational Linguistics, Databases, Operating Systems, Computer Architecture, Computer Security, Internet Protocols, Computer File Systems. Distributed Computing Architecture, Web Technology, Programming Language Topics
Web Link
WikiCS
Instagram
Normal User, Commercial User
Social
Reddit
Normal User, Popular User
World, Americas, Asia, Military, Europe, Russia, Africa,
Books
Ancient Civilizations, Middle East, Historical Study Educational Resources,
Australia & Oceania, Aretic & Antaretica Video Surveillance, Accessories, Binoculars & Scopes, Video,
E-Commerce
Photo
Lighting & Studio, Bags & Cases. Tripods & Monopods, Flashes, Digital Cameras, Film Photography, Lenses, Underwater Photography
Computer Accessories & Peripherals, Tablet Accessories, Laptop Accessories, Computers & Tablets, Computer Components, Data Storage, Networking Products, Monitors, Servers, Tablet Replacement Parts
Computer
Heterophilic
Student, Faculty, Staff, Course, Project
Setting
Cora
Citeseer
Pubmed
arXii
WikiCS
Instagram
Reddit
Books
Photo
Computer
Heterophilic
Semi-supervised Supervised
5.17%
3.77%
0.30%
4. 4.96%
10.00%
10.00%
10.00%:
10.00%
10.00%
10.00%
60.0%
60.0%
60.0%
53.7%
60.0%
60.0%
60.0%
60.0%
60.0%
60.0%
60.0%
C. Supplementary Materials for LLMNodeBed
C.1. Datasets
We selected 14 datasets from academic, web link, social, and e-commerce domains to create diverse graph database. Within LLMNodeBed, each dataset is stored in .pt format using PyTorch, which includes shallow embeddings, raw text of nodes, edge indices, labels, and data splits for convenient loading. The processed data is publicly available at https: : / thuggingg ng face. co/dco//a// coodatt ta aettesssss..... xxwu / LLL.Nodeeeeedd A description of the datasets provided below, with their statisties and additional details summarized in Table and Table 10, respectively.
Academic Networks: The Cora (Sen et al., 2008), Citeseer (Giles et al., 1998), Pubmed (Yang al., 2016), and ogbn-arXiv (abbreviated as "arXiv") (Hu et al., 2020) datasets consist of nodes representing papers, with edges indicating citation relationships. The associated text attributes include each paper's title and abstract, which we use the collected version as follows: Cora and Pubmed from He et al. (2024), Citeseer from Chen et al. (2024b). Within the dataset, each node is labeled according to its category. For example, the arXiv dataset includes 40 CS sub-categories such as AI (Artificial Intelligence) and cs. DB (Databases).
Web Link Network: In the WikiCS dataset (Mernyei & Cangea, 2020), each node represents a Wikipedia page, and edges indicate reference links between pages. The raw text for each node includes the page name and content, which was collected by Liu al. (2024). The classification goal is to categorize each entity into different Wikipedia categories.
Social Networks: The Reddit and Instagram datasets, originally released in Huang et al. (2024), feature nodes
17

---

When Do LLMS Help With Node Classification? A Comprehensive Analysis
representing users, with edges denoting social connections like following relationships. For Reddit, each user's associated text consists of their historically published sub-reddits, while for Instagram, it includes the user's profile page introduction. In Reddit, nodes are labeled to indicate whether the user is popular or normal, while in Instagram, labels specify whether a user is commercial or normal.
E-Commerce Networks: The Ele-Photo (abbreviated as "Photo") and Ele-Computer (abbreviated as "Computer") datasets are derived from the Amazon Electronics dataset (Ni et al., 2019), where each node represents an item the Photo or Computer category. The Hooks-History (abbreviated as "Books") dataset comes from the Amazon Books dataset, where each node corresponds to a book in the history category. We utilize the processed datasets released in Yan al. (2023). In these e-commerce networks, edges indicate co-purchase or co-view relationships. The associated text for each item includes descriptions, e.g., book titles and summaries, involves categorizing these products into fine-grained sub-categories.
reviews. The classification task
Heterophilic Datasets: The Cornell, Texas, Wisconsin, and Washington datasets are collected from Wang et al. (2025), which are the only available heterophilic datasets that have text attributes. Since the original release does not provide shallow embeddings, we generate fixed 300-dimensional embeddings for each node using Vec (Grover & Leskovec, 2016). For dataset splits, we assign semi supervipe and supervised settings with 1:1:8 and 6:2:2 splits for training, validation, and test sets, respectively.
C.2. Implementation Details and Hyperparameters Setting
For GNNs with arbitrary input embeddings, either from shallow embeddings or those generated by LMs or LLMs, we perform a grid-search on the hyperparameters as follows:
num_layers in [2,3,4] 4], hidden hwdden.dimensteeeeeeeeeeeeeeeeee www.ensiinnnnoommmmm in [32, 64, 128, 256]. and dropout in 0.5,
Additionally, we explore the design space by considering the inclusion or exclusion of batcch zation and residual.connect..
For shallow embeddings, the Cora, Citeseer, Pubmed, WikiCS, and arXiv datasets provide initialized embeddings in their released versions (Hu al., 2020; Sen et al., 2008). For remaining datasets lacking shallow embeddings, we construct these embeddings using Nede2Vec (Grover & Leskovec, 2016) techniques, generating fixed 300- dimensional embedding for each node based walk length of 30 and total of 10 walks.
For the heterophilic GNN, H&GCN (Zhu et al., 2020), which we employed on heterophilic graphs in the fine-grained analysis of L.LM-as-Encoder we perform a grid-search over the following hyperparameters:
num_layers in [1, 2, 3]. hi hidden.ciiensson in [32, 64, 128, 256], and learning.rate in [0.001, 0.005, 0.01].
For MLPs with arbitrary input embeddings, we perform a grid-search on the hyperparameters as follows:
num n layers in (2,3, 4), hidden. setetddiiiiiooonno in [128, 256, 512], and dropout in [0.5, 0.7].
For both GNNs and MLPs across experimental datasets, the Learning is consistently set to 1e - 2, following previous studies (He et al., 2024; Li et al., 2024c). The total number of epochs is set to 500 with a patience of 100.
For SenBERT-66M and RoBERTa-355M, we set the training epochs 10 for superiupe settings and 4 for supervised settings. The batch.size is set to 32, and the learning.zate is set to 2e 5.
For GLEM (Zhao et al., 2023a), we uniformly set the number of EM iterations to and the pseudo-labeling ratio to 0.5. For the GNN module within GLEM, set the h t ddme.ddmm 256 and num. lamerayers to 3. For the LM module, we use LoRA for optimization with batch_s' ze of 32. Except for the Pubmed, arXiv, and Computer datasets, where the LM is trained first during the EM iteration, all remaining datasets train the GNN first. This choice is based on empirical findings that suggest better performance.
For ENGINE (Zhu et al., 2024), we refer to the hyperparameter settings outlined in the original paper to determine the hyperparameter search space as follows:
num_la num_layers in 31, hi dideddddddid dimensioon in [64, 128), and learning.rate in [5e 4, le 3]. The neighborhood sampler is set to "Random Walk" for Cora while "k-Hop" with k 2 for the remaining datasets.
18

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
For TAPE (He et al., 2024), we utilize the provided prompt templates guide Mistral-7B and GPT-4o in conducting reasoning. The L.,, RoBERTa-355M, is fine-tuned based its default parameter settings, while the GNN hyper- parameters are explored with layers in [2,3, 4), hidden..diension in [128, 256]. and with or without batch.normalizate
For LLM Instruction Tuning, we use the LoRA (Hu et al., 2022) techniques fine-tune LLMss. The lora parameter (dimension for LoRA update matrices) is set to 8 and the 1ora_alpha (scaling factor) to 16. The dropout ratio is set to 0.1, the batch.s wecccssieee size to 16, and the learning rate to le - 5. For each dataset, the input consists of the node's original text along with a carefully crafted task prompt designed to guide the L.MMs in performing the classification task. The expected output is the corresponding label. For small-scale datasets such as Cora, Citeseer, and Instagram, the number of training epochs is 10 semi-supervised settings and supervised settings. For the remaining datasets, the training epochs are 2 and for semi-supervised and supervised settings, respectively. The maximum input and output lengths are determined based the average token lengths of each dataset.
For LLaGA (Chen et al., 2024a), we empirically find that the HO templates consistently outperform the ND templates. Therefore, we set the HO templates the default configuration, with num hop set 4. We use the text encoder RoBERTa-355M, The linear projection layer (.)-) consists of 2 layer MLP with a hidden. minesss on of 2048. The bat ch is set to 64 and learning_rate semi-supervised settings and 4 for supervised settings. For Qwen2.5-series, we encounter over-fitting issues in the Photo, Computer, and Books datasets, leading us to adjust the learning rate to 5e - 5 and reduce the number of epochs to 2 under supervised settings.
le 4. The number of training epochs is set to 10 for
For GraphGPT (Tang et al., 2023), it includes three distinct stages: (1) text-graph grounding, (2) self-supervised instruction tuning, and (3) task mask-specific instruction tuning. Our empirical findings indicate that the inclusion of stage (1) does not consistently lead to performance improvements, thereby rendering this stage optional. For stage (2), we construct superviie training data for each dataset to perform dataset-specific graph matching tasks, adhering to the provided data format?. In stage (3), we utilize the training data to create (instruction, ground-truth label) pairs following the original prompt design. The training parameters for stage (2) include epochs with 1 parninggrraa of le 4 and a bbtthh s si e of 16. For stage (3), we train for 10 epochs in semi-supervised settings and 6 epochs in supervised settings, with a batch_s ze of 32. Additionally, we adjust the maximum input and output lengths for each stage based on the dataset's 's text statistics.
For LLM Direct Inference, adopt two distinct categories of prompt templates: (1) advanced prompts that enhance the reasoning capabilities of LLMs, and (2) prompts enriched with structural information. These templates illustrated in Appendix B.2 and strictly adhere to the zero-shot setting.
For ZeroG, we adhere its original parameter configurations by setting k = 2, the number of SGC iterations to 10, and the learning.rate to le- 4. In experiments involving GFMs, the intra-domain training mode utilizes the following source-target pairs: arviv > Cora, arXiv >> WikiCS, Reddit -> Instagram, and Computer - Photo.
C.3. Distinct Features
A fair comparison necessitates a benchmark that evaluates all methods using consistent dataloaders, learning paradigms, backbone architectures, and implementation codebases. Our LLMNodeBed carefully follows these guidelines support systematic and comprehensive evaluation of LLM-based node classification algorithms. Unlike existing benchmarks (Li et al., 2024c), which primarily rely on each algorithm's official implementation. LLMNodeBed distinguishes itself in the following ways:
Systematical Implementation: We consolidate common components (e.g.. DataLoader, Evaluation, Backbones) across algorithms to avoid code redundancy and enable fair comparisons and streamlined deployment. For example, several official implementations involve extensive code snippets, and we have produced cleaner, more streamlined versions that enhance both readability and usability. This systematic approach makes L..MNodeBed easily extendable to new datasets or algorithms.
19

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Flexible Selection of Backbones: LLMNodeBBd incorporates a diverse selection of GNNs, LMs, and LLMs, which can be seamlessly integrated components in baseline methods.
GNNs: Our framework supports a wide range of variants, including GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), GAT (Veličković et al., 2018), GIN (Xu et al., 2019), Graph Transformers (Shi et al., 2021) and H2GCN (Zhu et al., 2020). These GNNs can be customized with various layers and embedding dimensions.
..Ms and L.MSs: Open-source models be easily loaded via the Transformers library?. In our experiments, primarily utilize ScnBERT-66M (Reimers & Gurevych, 2019), RoBERTa-355M (Liu et al., 2019), Qwen2.5-Series (Qwen et al., 2025), Mistral-7B (Jiang et al., 2023), and ALL.MA3.1-80 (Dubey et al., 2024). For close-source LLMs, we have formatted the invocation functions of DeepSeek- V3 (DeepSeek-Al. 2024) and GPT-4o (OpenAl, 2024). Additionally, LLMNodeBed allows users to specify and invoke any LM or LLM their choice, providing flexibility for diverse research needs.
LLMNodeBed incorporates comprehensive hyperparameter tuning and design space exploration to fully leverage the potential of the algorithms. For instance, recent research (Luo et al., 2024b) highlights that classic GNNs remain strong baselines for node classification tasks, especially when the design space is expanded through techniques like residual connections, jumping knowledge, and selectable batch normalization. LLMNodeBed supports these enhancements, enabling the full utilization of GNNs. Furthermore, we conduct multiple experimental enhance reliability and account for variability, which often overlooked in previous studies.
Robust Evaluation Protocols:
20

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
D. Supplementary Experiments on Large-scale Dataset
Table 11: Performance comparison (in %) large-scale dataset ogbn-products (Hu et al., 2020).
Classic GATssttefi
Encoder SAGE
Predictor
Metric
GCN
SeaBERT
RoBERTa
GCNu
GATuamiu
I.LGAA
Accuracy
75. 16
77.33
74.83
69.06
75.12
81.09
83.34
82.20
79.07
Macro-F1
36.16
35.63
34.32
24.89
30.29
39,96
41.14
40.49
36.36
We extend our evaluation to the large-scale graph, ogbn-products (Hu et al., 2020), which comprises 2,449,029 nodes and 123,718, 152 edges. In this graph, each node represents unique product sold on Amazon, and edges denote co-purchase relationships. The classification task involves predicting the category of product across 47 classes. This dataset is significantly larger than any other in LLMNNdeBBed, posing unique challenges for evaluating LLM-based node classification algorithms.
Experimental Setups: To evaluate LLM-based methods under constrained GPU resources (e.g., single H100-80GB GPU), we use the official data splits, which include 196, 165 nodes (8.03%) for training, 39,323 nodes (1.61 %) for validation, and a reduced test set of 221,309 nodes (9.04%) yielding a semi-supervised setting. The original dataset provides 100-dimensional node embeddings, which we directly adopt as shallow embeddings for classic GNNs.
Given resouree constraints, we compare classic methods (GNNs and LMs), LLM-as-Encoder methods (using embeddings derived from Qwen2.5-3B), and the such TAPE, are excluded due to the prohibitive cost of generating explanatory texts for the entire dataset via APIs, e.g.. approximately 500 USD for invoking GPT-4o.
Compared Methods:
method, LLaGA. Some other methods,
Results: The results for both Accuracy and Macro-F1 (in %) are summarized in Table 11. From the table, we observe that LLM-based methods outperform the best classic methods by substantial margins, e.g. 6% on Accuracy and 5% on Macro-F1. These results underscore the advantages of LLM-based methods on large-scale graphs.
21

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
E. Supplementary Materials for Comparisons among Algorithm Categories
Table 12: Performance comparison under semi-supervised and supervised settings with Macro-F1 (%) reported.
The best and second-best results are highlighted. LLMr the arXiv dataset requires extensive training time, preventing repeated experiments.
Semi-supervised
Citeseer
WikiCS
Instagram
Reddit
Computer
Avg.
GCNazketa
80.76 10:30
66.00
79,00 10.30
77.87 go0k
52.44
61.15gnm
22.18
62.65 1:44
61.33
62.60
SAGiEaatwea
80.88
65.06
77.88 Lo44
77.02
50.74
56.39_n44
24.37
74.17
71.67
64.24
Classic
GATaatwtts
79.65
14.82
78.43
77.57=101
40.19
60.37g1m
28.93
75.892mm
77.06
64.77
SenBERT-66M
64.54
56.65 1144
12.08
74.97 Looe
55.47 100s
55.75gns
43.15
66.08 10.
59.79
56,50
RoBLKIa-355M
63.36g175
40.82g
73.98
57.43.042
59.23gnm
51.34.a9
67.92.n
63.
GLEM
79.82,095
64.66g2.5
81,92.4497
73.88g001
52.09
49.30gg44
43.33
67.53g1.18
66.01
64,28
GCNELMEnb ENGINE
81 49.19
67.17
78.39
78.58
58,97
68.46_0091
39.64
79.87
77.3640.00
69.95
Encoder
82.52
67.15.a.r
77.53
78.89
57.25
69.56 0.31
34.04
78.55
75,86
69.04
Explainer
TAPE
81.89
66.80
78.46.1
1022
50.01
_1.23_nm
47.12
82.31
84,90
70.31
LLMrr
56.35 10.2
62.81
65.75 jaur
38.301004
44.41 1,86
39.441a4
60.71 1am
$7.38 jo.s
52.50
Predictor
GraphGPT LLaGA
g4.2
26.09
62.04
38.78
38788_n7x
42.85 gn04
65.77
66.69
71.14
52.5
45. 45.12
70.48
50.12
54.67 21124
39.70
79.32,242
78.01
60.12
Supervised
Cora
Citeseer
Pubmed
arXiv
WilCS
Instagram
Reddlt
Books
Photo
Computer
Avg.
GCNoaaeta
86.54_144
71m52
18.5
50.28 1os
82.11 jnss
54,91 .o4
65.00 10
34.39 11:25
66.04 12as
64v
66.39
SAGEsuteno
86.37 $6.37
31
90.16ga37
49,73gg
82.78g155
51.37
61.39
38.29
80.37
82.93
69.53
85.64 24
69.27
87.20go
49.71
82.14 14.144
50.26.2316
64. 11.0
42. 22.85
80.82200
84, 84.74g079
69:72
Classic
ScnBERT-66M
77.13-1.9
71.25a110
93,95
$2.48g011
84.43
56 12
58. aurs
52 52.96.177
70.
28.08
68.21
RoBERTa-355M
81.38=117
72.31
94,33
57,25
86.10
59. 10-1.0
60. 162034
57.24=127
72.89
70.64
71.14
GLEM
85.89
70.06:200
93,43
27
77.60
58.87.1.9
45.05
46.96.335
29.60
68.62g114
GCN IMte
1MM
72.71
87.26
58.22
82.78.144
60.38.001
20.641077
48. 18
80:51gnot
85,10g100
73.05
Encoder
ENGINE
13.
71.22
89.63
56.32ga00
83.80 100
60.02:114
71.17.005
48.24g224
82.77
84.15
73.30
Explainer
TAPE
g7
21.3
92
57.79ga uo
86.03
58.31
65.91
54.07
g3.41
86.78
74.52
LLM
66
66.93-454
52.22.221
93:45
57.48
78. 39.9..2
42 15:434
56.65
49 86-271
68. 18.74:21.4
62, 62,78
62 86
Predictor
GraphGPT
436
61.04
80.98
11.2
56.80
13.92
40.07
39.97
47.97 .1:4
74.22
11:75
62.32
LLaGA
$4.97
11
90.00.n1e
58.05
82.37
57.96_1.
62.14 62.14_11
54.89
83.56
86.97
73.35
Table 13: Performance comparison under zero-shot setting with both Accuracy (%) and Macro-F1 (%) reported.
Type & LLM
Method
Cora (82.52)
WikiCS (68.67)
Instagram (63.35)
Photo (78.50)
Avg.
Acc
Macro-FI
Acc
Macro-FI
Ace
Macro-Fi
Macro-F1
Acc
Macro-FI
Direct
68.06
60.07
71.41
65.21
42.42
39.42
65.25
56.92
61.78
55.40
CoT
68.08
60.32
71.83
60.73
43.47
28.22
63.97
57.61
61.84
$1.72
LLM DeepSeek-VV V3
ToT
66.61
59.10
57.35
54.68
36.95
23.94
59.25
54.39
55.04
48.03
ReAct 65.68
58.68
71.24
60.97
43,30
28.42
63.66
56.23
60.97
51.08
w. Neighbor
68.63
69.21
70.26
64.26
43.34
41.57
61.57
54.84
60.95
57.47
Summary
73.62
64.80
72.53
67.28
41.18
37.56
72.73
70.58
65.02
60,05
Direct
$9.65
70.13
67.80
44.29
42.16
57.54
$5.50
$7.90
$5.95
CoT
57.13
69.00
66.17
45.48
44.56
49,56
$1.42
55.52
$4.82
LLM Mistral-7B
ToT
58.78
57.20
67.56
64.52
45,39
44.73
44.25
54.00
53.33
ReAct 60.32
60.89
71.02
67.31
46.26
46.09
52.47
50.92
57.52
56.30
w. Neighbor
67.69
66.62
68.32
65.58
37.05
28.23
53.39
56.06
56.61
$4.12
w. Summmary
68.12
67.45
70.52
67.87
41,94
38.93
56.01
56.22
59.15
$7.62
22

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
F. Supplementary Materials for Fine-grained Analysis
F.1 -nnoodr-n Compared with LMs
Table 14: Comparison of LLM- and L.M-s--Encdder with Accuracy (%) reported under supervised setting. The best encoder within each method on a dataset is highlighted.
Method
Encoder
Computer
Cora
Pubmed
Photo
Books
Citeseer
WikiCS
arXiv
Instagram
Reddit
Homophily Ratio (%)
85,28
82.52
79.24
78.50
78.05
72.93
68.67
63.53
63.35
55.22
SenBERT
71.75
75.72+2.9
90.26
74.59
83.6 13.6
69.84,231
81.13
68. 68.604a.16
67.12_10:
58.40g0.5
RoBERTa
72.36.am
80.92,265
20.54
75.50xo2s
84.062a.3
74.1121.33
83.18
73.65
68.57.n84
61.06.0.31
MLP
Qwen-3B
69.25g004
81.29
42.02 22.00
74.35g0.47
83,43
72.88
85.462084
74.6240.9
68 62.622
61.39
Mistral-7B
71.38
81.62,nas
92.73
75.83
83,9620.4
73.85
86.92
75.29
69.03.n.3
62.49.0.30
SenBERT
90.53
88.33
89.26.1.24
86.79
84.60
75.3
24028
73.29
68. 18.13..
69.04.0.46
RoBERTa
87.9621.98
89.00
86. 28.499
84,42,0.3
76.57
84, 24.64
14.13
68.4320.51
69.288.0.5
GCN
Qwen-3B
88. 16
88,24 .4.79
88,42 42.42.
85,20.a.v ws
84,34,0
76.37
24.05
73.62.0
68. 18.332.7
71.04.e.r
Mistral-78
89.52,0a1
88. 15.79
88.38
86.07
84.23
76.45
84.78,a.66
74.39
68.27
70.65
SenBERT
90,86
87.36.1.9
90.93
87.4
85.13
74.73.a67
85,94
73.43
67.72.n41
64.13.041
RoBERTa
90.70goas
87.36
90,38
87.42 87.42.0.51
85, 35044
75.90
86.3 16.3120.00
75.284031
68. 84
64,85
SAGE
Qwen-3B
87 87.442006
87.36
89. 29.38
17..44
84.69
75.63
85.58
75.20
68.432155
63, 98.98
Mistral-7B
90.16,008
87.22
90,54.0.50
87,3410.45
85.01
15.20
87.87
76.18.n.34
69.39.n.52
64.34.0.x
We supplement the comparison between LLM--a-EEnooder and L.Mas-Encoderr under supervised settings in Table 14. The key takeaway that LLMs outperform LMs Encoders when graph structure is less informative about the labels, e.g., heterophilic graphs remains valid. This conclusion is particularly evident the arXiv dataset, where the performance gap between LM- and -LM-ennrrated embeddings reaches up to 7% on MLP and 3% on GraphSAGE. Additionally, we observe that in supervised settings, the performance gap between LM- and LLMM-as-Encoders becomes less pronounced compared to semi-supervised settings. We still consider the theoretical insights in Equation (1) for explanation: Increased supervision enhances the mutual information between labels and graph structure, i.e., (i)) thereby rendering the second term less significant and diminishing the advantages provided by more powerful encoders like LLMs.
F.2. P.reeioorred Sensitivity to LLM Backbones
Table 15: Sensitivity of LLaGA to different LLM backbones under semi-supervised Settings.
The best LLM backbone within each series and at similar scales is highlighted.
LLM
Cora
Citeseer
Pubmed
WikiCS
Instagram
Reddit
Books
Photo
Computer
Qwen-3BB
74.15
12.74
54.97
71.64
61.35
65.1
8..26_..45
83.85 o077
Same
series
Qwen-7B
74.23
64.79 14.79
6..58
71.40.222
59.09
81.38_1.s
10.75
84.47
Accuracy (%)
Qwen-14B
76.63
66.06
62.17=6.8
73.76
61.144384
66.59 :.13
81.40
20.4
85.08
Qwen-32R
77.01 13002
64.01 $2:59
58.60_n.98
71.31 23:0
60.24
66.19 66.19
82.34 19.44
82.85
85.74_1..5
Similar scales
Mistral-7B
78.94
62.61 23,0
65.91_2..
76.47:12.20
65.84gu71
70.10....
83.47_9.4
84.44.n90
87.82
Qwen-7B
21.2
64.79 .1.77
62.58
71.40
59.09 23:32
66.07 66.07
23:32
81.38_1.55
80.75
84.47.177
I.L.MA-8 -8B
75.34_1.00
61.33
61.84_362
72.15 72.15
55.77
65.09 1.33
81.30_.0.o
82.26
86.43.a22
Qwen-3B
64.48.433
g2.2
44.35
29.09
51.9228m.
42.422050
37.56
70.94:097
70.65
Macro-F1 (%)
Same series
Qwen-7B
67.30
53.43
47.55
58.9 38.97 23.67
49.78
11.32.1101
34.33_4.5
66.944464
69.23
Qwen-14B
67.85
55.39
45.90
64.10;0224
55,40
44.35
37.50_0.35
70.33
69.19 69.19
Qwen-32B
+6.227
43.59
46.52_27.4
60.50 60.50
39.50
43. 43.96g145
35.30_1.500
70,08
67.26
Mistral-7B
71.14:4.77
52.53
45.12
70.48
$4.67 10.12
39.70-2.44
79.32
78.01:136
Simllar
scales
Qwen-7B
67.30
53.43
47.55_1.16
58.9 18.97
49. 19,788
51.32
34.33-437
66. 64.94
69.23
ILaMA-8B
67.50
51.22
47.80_2.78
64.17
48.5614492
43.31
34.49
22.45
71.43
We evaluate LLaGA with different LLM backbones in semi-supervised settings, detailed in Table 15. We examine two primary trends: (1) Scaling within the same series: Assessing whether scaling laws apply to node classification tasks by using L.MSs from the same series, and (2) Model selection at similar scales: Identifying the most suitable L.M for node classification tasks by comparing models of similar scales.
Scaling within the same series: We plot performance trends across several datasets under both semi-supervised and supervised settings to clearly illustrate these dynamics. From Figure 3 and Figure 4, we conclude that scaling laws generally
23

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Performance (%)
Performance
Performance
60
*0-
Accuracy
Mocro F1
Accuracy
Macro F1
Accuracy
Macro F1
60
50
38
78
148
328
38
78
148
320
38
18
148
328
Qwen Size (Billion)
Qwen Size (Billion)
Qwen Size (Billion)
(a) Cora
(b) WikiCS
(c) Instagram
Figure 3: Performance trends within Qwen-series in different scales using LLLGG framework in semi-supervised settings.
65
Performance e (%)
Performance (%)
Performance
as
80
80
Accuraty
Macre-12
Accuraty
Macro-f1
Acturacy
Macro-F1
38
148
328
38
148
328
38
148
328
Qwen Size (Billion)
Qwen Size (Billion)
Qwen Size (Billion)
(a) Cora
(b) WikiCS
(c) Instagram
Figure 4: Performance trends within Qwen-series in different scales using LLaGA framework in supervised settings.
Table 16: Training and inference times of Qwen-series models ranging from 3B 32B parameters.
Semi-supervised Training Times
Supervised Training Times
Avg. Inference Times Per Case
GPU Device
LLM
Cora
WikiCS
Instagram
Cora
WikiCS
Instagram
Cora
WikiCS
Instagram
Qwen-3B
2.2min
7.1min
5.8min
8.6min
33.5min
20.2min
32.3ms
37.3ms
26.9ms
IVVIDI A6000-48G
Qwen-7B
4.7min
15.3min
8.2min
13.3min
59.4min
43.2min
50.9ms
55.9ms
43.4ms
Qwen-14PP 14B
8. 8.9min
25.5min
15. 1min
30.8min
2.3h
1.5h
97.6ms
103.0mu
23s2ms
NVIDIA A6000-48G
Qwen-32B
18.9min
43.6min
30.7min
52.2min
3.3h
2.0h
254.7ms
262.6ms
232.4ms
hold: as the Qwen model size increases from 3B to 32B parameters, performance improves, indicating that larger model sizes enhance task performance. However, the 7B and 14B models are sufficiently large, typically representing the point beyond which further increases in model size yield only marginal improvements but introducing huge computational costs (Table 16). Unexpectedly. in the Instagram dataset under supervisd settings, the Qwen-32B model experiences performance drop. This may be because 32B models require extensive data to train effectively, making them less robust and stable compared to smaller models. Based on these findings, we recommend the 7B or 14B models as they offer an optimal balance between performance and computational costs.
Model selection at similar scales: By comparing the performance of Mistral-7B, Qwen-7B, and I.L.MA8BB in Table 15, conclude that Mistral-7B outperforms the other two similarly scaled LLMs most cases. This makes Mistral-7B the optimal choice backbone LLM for node classification tasks.
24

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
Method
LLM
LLaGA
Mistra7B 7B
LLMp GraphGPT
Distributton (%)
LLaMA-63
Distribution
40
40
9:18
Valid Type
Valid
Hallucination
Valid
Valid Type
Hallucination
(a) Different LLM-as-Predictor Methods
(b) Different LLM Backbones within LLaGA
Figure 5: Biased predictions by ALM--s-Prediches methods the Instagram dataset: Comparison of ground-truth
label distributions with predictor-generated label distributions.
Mothod Groure Trut
LLM
50
Ground
LLaGA
Marra77
L.
Owen--8
Diverbuuton (N)
Grachcirr
Distributton (%)
40
00
30
20
20
2.52
10
Volld Valld Halluctnation
Volid
Valld
Hallucination
(a) Different LLM-as --eerdcctttto Methods
Different LLM Backbones within LLaGA
Figure 6: Biased predictions by LLM-as-Predictor methods on the Pubmed dataset.
F.3. L.M-as-Predictor: Biased and Hallucinated Predictions
During our experiments, found that LLMM-a--Pediictt methods vulnerable limited supervision. In addition standard metrics such as Accuracy (Table 2) and Macro-Fl scores (Table 12), their predictions also exhibit significant biases and hallucinations.
Biased Predictions: For datasets with fewer labels, Prreict-rr methods tend to disproportionately predict certain labels while under-predicting others. To illustrate this phenomenon, we compare the ground- -ruth label distributions with the predicted label distributions. Specifically, we present different LLM-as-Prediceh methods, LLMrT. GraphGPT, and LLaGA, Figure 5a, and the LLaGA method with various LLM backbones in Figure 5b, using the Instagram dataset in semi-supervised settings, which has two labels.
From Figure 5a, we can directly observe that LLaGA and GraphGPT predominantly bias towards the first class, while LLMI tends to predict the second class more frequently. The predicted label distributions of LLMs are more polarized compared to the ground-truth distributions, where the two labels are roughly in a 4 proportion. In contrast, LLMs tend to predict ratios such as 8 : or 9. This observation also holds across different LLMs, as shown in Figure 5b, where both Qwen-7B and LLaMA-8B tend to bias towards first label. A similar example on the Pubmed dataset, which contains three classes in a semi-super setting, is shown in Figure Here, the predictor methods tend bias towards the third class, while LLaGA with Qwen-7B -BB tends to predict the second class. Additionally, in the semi-supervised setting for Pubmed, the training data consists of only 60 samples, which is insufficient to train robust predictor model, leading to high levels of hallucinations across all methods.
Hallucinations: addition to biased predictions, we observed that a certain portion of the LLMs' outputs fall outside the valid label space or contain unexpected content that cannot be parsed. In superviuee settings, the limited training data restricts these predictor methods from developing effective models, resulting in failures follow instructions
25

---

When Do LLMS Help With Node Classification? A Comprehensive Analysis
Table 17: Average hallucination ratios (%) of LLM--s-Prediictr methods. The hallucination rate is calculated as the proportion of outputs containing invalid labels or unexpected content across all test cases, where higher values indicate poorer classification ability. Hallucinations > 1% are highlighted.
Setting
Method
Cora
Citeseer
Pubmed
WikiCS
Instagram
Reddit
Books
Photo
Computer
# Train Samples
140
120
60
580
1,160
3,344
4,155
4,836
8,722
L.
0.43
13.08
9.24
0.06
0.00
0.00
0.01
0.02
0.02
Semi-supervised
GraphGPP
7.56
2.51
15.97
7.76
0.28
1.78
0.51
0.72
0.27
LLaGA
0.35
0.20
0.29
0.00
0.01
0.02
0.00
0.00
0.00
# Train Samples
1,624
1,911
11,830
7,020
6,803
20,060
24,930
29,017
52,337
LLMrT
0.06
13.61
0.00
0.00
0.00
0.00
0.00
0.01
0.02
Supervised
GraphGPP
1.29
0.63
0.08
1.64
0.13
0.50
0.11
0.11
0.10
LLaGA
0.03
0.00
0.00
0.00
0.00
0.01
0.00
0.00
0.00
Table 18: Examples of hallucinations in GraphGPT's outputs on the Citeseer and Pubmed datasets.
Citeseer
Pubmed
Error Type
Prediction
Ground-truth
Prediction
Ground-truth
Misspelling
AGents
Agents
Type II diabetes
Type 2 diabetes
Logic and Mathematics Information Extraction
ML (Machine Learning)
Type 3 diabetes of the young (MODY)
Type diabetes
Non-existent Types
IR (Information Retrieval)
Genetic Studies of Wolfram Syndrome
Type diabetes
Pattern Recognition
ML (Machine Leaming)
Experimentally induced insulin resistance
Type diabetes
Multiagent Systems
Agents
Experimentally induced oxidative stress
Experimentally induced diabetes
Unexpected Contents
H.4.1 Office Automation: Workdlow Munagement The citation graph is given the following:
membrane altered by diabetes. What is the sensitivity and specificity of the IgA-EMA test
and understand dataset- classification requirements. To illustrate that, we provide both quantitative and qualitative analyses as follows:
Quantitative Analysis: Table 17 presents the hallucination rates of each Preddicco method across various experimental datasets in both semi-supervised and supervised settings. The hallucination rate is calculated as the proportion of outputs containing invalid labels or unexpected content among all test cases, where higher values indicate poorer classification performance. Hallucinations are most severe on the Pubmed and Citeseer datasets within semi-supervised settings, where the number of training samples does not exceed hundreds, making effective model training challenging. This demonstrates that the number of training samples significantly impacts the mitigation of hallucinations: even in semi-supervised settings, larger datasets like Books and Photo provide thousands of training samples, resulting in hallucination ratios consistently below 1%. Therefore, this further verifies that ---edccctr methods require extensive labeled data for effective model training.
Qualitative Analysis: We provide several examples to facilitate the comprehension of hallucinated predictions, which we categorize into three types: (1) misspellings of existing labels, (2) generation of non-existent types, and (3) unexpected content that cannot be parsed. Illustrative examples of these types from GraphGPT's outputs on the Citeseer and Pubmed datasets are presented in Table 18.
26

---

When Do LLMs Help With Node Classification? A Comprehensive Analysis
G. Supplementary Materials for Computational Cost Analysis
Table 19: Total training times of different methods in supervised settings. All recorded experiment times are based single NVIDIA H100-80G GPU.
Type
Method
Cora
Citeseer
Pubmed
WikiCS
ar
Instagram
Reddit
Books
Photo
Computer
Training Samples
1,624
1,911
11,830
7,020
90,941
6,803
20,060
24.930
29.017
52,337
GCNsloiooEEmb
1.8s
1.7s
5.2s
5.1s
51.2s
19.5s
8.5s
14.9s
19.7s
25.8s
GAT
2.1s
1.9s
7.9s
5.7s
1.5m
2.7s
6.9s
16.6s
28.0s
44.6s
Classic
1.7s
3.0s
7.6s
4.0s
1.3m
2.0s
7.2s
19.6s
20.
43.2s
SenBERT-66M
35s
41s
2.5m
7.4m
4.4m
1.8m
4.1m
RoBERTa-355M
1.3m
1.6m
9.2m
5.5m
40.8m
5.3m
15.9m
9.7m
11.9m
22.4m
GCNeLMEsb
1.2m
1,4m
13.4m
7.5m
1.4h
4.5m
1
23.6m
26.8m
44.8m
Encoder
ENGINE
2.2m
2.4m
16.1m
19.4m
2.6h
8.9m
24.2m
35.2m
44.2m
1.2h
Explainer
TAPE
27.4m
30.3m
5.9h
2.8h
37,4h
2.1h
8.3h
10.0h
12.0h
15.0h
LLMr
1.0h
1.3h
9.9h
4.2h
36.3h
2.7h
3.4h
5.7h
12.4h
Predictor
GraphGPT
26.4m
29.5m
2.7h
1.7h
7.8h
49.1m
3.4h
3.8h
3.6h
7.8h
LLaGA
5.6m
7.7m
25.6m
18.8m
7.7h
10.6m
32.2m
1.0h
1.4h
2.5h
Table 20: Inference times of different methods. Values in brackets denote the average inference time per case in milliseconds (ms). All recorded experiment times are based on single NVIDIA H100-80G GPU.
Method
Cora
arXiv
Instagram
Photo
WikiCS
Test Samples
542
48,603
5,847
2,268
9,673
Classic
GCN
0.9ms
21.8ms
2.0ms
7.5ms
4.4ms
Encoder
GCN
14.0s (26ms)
23.8m (29ms)
53.6s (24ms)
5.3m (33ms)
3.7m (38ms)
Explainer
TAPE
5.0m (551ms)
10.4h (767ms)
23.7m (627ms)
2.3h (863ms)
(813ms)
LLMr
1.2m (129ms)
3.3h (243ms)
2.7m (71ms)
24.1m Im (149ms)
5.8m (60ms)
Predictor
GraphiGTT
Im (104ms)
1.2h (87ms)
2.0m (52ms)
10.4m (64ms)
11.0m (1 (1ms)
LLaGA
11.2s (21ms)
57.1m (70ms)
1.3m (35ms)
4.4m (27ms)
2.4m (25ms)
Table 21: Memory costs (in GB) of different methods during training and inference stages.
Training WikiCS
Inference WikiCS
Type
Method
Cora
arXiv
Cora
arviv
GCN Embb
0.76
1.43
4.79
0.37
0.84
3.32
Classic
SenBHRT-66M
9.70
9.70
9.98
3.25
3.26
3.38
RoBERTa-355M
46.00
46.00
46.10
7.44
7.44
7.55
GCN MMEmh
0.85
1.62
7.36
0.40
1.02
5.79
Encoder
ENGINE
1.10
4.52
7.08
1.29
1.81
2.12
Explainer
TАРЕ
46.00
46.00
46.10
7.44
7.44
8.88
LLMp
72.12
67.74
69.13
32.08
31.53
33.12
Predictor
GraphGPT
41.42
53.10
60.41
36.02
36.07
36.97
LLaGA
34.60
35.54
41.94
20.45
20.94
29.00
27


Reviews:
[{"review_id": "gL9eAgeQoh", "reviewer": "Reviewer_4MPh", "summary": "In this paper, the authors provide guidelines for leveraging LLMs to enhance node classification tasks across diverse real-world applications. The authors introduce LLMNodeBed, a codebase and testbed for systematic comparisons, featuring ten datasets, eight LLM-based algorithms, and three learning paradigms. Through extensive experiments, the authors uncover key insights.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful and constructive feedbacks. \n\n--- \n> Weakness 1: Missed references \n\nThank you for pointing out these additional related works. \nWe have incorporated **GAugLLM** [Ref 1] and **GRENADE** [Ref 2] as  baselines. Due to time constraints, we have completed experiments on the Cora, WikiCS, Instagram, and Photo datasets, and will expand the discussion to all datasets in the revised manuscript.\n\nFor GAugLLM, we use Mistral-7B as the backbone LLM. For GRENADE, we consider two different LM backbones. Based on our categorization, GRENADE falls under the **Classic Methods** category, as it involves aligning GNNs and LMs for joint optimization. \n\nAs shown below, GAugLLM, categorized as an LLM-as-Reasoner method, achieves performance comparable to TAPE, further supporting Takeaway 3. For GRENADE, the performance surpasses that of single fine-tuned LMs, demonstrating the effectiveness of aligning LMs with GNNs. However, it still falls short of outperforming SOTA LLM-based methods.\n\n|Semi-supervised Acc (%)|Cora|WikiCS|Instagram|Photo|\n|-|-|-|-|-|  \n|GAugLLM|84.31±0.76|81.13±0.08|66.96±0.19|85.74±0.19|\n|GRENADE (SenBERT-66M)|83.97±0.23|80.99±0.24|66.10±0.24|85.73±0.06|\n|GRENADE (RoBERTa-355M)|83.40±0.58|81.09±0.16|66.57±0.31|85.77±0.20|\n\n|Supervised Acc (%)|Cora|WikiCS|Instagram|Photo|\n|-|-|-|-|-|  \n|GAugLLM|88.45±1.01|84.85±0.44|68.94±0.41|86.93±0.34|\n|GRENADE (SenBERT-66M)|87.50±0.73|84.74±0.40|67.61±0.62|86.67±0.33|\n|GRENADE (RoBERTa-355M)|87.82±0.87|84.84±0.41|68.20±0.32|87.04±0.39|\n\n> Weakness 2: Memory costs \n\nThank you for highlighting the importance of including memory costs. We present the memory costs during both training and inference stages in the supervised setting for three datasets (Cora, WikiCS, and arXiv), scaling from 2,708 to 169,343 nodes. All memory usage was measured on a single H100-80GB GPU to ensure consistency and comparability. \n\n**Training Memory Costs (in GB)** \n|Method|Cora|WikiCS|arXiv|\n|-|-|-|-|\n|GCN$_{\\text{ShallowEmb}}$|0.76|1.43|4.79|\n|SenBert-66M|9.70|9.70|9.98|\n|RoBERTa-355M|46.00|46.00|46.10|\n|GCN$_\\text{LLMEmb}$|0.85|1.62|7.36|\n|ENGINE|1.10|4.52|7.08|\n|TAPE|46.00|46.00|46.10|\n|LLM$_{\\text{IT}}$|72.12|67.74|69.13|\n|GraphGPT|41.42|53.10|60.41|\n|LLaGA|34.60|35.54|41.94|\n\n**Inference Memory Costs (in GB)**\n|Method|Cora|WikiCS|arXiv|\n|-|-|-|-|\n|GCN$_{\\text{ShallowEmb}}$|0.37|0.84|3.32|\n|SenBert-66M|3.25|3.26|3.38|\n|RoBERTa-355M|7.44|7.44|7.55|\n|GCN$_\\text{LLMEmb}$|0.40|1.02|5.79|\n|ENGINE|1.29|1.81|2.12|\n|TAPE|7.44|7.44|8.88|\n|LLM$_{\\text{IT}}$|32.08|31.53|33.12|\n|GraphGPT| 36.02 | 36.07  | 36.97 |\n| LLaGA| 20.45 | 20.94  | 29.00 |\n\nBased on the above statistics, we conclude the following: (i) LLM-as-Encoder methods have memory costs **comparable to classic GNNs**. (ii) Fine-tuning an LM is memory-intensive, especially for larger-scale LMs. (iii) LLM-as-Predictor methods are the most resource-intensive due to the high parameterization of LLMs.\n\nThese results highlight the importance of addressing efficiency challenges for deploying LLM-based methods in practice. We will include these findings in the revised manuscript.\n\n> Question 1: Experiments on larger datasets \n\nThank you for raising the scalability concern regarding the datasets in our benchmark. We have supplemented experiments on the **ogbn-products** dataset, which contains 2,449,029 nodes and 123,718,152 edges.\n\n**Experimental Setup** To evaluate LLM-based methods within one week under limited GPU resources, we use the **official data splits**, which include 196,165 nodes (8.03%) for training, 39,323 nodes (1.61%) for validation, and a reduced test set of 221,309 nodes (9.04%), constituting a **semi-supervised** setting. The original dataset provides 100-dimensional node embeddings, which are directly adopted as shallow embeddings for classic GNNs.\n\n**Compared Methods** Given resource and time constraints, we compare **classic methods**, LLM-as-Encoder method (**GCN** using Qwen-3B-derived embeddings), and LLM-as-Predictor method **LLaGA**. Other methods, such as TAPE, are excluded due to the high cost of generating reasoning texts for the entire dataset via APIs (e.g., ~ 500 USD for GPT-4o-mini). Similarly, $\\text{LLM}_{\\text{IT}}$ is omitted as it would require ~ 4 days for instruction tuning. \n\n**Results (%)** LLM-based methods outperform classic methods by **5%**. \n\n|Method|Accuracy|Macro-F1|\n|-|-|-| \n|GCN$_{\\text{ShallowEmb}}$|55.71|20.61|\n|SAGE$_\\text{ShallowEmb}$|56.04|20.41|\n|GAT$_\\text{ShallowEmb}$|56.15|20.35|\n|SenBert-66M|69.06|24.89|\n|RoBERTa-355M|75.12|30.29|\n|GCN$_{\\text{LLMEmb}}$|77.91|35.91|\n|LLaGA|79.07|36.36|\n\n--- \n\n[Ref 1] \"GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Models.\" In KDD, 2024.\n\n[Ref 2] \"GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs.\" In EMNLP, 2023.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "I believe that the method in this paper is relatively reasonable.", "theoretical_claims": "The paper does not include theoretical proofs, so there is no need to verify the correctness of the theoretical claims in the paper.", "experimental_designs_or_analyses": "Yes, the experimental designs and analysis of the paper are reasonable.", "relation_to_broader_scientific_literature": "Please refer to the weaknesses section.", "essential_references_not_discussed": "Please refer to the weaknesses section.", "other_strengths_and_weaknesses": "Strengths\n1. The authors release LLMNodeBed, a PyG-based testbed designed to facilitate reproducible and rigorous research in LLM-based node classification algorithms.\n2. The paper conducts comprehensive experiments to analyze how the learning paradigm, homophily, language model type and size, and prompt design impact the performance of each algorithm category.\n3. Through detailed experiments, the authors provide intuitive explanations, practical tips, and insights about the strengths and limitations of each algorithm category.\n\nWeaknesses\n1. Some references are missing. Methods[1] that utilize LLMs as enhancers, and approaches[2] that align graphs with LLMs should be categorized under which of the three learning paradigms proposed in this paper?\n2. The authors have provided the time costs for training and inference in the appendix, which is commendable. It is recommended that the authors also present the memory costs of these methods during training and inference, as this would facilitate a more comprehensive evaluation of these methods.\n\n[1]Gaugllm: Improving graph contrastive learning for text-attributed graphs with large language models. SIGKDD 2024\n\n[2]Grenade: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. The datasets selected in this paper are not large in scale. Do the authors have plans to conduct experiments on larger datasets(ogbn-products and ogbn-papers100M)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BA0y9L76CE", "reviewer": "Reviewer_yVQW", "summary": "In this paper, we conduct a fair and systematic comparison of LLM-based node classification algorithms. They developed LLMNodeBed, a comprehensive codebase and testbed for node classification using LLMs. Then,  they conducted extensive experiments, training and evaluating over 2,200 models, to determine the key settings and components that affect performance. Based on the experimental results, they uncover several key insights.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful and constructive feedbacks. \n\n---\n\n> Question 1: Model selection and determination \n\nThank you for the opportunity to clarify the selection process and total number of models evaluated.\n\nFor each baseline on a specific dataset, hyper-parameters were determined based on:\n* Original papers and related benchmarks: We follow settings from original papers, official Github repos, and benchmark reproductions (if applicable) to ensure fair comparisons. Detailed configurations for LLMNodeBed are provided in **Appendix C.2**.\n* Recent research on hyper-parameter tuning: Inspired by studies (e.g., [Ref 1]), we expanded the hyper-parameter search space with techniques like selectable normalization layers and jumping knowledge for GNNs.\n\nAfter performing the hyper-parameter search, we **finalize the configuration for each model** and fix it during subsequent experiments to ensure consistency and reproducibility.\n\nThe total number of 2,200 models is derived from all experiments described in the paper. Below is a detailed breakdown:\n\n* **Main experiments (Table 1)** Semi-supervised: 11 methods × 9 datasets × 4 runs = 396 models. Adding the supervised setting (10 datasets): 11×(9+10)×4 = 836 models.\n* **LLM-as-Encoder (Table 3)** 4 LM/LLM encoders × 3 methods × 9 datasets × 4 runs, excluding overlaps (e.g., GCN with Mistral-7B): (4x3–1)×9×4 = 396 models. Adding supervised: 396 + 440 = 836 models.\n* **LLM-as-Reasoner (Tables 4&5)** TAPE variant with GPT-4o: 9×4 (semi-supervised) + 10×4 (supervised) = 76 models.\n* **LLM-as-Predictor (Table 6)** 5 Additional LLM backbones × 9 datasets × 4 runs (semi-supervised) + 5×10×4 (supervised) = 380 models.\n\nSumming all parts mentioned above, we arrive at a total of ~ 2,200 models. \n\n> Question 2: Differences with existing benchmarks \n\nThank you for bringing our attention to clarify the differences between LLMNodeBed and existing benchmarks. We provide a preliminary comparison in Appendix A.4 and C.3, and we expand upon it for further clarification.\n\nThe most relevant benchmarks that evaluate LLM-based methods on node classification include **GLBench** [Ref 2], **GraphLLM** [Ref 3], and **CS-TAG** [Ref 4]. We first summarize the motivation behind each benchmark to show the overall differences: \n\n* **LLMNodeBed: Insights on when LLMs help node classification** LLMNodeBed is designed to answer this research question. Therefore,  we categorize existing LLM-based methods based on LLM's functional roles, incorporates a variety of variables (learning paradigms, LLM types and scales, dataset domain and homophily, prompt templates, etc.), and provides insights and guidelines for applying LLM-based methods in different contexts.\n* **GLBench: Overall performance comparison between classic and LLM-based methods** GLBench focuses on comparing the overall performance of both classic and LLM-based methods under consistent learning configurations and data splits. \n* **GraphLLM: Fine-grained analysis of LLM-as-Encoder and Reasoner methods** As an earlier benchmark in the LLM4Graph domain (released in July 2023), GraphLLM explores two specific LLM-based paradigms: LLM-as-Encoder (e.g., GNNs/MLPs with LLM-generated embeddings) and LLM-as-Reasoner (TAPE). It provides preliminary findings on the potential of these methods. \n* **CS-TAG: Benchmark for evaluating the performance of GNN, LM, and LM+GNN on text-attributed graphs** CS-TAG enhances the development of graph learning by introducing diverse datasets with text attributes. It systematically investigates the performance of GNNs, LMs, and hybrid LM+GNN methods. \n\nThe table below further provides a detailed comparison of LLMNodeBed and the aforementioned benchmarks across key dimensions, including datasets, evaluation scenarios, baselines, variables considered, and system implementation:\n\n|Benchmark|Datasets|Scenarios|Baselines|Considered Variables|Sys Imple|\n|-|-|-|-|-|-|\n|LLMNodeBed|10 datasets from 4 domains|Semi-supervised, Supervised, Zero-shot|Classic, LLM-as-Encoder, LLM-as-Predictor, LLM-as-Reasoner, GFM|8 LLMs, 2 LMs, 6 Prompts |$\\checkmark$|\n|GLBench|7 datasets from 3 domains|Semi-supervised, Zero-shot|Classic, LLM-as-Encoder, LLM-as-Predictor, LLM-as-Reasoner, GFM| 5 LLMs, 3 LMs|x|\n|GraphLLM|4 datasets from 2 domains|Semi-supervised, Supervised, Zero-shot|Classic, LLM-as-Encoder, LLM-as-Reasoner                     |1 LLM, 5 LMs, 6 Prompts|x|\n|CS-TAG|8 datasets from 2 domains|Semi-supervised, Supervised|Classic, LLM-as-Reasoner|4 LLMs, 11 LMs| $\\checkmark$ |\n\n--- \n\n[Ref 1] \"Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification.\" In NeurIPS, 2024. \n\n[Ref 2] \"GLBench: A Comprehensive Benchmark for Graph with Large Language Models.\" In NeurIPS D&B, 2024. \n\n[Ref 3] \"Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs.\" In KDD, 2024.\n\n[Ref 4] \"A Comprehensive Study on Text-attributed Graphs: Benchmarking and Rethinking.\" In NeurIPS D&B, 2023.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. In the paper, the authors claim the experiments involving 2200 models, it is unclear on how these models were selected and how the number (of models) was determined.\n\n2. For LLMNodeBed, it is better if more discussion related to the difference between the proposed one and existing other benchmarks, such as datasets, baselines.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yihLDLXox0", "reviewer": "Reviewer_HQNY", "summary": "This paper establishes a benchmark for the fair evaluation of different categories of LLM-based methods for node classification and uncovers some insights through performance analysis and comparison.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful and constructive feedbacks. \n\n---\n\n> E1: Experiments on KGs \n\nWe thank the reviewer for suggesting the inclusion of Knowledge Graphs. However, datasets like FB15k-237 and WN18RR primarily focus on the **Link Prediction** task, whereas most LLM-based algorithms are designed for node classification. Adapting these algorithms to link prediction lies beyond the scope of LLMNodeBed, as our benchmark specifically targets node classification, and the baselines we evaluated are tailored for this. \n\n> Weakness 1: GLBench \n\nThank you for the opportunity to clarify the differences between GLBench and our LLMNodeBed. We acknowledge GLBench’s pioneering contribution as the first comprehensive benchmark for evaluating GraphLLM methods, and it inspired aspects of our work.\n\nWhile GLBench focuses on providing **an overall performance leaderboard**, LLMNodeBed extends this by addressing the key question: ***When do LLMs help with the node classification task***? To answer this, we consider numerous variables (e.g., learning paradigm, LLM functional roles, LLM type and scale, dataset homophily, prompt templates, etc.) and perform extensive fine-grained analyses. These analyses aim to provide deeper insights and practical guidelines for the community. In addition, LLMNodeBed offers a systematic implementation of all baselines to ensure fair comparisons and fast deployment. For a detailed comparison, we invite the reviewer to refer to our response to **Question 2 of Reviewer yVQW**.\n\n> Weakness 2: Some insights are not novel \n\nThank you for the feedback. While some takeaways may seem intuitive, they are empirically validated through extensive experiments, providing concrete evidence rather than a vague conclusion. For example, in Takeaway 7, we show that the summary prompt is better than other prompts when integrating neighbor information.\n\n> Question 1: Guidelines for using LLMs instead of GNNs \n\nWe thank the reviewer for this insightful question. First, it's crucial to note that in industrial applications, even a marginal accuracy improvement (e.g., 2%) can yield disproportionately large profit gains given the substantial capital investments involved. \n\nWe agree that homophily alone is insufficient to fully characterize when LLMs should be used instead of GNNs. Based on our extensive studies, we provide the following guidelines: \n* Zero-shot settings: Stronger LLMs can leverage direct inference to achieve satisfactory performance. \n* Supervised settings: Beyond homophily, other factors should be considered like: ratio of supervision (LLM-based algorithms provide a larger gain in semi-supervised settings), and task-feature alignment (when the classification label relies heavily on textual attributes, LLM-as-Reasoner is the best).\n\nWe acknowledge that this question remains open and are committed to uncovering more practical guidelines in the future.\n\n> Question 2: Explanation for mutual information between structure and labels \n\nIn semi-supervised learning, the available labels are a subset of those in supervised learning. Thus, we have $\\mathcal{Y}^{semi} = g(\\mathcal{Y}^{full})$, where $g(\\cdot)$ is a pre-processing function to filter out some labels. Due to the data processing inequality, we have $I(\\mathcal{E}, \\mathcal{Y}^{semi}) \\leq I(\\mathcal{E}, \\mathcal{Y}^{full})$.\n\n> Question 3: LLM-as-Encoder compared with LM-as-Encoder \n\nThe strength of LLMs lies in their ability to better utilize textual information compared to LMs. In homophilic graphs, textual and structural information often overlap, as edges are typically formed between nodes with similar text. As a result, maximizing the use of textual information may yield limited performance gains, since this information is already captured and leveraged by GNNs through the graph structure. In contrast, heterophilic graphs exhibit less overlap between textual and structural information. Therefore, **more effective utilization of textual features in these graphs can result in greater performance improvements**. To support this claim, we present additional experiments on **heterophilic graphs**, which confirm Takeaway 8.\n\nWe include the Cornell, Texas, Wisconsin, and Washington datasets, which have homophily ratios of 11.55%, 6.69%, 16.27%, and 17.07%, respectively. Using the **SAGE** method, we evaluate node features derived from various LMs and LLMs. The semi-supervised performance (Accuracy in %) is summarized below, revealing a performance gap of **5–17%** between LLM-based encoders and their LM counterparts.\n\n|Encoder|Cornell|Texas|Wisconsin|Washington|\n|-|-|-|-|-|\n|SenBert-66M|52.55±1.58|61.73±1.37|70.47±1.75|65.54±2.44|\n|RoBERTa-355M|55.55±3.44|64.26±6.26|73.59±2.72|66.08±1.60|\n|Qwen-3B|**57.13±2.29**|**78.53±1.76**|83.21±1.39|72.18±3.66|\n|Mistral-7B|56.86±1.37|76.53±2.40|**83.96±1.55**|**73.91±0.97**|", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No proofs.", "experimental_designs_or_analyses": "The evaluated benchmarks are all homogeneous. I recommend testing on **knowledge graphs** such as FB15k-237, where GFM has demonstrated significant improvements over classic GNNs [1,2].\n\n[1] Liu, Hao, et al. \"One for all: Towards training one graph model for all classification tasks.\" (ICLR 2024)\n[2]. Kong, Lecheng, et al. \"Gofa: A generative one-for-all model for joint graph language modeling.\" (ICLR 2025).", "relation_to_broader_scientific_literature": "The paper provides a comprehensive evaluation of LLM-based methods for node classification and offers practical guidelines for their usage. Additionally, it introduces _LLMNodeBed_, a codebase designed to facilitate reproducible research in this field.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**\n\n-   The paper presents a fair and comprehensive evaluation of LLM-based methods for node classification, a highly relevant and timely topic in the field, supported by commendable engineering efforts. Additionally, it provides practical guidelines for future use.\n\n**Weaknesses**\n\n-   This work is a direct extension of **GLBench**, the first comprehensive benchmark for evaluating GraphLLM methods in both supervised and zero-shot scenarios. This somewhat diminishes its originality.\n-   Some of the insights provided are not particularly novel or compelling. Certain takeaways are intuitive (e.g., Takeaways 4 and 7), while some have already been established in prior GraphLLM research (e.g., Takeaways 1).\n-   Some explanations for the findings lack sufficient supporting evidence—see _Questions for Authors_.", "other_comments_or_suggestions": "No.", "questions_for_authors": "-   As shown in Tables 1 and 2, classical methods achieve sufficiently strong performance on most node classification benchmarks, even in semi-supervised settings, while being significantly more efficient (milliseconds vs. several hours). This raises a key question: **Under what circumstances should we use LLMs instead of pure GNNs, in terms of both usage scenarios and dataset characteristics?** I believe the homophily ratio alone is not a sufficient metric to fully characterize a dataset, and there is still room to better address this question.\n    \n-   In **Takeaway 2**, you state: _\"In semi-supervised settings, the mutual information between structure and labels is relatively low, allowing LLMs to contribute more significantly to performance.\"_ Do you have any theoretical justification or intuitive explanation for this claim?\n    \n-   In **Takeaway 8: LLM-as-Encoder significantly outperforms LMs in heterophilic graphs**, you analyze the results using mutual information. However, both LLM-as-Encoder and LMs process only node features and are not influenced by graph structure. Thus, I believe the homophily ratio does not contribute to the observed performance differences.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TxTuIsH716", "reviewer": "Reviewer_tHvd", "summary": "The paper systematically analyzes LLM-based node classification, introducing LLMNodeBed, a benchmark with 10 datasets and 8 algorithms for fair comparisons. It finds that LLMs significantly outperform traditional methods in semi-supervised settings but offer marginal gains in supervised learning, with Graph Foundation Models trailing proprietary LLMs like GPT-4o. While LLM-as-Encoder methods are cost-effective and excel in heterophilic graphs, LLM-as-Reasoner works best when labels depend on textual data, and LLM-as-Predictor performs well only with abundant labeled data, though all LLM-based methods remain computationally expensive compared to GNNs.\n\n## update after rebuttal\nThere are remaining concerns as follow:\n- misuse of term usage, e.g., LLM-as-reasoner, supervised learning, semi-supervised learning.\n- wrong dataset with labeling bias\n- insufficient text classification baselines.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your valuable feedback.\n\n--- \n\n> Definition of Reasoning, LLM-as-Reasoner and LLM-as-Predictor, and Graph Foundation Models (GFM)\n\nSorry for any confusion on the terminologies. The categorization of LLM-as-Enhancer, LLM-as-Predictor, and GFMs is adopted from the survey paper [Ref 1]. Specifically, our use of LLM-as-Reasoner corresponds to the explanation-based LLM-as-Enhancer paradigm described in that survey. In this paradigm, LLMs generate predictions accompanied by explanations, requiring reasoning capabilities and general knowledge, as shown in TAPE [Ref 2].\n\nThe key difference between LLM-as-Reasoner and LLM-as-Predictor lies in **whether we directly use the predictions generated by the LLM** [Ref 1]. In the LLM-as-Predictor paradigm, the prediction is the label directly generated by the LLM. In contrast, in the LLM-as-Reasoner paradigm, although the LLM provides an initial prediction, the focus is on its reasoning process, i.e., the explanation text it generates. This reasoning is then used for augmentation, and the final prediction depends on other components like GNNs.\n\nFor GFMs, according to existing survey papers [Ref 1, 3], they are defined as models pre-trained on extensive graph corpora. These models utilize large-scale training data along with specifically designed techniques to achieve strong generalization capabilities across tasks and domains.\n\n> Takeaways 2, 3, 9 \n\nWe thank the reviewer for the opportunity to further clarify our contributions. While Takeaways 2 and 3 may seem intuitive, they have not been explicitly discussed or established in prior research on LLMs for graphs. Our work provides empirical evidence and theoretical understanding to support these observations, contributing novel insights to the field.\n\nIn our paper, we do not have a Takeaway 9, so we assume the reviewer is referring to Takeaway 8. Takeaway 8 does not state that LLMs can perform better if the graph is heterophilic compared to a homophilic counterpart. Takeaway 8 states that in the LLM-as-Encoder paradigm, LLMs perform significantly better than smaller LMs on heterophilic graphs, while achieving only similar performance on highly homophilic graphs. To further support Takeaway 8, we have also added more explanations and four heterophilic datasets and they are given in **Question 3 of Reviewer HQNY**.\n\n> LLM API inference costs\n\nWe thank the reviewer for the valuable suggestion. Due to space limits, we use the Photo dataset as an example, reporting average token consumption and corresponding costs (in USD) for each prompt template across two LLMs. Completing four test datasets requires 59,259,894 tokens, costing ~ 19.5USD for GPT-4o-mini and 35.6USD for DeepSeek-V3. We will extend this discussion to all datasets in the revised manuscript. \n\n|Prompt| Avg. Token| GPT-4o-mini Cost|DeepSeek-V3 Cost|\n|-|-|-|-|\n|Direct|282.6|0.2|0.4|\n|CoT|371.3|0.4|0.8|\n|ToT|505.0|0.7|1.2|\n|ReACT|514.8|0.7|1.2|\n|Neighbor|2607.6|3.1|5.6|\n|Summary|2019.3|5.2|9.5|\n\n> Larger evaluation datasets \n\nThank you for raising the concern regarding the size of the evaluation datasets. We would like to clarify that the arXiv dataset consists of **169,343 nodes**, exceeding the stated threshold of 100,000 nodes. Following standard practices like TAPE [Ref 2], we use the arXiv dataset primarily in the supervised setting. \n\nWe have conducted experiments on a larger dataset, ogbn-products, which contains **2,449,029 nodes** and 123,718,152 edges. Due to space limitations, we invite you to refer to our response to **Question 1 of Reviewer 4MPh** for detailed results and discussion.\n\n> Biased label annotation on arXiv \n\nWhile paper categorization could align with a multi-label classification task, existing research on node classification in Academic Networks consistently adopts a single-label classification approach. Specifically, the arXiv dataset [Ref 4] assigns each paper to its primary computer science sub-category, a setting **widely accepted in prior works**. Modeling this task as multi-label classification is promising but currently infeasible due to the lack of datasets with multi-label annotations. Therefore, our methodology and evaluation adhere to established practices and are appropriate and reasonable.\n\n> Lack of text classification baselines \n\nWe do include text classification baselines in Table 1, i.e., two fine-tuned LMs of different scales (***SentenceBERT-66M*** and ***RoBERTa-355M***). \n\n--- \n[Ref 1] \"A Survey of Graph Meets Large Language Model: Progress and Future Directions.\" In IJCAI, 2024. \n\n[Ref 2] \"Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning.\" In ICLR, 2024. \n\n[Ref 3] \"Graph Foundation Models: Concepts, Opportunities and Challenges.\" In TPAMI, 2024. \n\n[Ref 4] \"Open Graph Benchmark: Datasets for Machine Learning on Graphs.\" In arXiv preprint, 2020.", "claims_and_evidence": "1. The claim of reasoning is confusing, as reasoning involves applying logic to seek the truth, but it is unclear what logical reasoning means in a text classification task. Instead, the process aligns more with cognition, which involves acquiring knowledge through thought and experience. \n\n2. Consequently, the definitions of LLM-as-Reasoner and LLM-as-Predictor are unclear. \n\n3. The definition of Graph Foundation Models (GFM) is also problematic, as it is based on pre-training on graph corpora rather than the model’s ability to generalize across tasks and domains. The referenced papers do not involve graph pre-training but instead, fine-tune on a single graph dataset. \n\n4. Takeaways 2 and 3 are not novel findings but rather common knowledge. \n\n5. Takeaway 9, which claims LLM-as-Encoder performs better on heterophilic graphs, contradicts the fact that LLMs are structure-agnostic. This would imply that GNNs work better on heterophilic graphs than homophilic ones, which is counterintuitive.", "methods_and_evaluation_criteria": "See in Experimental Designs Or Analyses", "theoretical_claims": "The paper does not have any theoretical claims", "experimental_designs_or_analyses": "1. The paper lacks discussion on the cost of LLM API inference. \n2. The evaluation dataset is small, with no more than 100,000 nodes, and the arXiv dataset is only partially used. \n3. Experimental results on LLMs are unconvincing due to biased label annotation, as Ogbn-arxiv simplifies a naturally multi-label task into a single-label one, making LLM predictions seem incorrect when they may actually be reasonable. A comprehensive study, not selective examples, is needed to clarify this issue. \n4. The paper also lacks text classification baselines. Since many LLM methods ignore structural information, the task can be seen as text classification, where simple BERT with bidirectional attention and fine-tuning may perform better than LLMs, which focus on generative tasks rather than deep language understanding.", "relation_to_broader_scientific_literature": "The findings does not have any contribution to the broader scientific literature.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper provides a benchmark analysis for node classification based on LLMs. Specifically, it covers five different ways of using LLMs for node classification as well as different learning paradigms. The comprehensive analysis is a key strength of the paper, highlighted by most reviewers. Nevertheless, the reviewers have also indicated several limitations, mainly related to the following points: (i) Size of the datasets used along with detailed experiments on large-scale graphs. (ii) Choice of baseline methods. (iii) Interpretation of the empirical claims made in the paper. Considering the effort of the authors to address the key points in the response, I'm in favor of accepting the paper, which overall makes a good contribution toward understanding how LLMs can be used in graph machine learning.
Decision: Accept (poster)