Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inference
Authors: Ben Finkelshtein
Silviu Cucerzan
Sujay Kumar Jauhar
Ryen W White
Venue: ICLR 2026 Oral (2026)
Keywords: Large Language Models
Prompting
In-Context Learning
Tool-augmented Reasoning
Text-rich Graphs
Primary area: foundation or frontier models, including LLMs
TL;DR: A comprehensive study of LLMs for node classification, providing a principled understanding of their capabilities in processing graph information that practitioners can apply in real-world tasks

Abstract: Large language models (LLMs) are increasingly leveraged for text-rich graph machine learning tasks, with node classification standing out due to its high-impact application domains such as fraud detection and recommendation systems. 
Yet, despite a surge of interest, the field lacks a principled understanding of the capabilities of LLMs in processing graph data.
In this work, we conduct a large-scale, controlled evaluation across the key axes of variability: the LLM-graph interaction mode, comparing prompting, tool-use, and code generation; dataset domains, spanning citation, web-link, e-commerce, and social networks; homophilic vs. heterophilic regimes; short- vs. long-text features; LLM sizes and reasoning capabilities. We further analyze dependencies by independently truncating features, deleting edges, and removing labels to quantify reliance on input types.
Our findings provide actionable guidance for both research and practice. (1) Code generation mode achieves the strongest overall performance, with especially large gains on long-text or high-degree graphs where prompting quickly exceeds the token budget. (2) All interaction strategies remain effective on heterophilic graphs, challenging the assumption that LLM-based methods collapse under low homophily. (3) Code generation mode is able to flexibly shift its reliance to the most informative input type, whether that be structure, features, or labels.
Together, these results establish a clear picture of the strengths and limitations of current LLM–graph interaction modes and point to design principles for future methods.

Published as a conference paper at ICLR 2026
ACTIONS SPEAK LOUDER THAN PROMPTS: A LARGE- SCALE STUDY OF LLMS FOR GRAPH INFERENCE
Ben Finkelshtein University of Oxford
Silviu Cucerzan
Sujay Kumar Jauhar Microsoft Research
Ryen White Microsoft Research
Microsoft Research
ABSTRACT
Large language models (LLMs) are increasingly used for text-rich graph machine learning tasks such as node classification high-impact domains like fraud detec- tion and recommendation systems. Yet. despite surge of interest, the field lacks a principled understanding of the capabilities of LLMs in their interaction with graph data. In this work, conduct large-scale, controlled evaluation across several key axes of variability to systematically assess the strengths and weaknesses of LLM-based graph reasoning methods in text-based applications. The axes in- clude the LLM-graph interaction mode, comparing prompting, tool-use, and code generation; dataset domains, spanning citation, web-link, e-commerce, and social networks; structural regimes contrasting homophilic and heterophilic graphs; fea- ture characteristics involving both short- and (ong-text node attributes; and model configurations with varying L.M sizes and reasoning capabilities. We further analyze dependencies by methodically truncating features, deleting edges, and removing labels to quantify reliance on input types. Our findings provide practical and actionable guidance. (1) LLMs as code generators achieve the strongest overall performance graph data, with especially large gains long-text high-degree graphs where prompting quickly exceeds the token budget. (2) All interaction strategies remain effective on heterophilic graphs, challenging the assumption that LLM-based methods collapse under low homophily. (3) Code generation is able flexibly adapt its reliance between structure, features, or labels to leverage the most informative input type. Together, these findings provide a comprehensive view of the strengths and limitations of current LLM-graph interaction modes and highlight key design principles for future approaches.
INTRODUCTION
Large language models (LLMs) have rapidly evolved into versatile problem solvers with strong in-context learning, reasoning, and tool-use abilities (Brown et al. 2020; Wei et al.) (022). Their capabilities extend across natural language (Raffel et al. (2020). code completion and synthesis (Chen
et al. 2021a), and cross-modal applications (Liu et al. 2023). Much like in other domains, recent work in graph machine learning has explored leveraging LLMs for tasks such as node classification
(Fatemi al.) 2024 Wu et al. 2025), graph property prediction (Guo et al. 2023) and knowledge graph reasoning (Zhu et al. 2023), with node classification emerging as a dominant task.
This recent wave of interest stems from many high-impact node classification applications that are inherently text-rich and well suited to the strengths of LLMs. These include information retrieval (Su et al. 2024), fraud detection (Yang al. 2025). and recommendation systems (Robinson et al.) 2024). For instance, in fraud detection, accounts are nodes, transactions are edges, and the goal is use textual and relational metadata assign risk labels nodes early prevent financial losses.
Consequently, LLMs have emerged as a viable alternative to the dominant paradigm for graph understanding, Graph Neural Networks (GNNs) (Kipf & Welling 2017a Velicković et al. 2018 Finkelshtein et al. (224),, and exhibit competitive performance on text-rich graphs YYe et al. 2024). While GNNs typically trained per task and dataset and do not transfer across domains label spaces (Finkelshtein et al. 22025,, key advantage of LLMs is their broad world-knowweede (Roberts)
Work performed while Microsoft Research.

---

Published as a conference paper at ICLR 2026
et al. 2020), which can benefit long-text graph datasets. such e-commerce, web- link, and social networks (Shchur et al., 2019; Hu et al., [2020] Pei et al. 2020; Mernyei & Cangea 20200] Hamilton et al. 2017). Furthermore, LLMs have many ways process and reason over graph information: through linearization of text and prompt augmentation; to specialized tool usage for querying the
underlying graph: to generating arbitrary code that operates over the graph. We refer these different
approaches as LLM-graph interaction strategies or modes.
However, despite rapid adoption of LLMs in graph understanding, and node classification in particular, most prior work targets performance for specific domains, graphs, or tasks. As a result, the field currently lacks a principled understanding of the capabilities of LLMs in ther interactions with graph information, and learnings that practitioners can leverage when integrating them into their scenarios.
This principled understanding especially important, since there are many axes of variability and blindly applying LLMs to graphs risks sub-optimal or even detrimental outcomes. Thus, in this paper we conduct a comprehensive, controlled, large-scale evaluation that factorizes these key axes: (1) the LLM-graph interaction mode, comparing four prompting variants, two ReAct-style tool-using variants (Yao et al., 2023), and a programmatic Graph-as-Code medium: (2) dataset domaīns, spanning citation, web-link, e-commerce, and social networks; (3) structural regimes, including homophilic and heterophilic graphs | (4) feature characteristics, comparing short- and long-ter attributes; (5) model scale, ranging from smaller to larger LLMs (across both open- and closed-source families) and (6) reasoning capabilities, contrasting reasoning and non-reasoning variants of L.M.s.
Furthermore, to shed light the inner workings of LLM-based approaches, move beyond reporting overall accuracy and deepen our analysis by probing their reliance on features, structure, and labels inference time. Specifically, we independently truncate textual features, remove known labels, and delete edges, producing 2D accuracy heatmaps that reveal each interaction mode's information dependencies. By isolating the contributions of features, labels, and structure, practitioners can identify which mode exploits particular types of information most effectively, thereby guiding the choice of interaction mode that best matches their application's characteristics (e.g., feature length, homophily) rather than relying on opaque, one-size- silessl--itt-lll solutions.
Findings. Our evaluation yields key insights and guidelines applying LLMs to graph data:
Graph-as-Code achieves the strongest overall performance, with especially large gains on long-text or high-degree graphs where prompting quickly exhausts the token budget.
All LLM-graph interaction modes are effective on heterophilic graphs, challenging the common assumption that they collapse under low homophily (Huang et al. 2024a).
Graph-as-Code is able to flexibly shift its reliance between structure, features, or labels, leveraging the most informative input type.
Experiments on LLM size and reasoning capabilities can be found in Sections B.1 and B.2
RELATED WORK
Textualization and prompting of graphs. Motivated by LLMs' cross-domain performance, early work encodes graphs as text for LLMs, benchmarking varying encoding styles such adjacency lists, edge lists, shortest-path descriptions, and narrative-style encodings (Fatemi et al. 20244). Subsequent studies employ these textualizations evaluate LLMs on node classification using prompting setups
(Huang et al. 2024a] Wang et al. 2024 Li et al. 2024b Dai et al. 2025 Ye et al. 2024 Guan et al. 2025). They find that carefully designed prompts can allow LLMs to compete with GNNs (Ye et al. 2024). that performance often hinges on neighborhood homophily (Huang et al. (024a).. and that their abilities remain brittle and sensitive to input data and formatting (Wang et al.J (024). More mechanistic analyses of attention patterns further suggest that LLMS may mirror prompt format rather than execute explicit graph computation (Guan et al. 2025).
Concurrently, alternatives to textualization have emerged, either by tokenizing each node based on local structure and features (Zhao et al. 2024) or by introducing learnable components that encode structure and features (Perozzi et al. 2024). Yet prompt-based approaches remain the default, with
'Homophily is the tendency of nodes to connect with others of the same class, whereas heterophily is the tendency connect predominantly with nodes of different classes.
2

---

Published as a conference paper at ICLR 2026
Data
Gouph
Groph Data
Instractions
Agent
</>
Instructions
Agant
Action
Instructions
Agent
Pandex code
Prompting
ToaphTool
Graph- as- Code
Figure Illustration of the LLM -aMMMggrphhhhhh interaction strategies described in Section 3.1
most recent work still relying on them (Wu et al. 2025 Tang et al. 2024
2025 Guan et al. 2025)
Our analysis contrasts prompting with tool-use and Graph-as- -odee interaction modes, revealing competitive and often superior performance on graphs with large textual features or high-degree nodes. Contrary conclusions drawn from prompting alone (Huang al. 2024a), also find all modes vi- able on heterophilous graphs, with the added advantage that Graph-as ased--s--Cdee shows reduced brittleness.
Tool-calling for graph reasoning. Recent advances in LLM orchestration have introduced tool- calling and ReAct-style paradigms, enabling LLMs to interface with external APIs or reasoning modules for enhanced problem-solving (Yao al. 2023 Schick et al., 2023). These approaches interleave natural language reasoning with calls to task specific tools, a allowing LLMs to retrieve,
manipulate, or compute over structured data. Variants such as Plan-and and-Execute (Wang et al. 2023).
Reflexion (Shinn et al., 2023), and Graph-of-Thoughe (Besta et al.) 2024) have shown that LLMs can decompose complex tasks into sequences of actions and adapt plans via feedback. Tool-calling has been extensively benchmarked for real-world utility across domains planning (Huang et al.) 2024b),
code API usage (Patil et al. 2024]. mathematical reasoning (Gou et al., 2023), and multi-moda reasoning (Lu et al. 2023). In the context of graph data, LLMs have been combined with tool-calling to perform graph classification, knowledge graph reasoning and node classification, achieving high performance (Zhang) 2023] Edge al. 2024). However, these works focus on specialized workflows or proof-of-concepl demonstrations. We extend this line of research by systematically evaluating varying tool-calling paradigms for node classification across diverse datasets and graph regimes, highlighting their strengths, limitations and dependencies on features, labels and structure.
An additional related work discussion can be found in Section A
3 AXES OF VARIABILITY
Our goal to build principled understanding of the capabilities of LLMs in processing graph infor- mation. In this section, factorize key axes of variability to enable controlled comparisons that iso- late the influence of each factor,and clarif their dependencies on graph features, structure, and labels.
Notations. Let T = - nn be the set of all finite token sequences over vocabulary L. We consider an unweighted graph G = (V, F., X, Y) with N = |V| nodes, adjacency matrix A E {0, 1) NxNN node features X E T and labels Y E CN for label set C S T of size C. For any matrix M E NxD and node subset S C Ms E RISSD denotes the submatrix of rows indexed by S.
Node classification. The task is to predict labels Yo E ClQl for a set of query nodes Q C V, given (i) known labels YK E C for a set of labeled nodes K C V \ Q, (ii) the graph structure (either A or E), and (ii) the textual features X E T of all nodes.
We model this task using the LLM-graph interaction models Oprompe Proob Peode T x T x
{0, 1] NNN + T, corresponding to Prompting, GraphTool and Graph-as-Code respectively. Each mode encodes the chat history in T. node features in T and graph structure into finite sequence, which is then processed by an LLM, T + T with parameters A E RD. The output may provide predicted labels in C C T or update the chat history for further interaction.
3.1 VARIABILITY OVER INTERACTION STRATEGIES AND MODEL CONFIGURATIONS
We study the following three I.MM-grahh interaction modes (Figure
3

---

Published as a conference paper at ICLR 2026
(1) Prompting 0 Oprompt. In this simple and widely-used mode (Fatemi et al. 2024 Huang al. 2024a| Guan et al., 2025), the entire context for the model is constructed and issued to the model as single-turn inference. The prompt (i) provides all classes (ii) presents the target node's ' textual description and known label (if available), and (iii) serializes the k-hop neighborhood grouped by hop distance, specifying for each encountered node its description and label (or None for held-out nodes). A complete prompt template for this mode is provided in Template
The hop number is a hyperparameter controlling the degree of neighborhood information. We experi- ment with three variants: 0-hop prompt, 1-hop prompt, and 2-hop prompi, corresponding to radii of 0, and 2. respectively. To keep the context within token budget for long-text datasets, we also experi- ment with an additional budget prompt variant, which caps the neighbors at each hop by subsampling.
(2) GraphTool Groot- Motivated by ReAct (Yao et al. 2023), we frame node classification as an iter- ative think-act-obsera loop. At each step, the L.M reasons about what is known and what remains missing, then issues single action from fixed tool set. The environment executes the action on the graph and returns the result, which is appended to the interaction history. The process repeats until the LLM decides to terminate and predict a label. This ReAct-style interaction encourages planning and targeted retrieval of graph structure and text, reducing irrelevant exposure and token usage.
In our basic variation, GruphTool, the following actions are available: (0) The terminal action submits the final label. (1) A topology-only action retrieves the neighbors of a specified node, enabling exploration without consuming feature tokens. (2) A feature-only action returns the textual description of a specified node. (3) A label-only action reveals the label of the requested node if in the training set (and None otherwise), allowing the model to anchor reasoning on known examples while avoiding leakage on held-out nodes. We also introduce GraphTool+, which extends the base GrophTool variant with additional exact-k hop retrieval actions: (4) retrieves the textual descriptions of all nodes exactly hops away from a specified node; (5) retrieves their labels (or None for held-out nodes). A complete prompt template for this mode is provided in Template
(3) Graph-as-Code écode- Building on LLMs' strong code generation capabilities (Chen al. 202 la Liu et al. 2025), we extend the ReAct paradigm beyond a fixed, predefined action set. In this mode, the graph data is represented as a typed table indexed by node id with columns Features (text), ne neighbors (list of node IDs), and 1 label (integer or None). The LLM generates, executes, then reasons over the outputs of compact programs an iterative fashion. The process repeats until the LLM decides to terminate and predict a label. This code-native mode enables compositional access to structure and features and can collapse multi-step tool sequences into a single query, improving step and token efficiency while remaining transparent and auditable. A complete prompt template for this mode is provided in Template 3
Variability across LLM sizes and reasoning. We evaluate models from small (L1 (mlama) to large (CPT-5), including reasoning and non-reasoning variants of Phi-4 and Owen, to assess the impact of scale and reasoning. o4 4-minni is our primary model; additional results are in the Appendix.
3.2
VARIABILITY OVER DATASET DOMAINS, HOMOPHILY LEVELS AND TEXT LENGTHS
Dataset domains. We evaluate LLM performance across diverse graph domains, such as citation
network datasets cora, pubmed, and arxiv, where nodes are papers with titles as features (Huang et al.
2024a); e-commerce graph datasets products, computers and photo (Huang et al. 2024a] Wu et al.
2025). where nodes are items with product title, descriptions or reviews; web-link network datasets
cornell, texas, washington, wisconsin, and wiki-cs (Wu et al.] 2025), where nodes are webpages
described by their page-level text; social network datasets reddit and instagram with user profiles and comment snippets (Wu et al. (0255). This domain variability enables comprehensive assessment of LLM generalization and adaptation to different graph types.
Graph structure regimes. In homophilic graphs such citation networks, e-commerce and social networks, local label information is highly important for correct prediction. Conversely, heterophilic graphs such as web-link networks, challenge models to rely less simple local label information and more on node features and graph structure. By evaluating performance across these regimes, we aim to uncover the varying dependencies of LLMs on graph features, structure, and labels.
Textual feature characteristics. The datasets also vary in the richness and complexity of node textual features. Short-text datasets, such cora, pubmed, arxiv and products provide only titles product
4

---

Published as a conference paper at ICLR 2026
Table 1: Accuracy of baselines and L.M-graph interaction modes Prompting, Groph and Graph-
as-Code on short-text homophilic datasets. Best per-dataset results are bold, runner-up underlined.
cora
pubmed
urxiv
products
# Classes
7
3
40
47
Avg. text length
13
110.47
68.99
54.08
Hom. (%)
82.52
79.24
63.53
63.84
Avg. degree
4.92
6.30
13.64
61.37
Random
14.13:1::06
33.10:11.19
2.44110223
2.33:1:.25
Majority label
29.00:10.89
41. 41.10.44088
5,90
26.10:12.49
Label propagation
76.61:1.94
80.80:12.93
6..00-1.666
70.40=11.66
O-hop prompt
64.11:110.68
89.20
68. 10::3.111
70,00
1-hop prompt
81.92+11.8
91.30++2.02
73.80 +1.2
82.20+3.98
2-hop prompt
83.43-+22.25
9188++2.17
74.30+22.53
Tokenl.imi
GraphTool
74.02+11.18
89.50+2.32
57.50+5.500
7..30+3.06
GrophTool+ Graph-as-Code
81.40=t3300
91.90++2.16
73.30+22.86
3..50+3.443
85.16:11.47
19.90+
74.40+3.02
82.70=2.666
names, offering limited semantic signal for classification. In contrast, long-text datasets such puters, photo, reddit, instagram and wiki-cs include detailed descriptions or user profiles, presenting both opportunities for deeper reasoning and challenges for efficient context processing by LLMs.
EXPERIMENTS ACROSS AXES OF VARIABILITY
We evaluate L.Mss across multiple axes of variability. We organize this section into three parts: short- text homophilic datasets (section I. heterophilic datasets (section 4.2) and long-text homophilic datasets (section 4.3) In each setting, we introduce additional baselines to contextualize performance.
Baselines. We compare against several baselines: Random, which predicts labels uniformly at random, serving as a natural lower bound; Majority Label, which assigns the most frequent label from the training and validation sets to all test nodes; and the classic Label Propagation (LP) algorithm. In LP, node labels are represented as one-hot vectors E {(0, 1J unknown labels initialized zero vectors. The (random-walk) normalized adjacency matrix is defined as Á D A, where D diagonal degree matrix D diag(d),... dn) with
Σラ= Aij
denoting the degree of node Predictions Y E NCC computed by Y ===a Á'Y, where = 10 the number of steps, and each node assigned the label with the highest score.
4.1 SHORT-TEXT HOMOPHILIC DATASETS
Finding 1. Prompting and Graph-as- Code are closely competitive on short-text homophilic datasets.
Table reaffirms prior work by showing that all LLM-based approaches substantially outperform trivial baselines such random guessing and majority label assignment. This confirms LLMs leverage both textual node features and graph structure for classification in homophilic regimes. Furthermore, within the Prompting interaction mode, accuracy increases with the inclusion of neighborhood context, moving from self to 1-hop and 2-hop prompt variants, consistent with established findings Huang et al. (2024a); Wu et al. (2025). However, graphs with high average degree, context token limits are reached quickly, restricting possible gains from additional neighborhood information.
Finding 2. In ReAct-basee methods, more flexible variants perform better on homophilic datasets.
The ReAct-based interaction strategies exhibits clear trend from GraphTool to GraphTool++ to Sraph-as-Code As the LLM is given greater agency and adaptivity in interacting with the graph moving from fixed tool invocation (GraphTool), to enhanced retrieval options (GraphTool+ and finally to the fully programmatic Graph-as-Ce Lass-Code-aaee clade-llasifiiatiion. accuracy improves (Finding 2 We believe that adaptivity valuable in homophilic settings, as local neighborhood labels are highly informative but the optimal aggregation strategy may also depend node degree and graph topology. Thus, increased agency empowers the LLM to tailor its reasoning and retrieval to the specific structure of each instance, resulting in stronger overall performance.
5

---

Published as a conference paper at ICLR 2026
Table 2: Accuracy of baselines and L.--graph interaction modes Prompting. Troph and Graph- as-Code on heterophilic datasets. Best per-dataset results are bold, rumner-up underlined.
cornell
texas
washington
wisconsin
# Classes
5
5
5
Avg. text length
2039.69
2427.40
1597.53
2109.55
Hom. (%)
11,55
6.69
17.07
16.27
Avg. degree
1.53
1.66
1.72
1.89
Random
21.74=33.19
8.40=11.56
20.43-4-2.87
18.11=3.62
Majority lahel
42.43-+:1.56
58.70+11400
45.94-13.64
44.15-2.20
Label propagation
41.74++1.06
78.90+ 1.67
15.07++4.21
14,21-2.69
0-hop prompt
81.57::1.800
53.20:+3.19
10.14+2..54
84.78
1-hop prompt
81.39:10.99
21.40::2.00
81.74::1..0
88.81=1.43
2-hop prompt
84.17=1.43
TokenL.mmt
84.35+1.67
91.45-1.87
GraphTool
91.30+22.46
59.60+-2.33
80.14+1.32
87.04-1.58
GraphTool+
91.13+2.97
63.70++2.36
80.41+00.44
37.42-1.66
Graph-as Code
92.70
3.760
81.96:t2..2
89.17 2.69
Table 3: Accuracy of baselines and LLM-graph interaction modes Prompting, GraphTool, and Graph- as-Code on long-text homophilic datasets. Best per-dataset results are bold, runner-up underlined.
citescer
reddit
computer
photo
instagram
wiki-es
# Classes
10
12
10
Avg. text length
1018.97
761.82
792.77
797.82
509.64
3215.56
Homn (%)
72.93
55.52
85,28
78.50
63.35
68.67
Avg. degree
1.34
5.93
8.27
10.36
12.70
18.45
Random
1..80
51.60=2.97
10.00-12.21
8.20
50.30:13.07
9.90-12.113
Majority label
21.90:12.10
52. 2..9772.97
24.20
42.30:12.25
65.10:1..90
21.50:13.89
Label propagation
37.30=5.03
10.50=166
1.0
75.301:0.91
52.00:12.98
71.90=2.19
0-hop prompt
68.20=2.77
47.90=3.86
65.50=3332
19.80-33.15
48.00+:3.66
74.00=3.10
1 1-hop prompt
68.30+22.59
59.30=4.12
86.10+2.55
85.80++1.92
16.10++2.30
Limt
2 2-hop prompt
69.40+22.10
Tokenl.imit
limit
Tokenl
TokenLimit
LikenLimit
2-hop budgel prompt
70.20:12.83
54.40-3.885
86.0012.41
85.60 63.60111.63
54.50-14.44
80,8011344
GraphTool
68.30=
56.25-1.84
80.80++4.61
77.00++2.32
47.80++4.09
76.27++.18
GraphTooke Tool+
68.70+-2.31
61,80=1.15
83.10-+2.63
81.30-10.99
48.20++2.92
80.50+3.10
Graph-as- -ocd-e
71.80
61.60-2.36
86.20
86.40.12.65
56.40:12.56
82.2013.63
4.2 HETEROPHILIC DATASETS
Finding 3. All LLM-graph interaction modes are effective on heterophilic datasets.
Table 2 challenges common assumptions and prior work (Huang et al. 2024a) which suggests LLMs struggle on heterophilic graphs. In this setting, local label information can be peed-reddiiive even misleading, making it difficult for LLMs to rely on neighborhood cues for accurate classification. De- spite low levels of homophily, all LLM-graph interaction modes (Oprompt, Ptool, @cnde) achieve strong accuracy (Finding 3. consistently outperforming classic baselines such as majority labeling and label propagation. This demonstrates that LLMs exploit non-local or feature-based cues for classifi- cation, rather than relying solely on simple neighborhood voting heuristics. Here, the context window token limit is reached again, primarily due long textual features, rather than graph degree, which constrains the amount of context that can be included prompts. Nevertheless, providing more con- text in prompting proves beneficial, contrary popular belief, and similar the homophilic setting.
Furthermore, ReAct-based variants exhibit a clear advantage with increased agency and adaptivity in interacting with the graph moving from fixed tool invocation (GraphTool), to enhanced retrieval options (GraphTool+), and finally to the fully programmatic Graph-as-Code reinforcing Finding 2 in the heterophilic setting as well. The Graph-as-Co -Cdde variant (doode in particular excels, likely due to its compositional access to features and structure, which is especially advantageous when neighborhood labels are diverse or uninformative.
4.3 LONG-TEXE DATASETS
Finding 4, Graph--ss- Code significantly outperforms Prompting and GraphTool long-text datasets.

---

Published as a conference paper at ICLR 2026
cor3
cornell
100
26.4
24.7
20.0
4.3
15.0
12.0
2.3
102
723
10
TL.
821
90
Prompting
Edge Deletion
75
56.3
56.5
1122
452
64.9
75-
35.0
51.1
95.8
33.6
75
77.7
735
60.
50
712
29.3
67.0
15.6
85.5
$6.0
50
07.2
64.3
61.3
62.
30.7
55.7
50
00.9
135
76.0
725
03
23
25
78.0
70.2
76.7
25
02.4
07.5
01.4
01.7
76.3
74
09.
20
25
90
90
100
25
50
75
100
50
75
90
100
102
Edga Deletinn [88]
90
732
64.5
42.5
42.2
417
4
90
72.2
67.1
62.6
91.1
43.9
39.3
92
92.9
675
67.5
75.8
34
Graph-as-Cool
76.6
70.6
56.0
90.
516
16.0
75
73.7
25.1
67.5
60.1
55.0
51.1
932
88
08
63.7
78.80
75.6
77.3
10.9
92.9
71.0
69.5
64.2
92.2
68
78.3
51.1
25
85.2
63.6
622
61.3
81.0
64.6
16.4
7..4
71.2
69.8
68.9
69.3
2 92.7
08
65.7
79.1
22.4
25 50
90
100
25
30
100
50 75
90 100
Reature Deletion (%]
Feature Delerion (96]
Reature Detec on 196]
Figure 2: Accuracy of 2-hop prompting and Graph-as- Codde on the cora, arxiv, and cornell datasets under varying ratios of randomly removed edges and truncated text features.
suggesss of performance gap, with Prompting (éprompt) performing worst and Graph Code best (Finding[4) 4). In this setting. Prompting is constrained by node feature length and neighborhood size, both quickly exhausting the model's s' token budget. Notably, similar loken-limin issues appeared in previous sections-for for the products dataset in short-text homophilic benchmarks and for the rexas and washington datasets in the heterophilic regime when features were long or neighborhoods large.
To mitigate these constraints, we introduce the 2-hop budget prompting variant, which caps the number of neighbors per node via sampling. While this adjustment helps avoid hitting the token limit and allows the model to reason over a sampled subset of context, Prompting still remains the least effective variant, with Graph-as- Code the most effective one (Finding()). This likely due the noise and information loss introduced by sampling, which can obscure important neighborhood signals.
These results demonstrate that Graph-as- (donde) can offer substantial ad advantages for LLM-based node classification in dense or feature-rich graphs, which represent a large proportion of real-world networks such as e-commerce and recommendation networks. Furthermore, while context windows are indeed expanding, prior work has shown that LLMs often struggle to effectively leverage long inputs, and that more context does not always translate to better reasoning (An et al.. 2024 [i et al. 2024a). Thus, approaches like Graph-as-Code that restructure graph information to highlight salient structure and reduce redundancy remain crucial, even as model capacities grow.
5 EXPERIMENTS ON FEATURES, STRUCTURE, AND LABELS DEPENDENCIES
In this section, we examine whether the widely used Prompting (ppromp)) and the best performing Graph-as- Graph-as-Code Code (dcnde) interaction modes rely similarly on node features, graph structure, and labels. We organize the analysis into two parts: (i) the effect (or dependency) of removing portions of node features and edges (section 5.T). and (ii) the effect of removing labels and edges (section 5.2). We visualize results with 2D heatmaps of accuracy against feature, edge, and label removal rates.
Additional setup. For each dataset, deletion rates, and LLM-graph interaction strategies, we predict the labels of 1, 000 randomly sampled test nodes per seed and average over five independent runs, reporting the mean as in previous experiments. However, in both Section 5.1 and Section 5.2 we run partial-deletie -elleton experiments: edges and labels are removed uniformly random, while feature dele- tion is implemented by truncating each node's 's text to the fixed percentage of tokens. We adopt trunca- tion as provides a simple, model-agnostic, and reproducible way to scale the information available to the LLM, while avoiding additional assumptions about feature semantics that could bias results.
5.1 FEATURES VS. STRUCTURE DEPENDENCIES
Finding 5. Prompting and Graph--s- Code exhibit comparable use of node features and structure.
7

---

Published as a conference paper at ICLR 2026
Figure 2 shows that, for all three datasets, the two panels have nearly identical characteristics, indicating that Prompting (Qprompi) and Graph-as- -odde (ócode) share the same dependence on features and structure. On cora and arxiv, accuracy drops mainly with edge deletion, while cornell is driven primarily by feature deletion (Finding (5) This alignment arises from the inherent characteristics of the datasets. Both cora and arxiv are highly homophilie, meaning nodes of the same class are densely interconnected. In these settings, structural information, specifically the local label context provided by edges, has high impact on accuracy, so removing edges disrupts information flow and leads to a decrease in accuracy for both methods. Conversely, cornell is a heterophilic dataset, where nodes of different classes are more likely be connected and the graph structure is less informative. Here, node features are more discriminative than the sparse and less meaningful edge connections, making feature deletion the dominant factor impacting performance.
Finding 6. Graph-as-Cou Code is more robust than Prompting to feature, structure and label deletion.
Beyond the shared dependency, Graph-as as-Code (Éoode) consistently outperforms Prompting and is more resilient to perturbations (Finding (). When structure is completely removed but features are intact, Graph-as-Code
preserves high accuracy on all datasets, whereas Prompting collapses. This difference arises because Graph-as-Conn can access feature and label information of other nodes even when edges are absent, whereas Prompting depends on edge connections to retrieve this information.
Finding 7. When the prompt size reaches the token limit, the behavior of Graph-as Code and Prompting diverges, with Graph-as- -ode performing significantly better.
Long-text homophilic graphs further challenge Finding[5 5 because 2-hop prompts are prone to hitting the token limit. Figure 3 shows that on photo we indeed observe a di-
photo
100
35.5
20.4 27.2
23.0
18.1
11.7
Ecge Doletion (S)
52.2
vergence in behavior between the two methods (Find-
Prompting
ing 77 This divergence on photo is expected given the
52.2
638
61.6
52.5
615
dataset's characteristics. Photo is highly homophilic and contains nodes with rich, lengthy feature descriptions. In
60
20.3
24.3
30.0
65.9
09.3
74.1
homophilic graphs, nodes are densely connected to others of the same class, so 2-hop prompts accumulate substan- tial amount of feature text from numerous neighbors. This quickly exceeds the LLM's is context window, leading to
Hean Accuracy [S)
11.6
14.4
13.8
27.9
47
81.0
50
90
100
-5
100
72.0
67.0
61.8
57.2
51.2
40.3
significant drop in accuracy for Prompting. In contrast, Graph-as- Code (dende) is designed to selectively retrieve and compose only the necessary structure and features for each query. This allows it to avoid exceeding the token
40
Graph-as as Code
90-
Edpe. Dotetton
83.9
72.3
65.5
65.6
58
58.4
30
50-
85.2
70.4
68.8
67.1
66.5
limit and maintain high accuracy, even in the presence of long node descriptions and dense connectivity.
68.2
68.1
20
26.4
65.6
68.7
00.4
68.5
71.0
This result ties back to our long-text homophilic exper- iments and findings in Section 4.3] There we observed a large gap in favor of Graph- as Code (dewde); the photo ablation reveals the same characteristic: Prompting is fundamentally bottlenecked by the context window and can even benefit from discarding feature text to fit within it, whereas Graph-as-Code (Reode) retrieves and composes the needed structure and features without exceeding the token budget. Thus, the two methods share the same dependence on features and structure when prompts fit the context window, but once the token limit becomes a limiting factor, their behaviors diverge and Graph-as-Code proves superior. Practitioners should therefore assess the graph density and average feature length before choosing an LLM-graph interaction mode, prioritizing adaptive methods such as Sraph-as-Codd for cases with high density or long-text features.
50
Featuure Delation (%6
Accuracy of 2-hop prompt- ing and Graph-as-Code on the photo dataset under varying ratios of randomly removed edges and truncated features.
Figure 3:
5.2 LABELS VS. STRUCTURE DEPENDENCIES
Finding 8. Prompting and Graph-as-Con Code exhibit different dependencies on labels and structure.
Figure 4 demonstrates a stark contrast between the dependence patterns of Prompting (dponpp) and Graph-as- Capphe (ounte) when subjected to edge and label deletion (Finding (). This behavior is in contrast to the alignment on feature and structure dependencies observed in Section 5.1
8

---

Published as a conference paper at ICLR 2026
COT3
cornell
100
25.4
26.3
22.7
23.4
22.3
21.7
23.3
23.0
72.
79-3
D0.7
Prompting Edge Deletion
51.3
52.1
18.0
31.9
75
32.8
57.
53.8
25.8
75
77.7
81
795
79.9
50
71.2
20.0
85.5
53.6
48.1
29.5
50
07.2
07.4
08.
016
31.2
25.7
53
00.9
81
62.5
00
82.6
823
77.7
78.3
16.1
43.0
25
7..
71.3
03.3
01.7
15
72.
25
50
90
100
25
50
100
50
75
90
100
102
Edga Doletion [88]
90
7322
762
72.4
66.
51.3
53
90
72.2
72-1
70.6
70.4
68.5
66.9
92
92.9
91.
52.5
90.8
91.
Graph-as-Cool
76.9
76.5
38.5
10.1
66.1
431
75
73.7
72.0
71.3
21.8
68.5
21.1
932
80
60.2
91 91.5
61.
77.1
45.4
73.5
92.2
91.5
00.7
25
731
85.2
85.2
85.1
17.4
45.7
46.0
16.4
74.4
73.3
75.1
76.3
69-4
2 92.7 90.6
52.5
90.3
96.1
90.1
25
50
90
100
30
100
75
90 100
Lobel Deletion 196]
Label Deletion (96)
Label Detetion 191
Figure 4: Accuracy of 2-hop prompting and Graph-as- Codpe on the cora, arxiv, and cornell datasets under varying ratios of randomly removed edges and known labels.
Finding 9. Graph-as- Craph-as-Code is able to flexibly shift its reliance between structure, features, and labels leverage the most informative input rype.
For Prompting (pprompi), the results across all datasets reveal that accuracy degrades rapidly along both axes, confirming that Prompting requires both structure and labels to perform reasonably. Graph-as- Code (doode) by comparison, displays a notably different pattern. Its accuracy remains nearly constant as edges are deleted, as long as either features or labels are present. This might suggest that Graph-as- Code ignores structure entirely; however, as previously observed in Section 5.1 structural information becomes crucial only when features are truncated. Thus, Graph-as Traph-as-Code Code does not disregard structure, but instead leverages it only when it is more informative relative to other available signals.
This leads to a key insight: Graph-as Graph-as-Code (dcote) can flexibly shift its reliance to the most informative input type, and is thus only vulnerable when multiple sources of information heavily degraded (Finding 19). This adaptive behavior contrasts sharply with the brittleness of Prompting and re-emphasizes the robustness of Graph-as-Code (Finding(6).
CONCLUSIONS
In this work, we conducted the first comprehensive, controlled evaluation of LLMs for node classi- fication across key of variability: L.LM-graph interaction mode (Prompting, ReAct-style tool use, and Graph-as-Code) dataset domain (citation, web-link, e-commerce, social), structural regimes (homophilic vs. heterophilic), textual feature characteristics (short vs. long), model size (from small to large) and reasoning capabilities. Our large-scale study reveals that the Graph-as-Code method, which leverages LLMs' coding capabilities, achieves the strongest overall performance especially on graphs with long textual features or high-degree nodes, where the widely-used prompting method quickly becomes infeasible due to context window limitations. We also find that all LLM-graph inter- action methods remain effective on heterophilic graphs, challenging the commonly held assumption that LLM- methods fail in low- wwwhhompphlly settings (Huang et al. 2024a).
Through series of controlled dependency analyses, we independently truncate features, delete edges, and remove labels to quantify reliance on different input types. Experiments show that Caaphaa--Code flexibly adapts its reliance to the most informative signal, be it structure, features, labels.
Our findings provide actionable guidance for both practitioners and researchers: (1) Code generation is the preferred LLM-graph interaction mode, particularly as graphs grow in size and complexity; (2) LLMs remain effective on heterophilic graphs; (3) Graph-as-Code 's adaptive reliance can be used to robustly handle noisy or partially missing data, where different input signals may be degraded.

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work studies how large language models interact with graph-structured, text-rich data across prompting, tool-use, and code-generation modes. The work focuses on the node classification task in domains such as citation, web-link, e-commerce, and social networks. Potential benefits include safer fraud detection, improved recommendation, and better information retrieval; there are no potential risks. Our study uses established benchmark datasets and does not involve new human subjects data collection. Where social or user-generated content present benchmarks, we follow dataset licenses and use standard train/validation/tess protocols; labels for held-out nodes are never revealed at inference time in our setups. We report results averaged over multiple seeds and include ablations to surface model dependencies on structure, features, and labels, which can help practitioners assess failure modes before deployment. We will comply with the ICLR Code of Ethics
REPRODUCIBILITY STATEMENT
To facilitate reproduction: (I) we specify all interaction templates used in our experiments (Prompting, GraphTool+, and Graph-as-Code) in the appendices, along with detailed task instructions and action formats. (2) We document data sources, domains, homophily levels, average degrees and text lengths. (3) We specify the model types used, along with their reasoning or non -eaasonnning variants. (4) We provide full detail over the evaluation procedures including per-seed sampling.
REFERENCES
Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of lims fall short?, 2024.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, et Graph of thoughts: Solving elaborate problems with large language models. In AAAI, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Qiaolong Cai, Zhaowei Wang, Shizhe Diao, James Kwok, and Yangqiu Song. Codegraph: Enhancing graph reasoning of Ilms with code. arXiv, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv, 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv, 2021b.
Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. In ICML, 2024.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, arXiv, 2023.
Antonia Creswell and Murray Shanahan. Selection-inference: Exploiting large language models for interpretable reasoning. In EMNLP, 2022.
Xinnan Dai, Haohao Qu, Yifen Shen, Bohang Zhang, Qihao Wen, Wengi Fan, Dongsheng Li, Jiliang Tang, and Caihua Shan. How do large language models understand graph patterns? a benchmark for graph pattern comprehension. In ICLR, 2025.
10

---

Published as a conference paper at ICLR 2026
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach query query-focused focused summarization. arXiv, 2024.
Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. In ICI.R, 2024.
Ben Finkelshtein, Xingyue Huang, Michael Bronstein, and ismail ilkan Ceylan. Cooperative graph neural networks. In ICML, 2024.
Ben Finkelshtein, ismail Ilkan Ceylan, Michael Bronstein, and Ron Levie. Equivariance everywhere all at once: A recipe for graph foundation models. arXiv, 2025.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In ICMI., 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv, 2023.
Zhong Guan, Hongke Zhao, Likang Wu, Ming He, and Jianpin Fan. Langtopo: Aligning language descriptions of graphs with tokenized topological modeling, 2024.
Zhong Guan, Likang Wu, Hongke Zhao, Ming He, and Jiampin Fan. Attention mechanisms perspec- tive: Exploring IIm processing of graph-structured data. In ICML., 2025.
Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. In NeurIPS, 2023.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurlPS, 2017.
Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Harness- ing explanations: Lim-to-Im interpreter for enhanced eext-attributed graph representation learning. In ICLR, 2024.
Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. Unigraph: Learning a unified cross-domain foundation model for texi-attributed graphs. In KDD, 2025.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS, 2020.
Jin Huang, Xingjian Zhang, Qiaozhu Mei, and Jiaqi Ma. Can ilms effectively leverage graph structural information through prompts, and why? In TMLR, 2024a.
Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang. Lifeng Shang, et al. Planning, creation, usage: Benchmarking Ilms for comprehensive tool utilization in real-world complex scenarios. arXiv, 2024b.
Thomas Kipf and Max Welling. Semi Semi-supervised classification with graph convolutional networks. In ICLR, 2017a.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017b.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv, 2023.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long in-context learning. In TMI., 2024a.
11

---

Published as a conference paper at ICLR 2026
Xin Li, Weize Chen, Qizhi Chu, Haopeng Li, Zhaojun Sun, Ran Li, Chen Qian, Yiwei Wei, Chuan Shi, Zhiyuan Liu, et al. Can large language models analyze graphs like professionals? benchmark, datasets and models. In NeurlPS, 2024b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. AAAS, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv, 2023.
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Shieh, and Wenmeng Zhou. Codexgraph: Bridging large language models and code repositories via code graph databases. In ACL, 2025.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chur Zhu, and Jianfeng Gao. Chameleon: Plug-ane Blug-and-ple and--lay compositional reasoning with large language models. NeurlPS, 2023.
Peter Mernyei and Cätälina Cangea. Wiki-es: A wikipedia- benchmark for graph neural networks. arXiv, 2020.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. NewrIPS, 2024.
Hongbin Pei, Bingzhe Wei, Kevin C.-C. Chang, Yizhou Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In ICLR, 2020.
Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami A1-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for llms. arXiv, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In EMNLP, 2020.
Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Mathhis Fey, Jan F. Lenssen, Yiwen Yuan, Zecheng Zhang, Xinwei He, and Jure Leskovec. Relbench: A benchmark for deep learning on relational databases, 2024.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda. and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In NeurlPS, 2023.
Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning. arXiv, 2025,
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günneeann. Pitfalls of graph neural network evaluation, 2019.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. NeurIPS, 2023.
Weihang Su, Qingyao Ai, Xiangsheng Li, Jia Chen, Yiqun Liu, Xiaolong Wu, and Shengluan Hou. Wikiformer: Pre-training with structured information of wikipedia for ad-hoc retrieval, 2024.
Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. SIGIR, 2024.
Jianheng Tang, Qifan Zhang, Yuhan Li, Nuo Chen, and Jia Li. Grapharena: Evaluating and exploring large language models on graph computation. ICLR, 2025.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
12

---

Published as a conference paper at ICLR 2026
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? NeurIPS, 2024.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. shain-of-thougee prompting elicits reasoning in large language models, In NeurIPS, 2022.
Xixi Wu, Yifei Shen, Fangzhou Ge, Caihua Shan, Yizhu Jiao, Xiangguo Sun, and Hong Cheng. When do Ilms help with node classification? a comprehensive analysis. In /CML, 2025.
Chengdong Yang, Hongrui Liu, Daixin Wang, Zhiqiang Zhang, Cheng Yang. and Chuan Shi. Flag: Fraud detection with llm-enhanced graph neural network. In SIGKDD, 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Prithviraj Ammanabrolu, Matthew Hausknecht, and Karthik Narasimhan. React: Synergizing reasoning and acting in language models. arXiv, 2023.
Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Language is all graph needs. In ACL, 2024.
Jiawei Zhang. Graph-toolformer: To empower Ilms with graph reasoning ability via prompt aug- mented by chatgpt. arXiv, 2023.
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael M Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph learning in text space. arXiv, 2024.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang. Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking humaneval-x. In SIGKDD, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Swaroop Mishra, Alexis Nguyen, Christopher Potts, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. In NeurIPS, 2022.
Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Hua- jun Chen, and Ningyu Zhang. Lims for knowledge graph construction and reasoning: Recent capabilities and future opportunities. In arXiv, 2023.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORKS
LLMs coding capabilities. Code generation has become native capability of modern LLMs, with early systems demonstrating high-quality programs from natural language prompts (Chen et al. 2021b), even reaching competition-level performance (Li al. 2022) and fueling the development of strong open-source models such as StarCoder (Li et al. 2023) and CodeGeeX (Zheng et al. 2023). Beyond pure code synthesis, program-aided approaches have leveraged code as a medium for reasoning- geasoning-enabliing LLMs to generate and execute short programs for mathematical problem solving (Gao al., 2023), automate scientific workflows (Seo et al. 2025), and enhance reliability through self-debugging and iterative refinement (Chen et al. [023]. Recently, these capabilities have been extended graph domains, where LLMs are prompted to generate code for sol solving simple graph problems such as cycle detection, connectivity analysis, and node degree estimation (Cai et al.) 2024). We extend this further with Graph-as-Code the LLM iteratively writes and executes concise programs over standardized API to flexibly combine structural queries, label propagation, and textual feature processing. We then compare Graph-as-Code to prompting and tool-calling across datasets and asses its dependencies on the input feature, structure and labels.
Hybrid LLM-GNN architectures. Beyond textualization, recent works have proposed hybrid frameworks that integrate Graph Neural Networks (GNNs) trainable projectors directly into the LLM architecture. Approaches closer to textualization, such as LLaGA (Chen et al.) (024), serial- ize each node's s neighborhood into structure- aware sequences; however, instead of using raw text, these sequences are mapped via a trained linear projector into the LLM's input embedding space, enabling the processing of graph features as soft tokens. Other frameworks focus on explicit modality alignment. LangTopo (Guan et al. 2024) employs GNN to map textual and spatial information into a discrete codebook of topological embeddings via Gumbel-softmas relaxation. maximizes consistency between the L.M''s natural language representations and these quantized codes, effec- tively transferring structural modeling capabilities the LLM. Alternatively, cascaded architectures like UniGraph (He et al. 2025) directly fuse inputs by generating dense vector embeddings with pre-trained GNN and concatenating them with the discrete token embeddings natural language instructions. This combined sequence is fed into an LLM instruction-tuned to unify label spaces. While these hybrid methods rely on training auxiliary encoders or fine- tuning to fuse modalities, our work investigates the capabilities of frozen LLMs, evaluating how they be orchestrated to reason over graphs at inference time through prompting, tools, and code generation.
B ADDITIONAL AXES OF VARIABILITY
In this section, we present experiments and findings across additional axes of variability. In Section B.I and Section B..2] we analyze the impact of LLM size and reasoning, respectively, on accuracy across various graph domains, homophily levels, and feature text lengths.
B. LLM SIZES
Finding 10. Larger LLMS consistently deliver better performance.
Table 4 demonstrate clear and consistent trend from Llama o4-mini, DeepSeek R1, and GPT-5: 5: increasing LLM size yields improved node classification performance (Finding 10). This holds across all datasets and both interaction modes. This result is natural as larger LLMs possess greater capacity to model complex relationships and capture subtle dependencies between textual features and graph structure (Dai al. (023). As a results. the performance gap between Prompting and Graph-as-Code also persists at each model size, strengthening earlier findings from both short-text and long-text datasets that Graph-as- as-pee consistently provide an advantage (Findings and
We also note that the incremental improvements between 04-mini, DeepSeek R1, and GPT-5 are less dramatic than the jump from Llama, suggesting diminishing returns at the upper end, but the trend is robust. This aligns with the broader literature LLM scaling laws, where larger models better capture complex dependencies.
Finding 11. Context window token limits consistently constrain the performance of Prompting, regardless of L.L size.
14

---

Published as a conference paper at ICLR 2026
Table 4: Accuracy of LLM-graph interaction modes Prompting and Graph-as-Code across L.MSs of
varying sizes.
Dataset cora
Llama
04-mini
DeepSeek R1
GPT-5
72.17=2.07
85.16++1.47
86.20+1.110
87.10+0..0
arxiv
62.80-2.20
74.40+:3.02
75.30+2.50
76.20+22.00
cornell
80.05=1.90
84.17
85.50=11.20
86.20
TokenLimit
TokenLimit
TokenLimit
TokenLimit
2-hop prompt
citeseer
14.10=4.55
69,40=2.10
71.00=1.90
72.20+1.70
photo
TokenLimit
TokenLimit
TokenLimit
TokenLimit
wiki-cs
TokenLimit 57.28 2.68
TokenLimit
TokenLimit
TokenLimit
average
78.78+22.01
19.75
80.4
cora
75.23 2.00
85.16+1.447
86.50++0.95
87.70-H0..0
arxiv
65.60 2.30
74.40+3..0
75.60+2.110
76.50+1.80
cornell
86.00-2.10
92.70+#2.35
93.10:2..00
93.80+1.80
texas
68.00-2.50
73.6013.78
75.1042.60
76.0012.400
Graph-as-Code
citeseer
16.30-4.00
71.80+2.22
12.80+1.95
73.60+
photo
83.69-2.59
86.40+2.65
87.00+2.550
87.60
wiki-es
79.76-2.94
82.20-13.63
83.10
83.80+2.90
average
67.80-2.35
80.75+2.700
81.89+2..17
82.71
Table Accuracy of LLM-graph interaction strategies Prompting and Graph-as-Code across LLMs with reasoning and non-reasoning modes.
Phi-4
Qwen
Dataset
W.O. reasoning
reasoning
w.o. reasoning
reasoning
cora
10.16=11.00
8..3
80.72
88.05+1.255
arxiv
69.40+22.80
75.83+2.32
70.10+2.77
77.11
cornell
79.17-
85.09+1..35
79.62-t
8..01
texas
TokenL.imit
TokenL.mmit
TokenI.mmit
TokenI.mmit
2-hop prompt
citeseer
64.40+2.10
7..84+1.90
64.99-2.005
7..42+1.80
photo
TokenLimit
TokenLimit
TokenLimit
TokenLimit
wiki-cs
TokenLimit
TokenLimit
TokenLimit
TokenLimit
average
73.28+11.98
79.29+11772
73.36+1.95
80.15+1.60
cora
81.08+1.477
19.12+1.10
81.67+1.52
89.83+11008
arxiv
71.40+3..0
78.5442..0
72.01
79.21
cornell
87.70+22.35
96.47+2.00
88.19+2..0
97.01
Graph-as-Code
texas
60.40++3.10
71.62++2.77
65.18+3.005
72.29
citeseer
66.80+2.22
74.48+2.00
67.3942.14
15.18+1.90
photo
82.08-12.65
90.29+12.00
82.58-2..0
91.0942.188
wiki-cs
78.20-13.63
85.8143.10
78.6543.59
86.5443.06
average
75.38+22.2
83.19+2.10
75.67+2.39
83.45+2.04
A notable secondary observation is that token limits remain bottleneck for Prompting in several datasets (texas, photo, wiki-cs), regardless of LLM size (Finding []]. This expected as context window limitations are fundamental property of LLM architectures no matter how large the model, there a maximum input length that cannot be exceeded. As neighborhood size or feature length increases, the input quickly outgrows this limit, preventing the LLM from accessing all relevant context. This limitation highlights a fundamental constraint of the Prompting strategy: as neighhorhood or feature size grows, even large models cannot circumvent context window restrictions.
15

---

Published as a conference paper at ICLR 2026
B.2 REASONING CAPABILITIES
Finding 12. Reasoning consistently improves performance.
Table reveells a consistent and substantial boost in accuracy when reasoning modes are enabled for both Phi-4 and Owen LLMs, regardless of the interaction strategy used. This pattern expected, as reasoning has been shown to elevate LLM performance across various domains such as question answering, knowledge graph completion, and complex recision-making (Wei et al. 2022, Creswell & Shanahan, [022] Zhou et al., 2.22). It encourages the LLM to engage in step-by-stee inference, explanation, and synthesis - capabilities that are crucial for effectively leveraging both node attributes and graph structure. For example, reasoning can help the LLM to better contextualize information from multi-hop hop neighborhoods and to integrate evidence from various sources, reducing the risk of overlooking important relationships introducing spurious correlations.
C TOKEN AND LATENCY EFFICIENCY
In this section, we analyze the practical overhead of the evaluated LLM-graph interaction modes. Specifically, assess token consumption in Section C.I and computational latency in Section C.2 to highlight the real-world deployment trade-offs between the different interaction modes.
C.I TOKEN CONSUMPTION
To assess the efficiency and scalability of the evaluated interaction strategies, report the average token consumption per query on long-text datasets in Table These metrics illustrate why context window limits critical bottleneck for prompting variants.
Table Average token counts per query for each L..M-graph interaction modes Prompting, Graph- Tool and Graph-as-Code over long-text datasets.
Dataset 0-hop prompt
I-hop prompt
2-hop prompt
GraphTool
Graph-as-Code
reddit
1,762
10,300
42,100
16,100
14,900
photo 3,000
36,100
104,700
43,300
41,500
wiki-cs 30,200
202,500
1,157,000
61,700
57,100
As demonstrated in Table the total token count increases significantly from reddit photo, and again to wiki-cs. This trend is expected, as it mirrors the increase in the average text length and average node degree for these datasets. Consequently, 1-hop and 2-hop prompts quickly exceed the context window of 200, 000 tokens for our default o4 minnn model forcing truncation. In contrast, GraphToo Tool and Graph-as- Code use much smaller token counts, as they only retrieve specific pieces of information (e.g., list of neighbor IDs or single node's features) and build context iteratively.
C.2 COMPUTATIONAL EFFICIENCY
We assess the computational efficiency of the evaluated interaction strategies for practical use by measuring the average wall-clock time per query on the long-text datasets reddit, photo, and wiki-es. This latency is measured from the submission of the initial prompt until the LLM yields its final prediction, with results averaged across evaluated nodes.
Table 7: Average latency (wall-clock time in seconds) per query across for each LLM graph interac- tion modes over long-tes text datasets.
Dataset 2-hop prompt
Graph
Graph-as-Code
reddit
41
45
43
photo
125
131
128
wiki-cs
TokenLimit
159
155
16

---

Published as a conference paper at ICLR 2026
The results in Table 7 indicate that the primary driver of latency is the dataset's characteristies (e.g.. text length, graph density) rather than the interaction mode itself. This trend clear across the datasets: wiki-cs, which has the longest text features and highest average degree, shows the highest latency. This is followed by photo, which is also text-rich and has a relatively high degree. Reddit, in contrast, has the shortest text and lowest average degree of the three, which results in the lowest latency. This is because processing longer text and larger, more complex neighborhoods simply requires more overall computation from the LLM, regardless of the method.
The modes are all dominated by the LLM's inference time, which is why their latencies are so comparable. While GraphTool+ and Graph-as-Code Code involve multiple iterative calls, each call is very "lightweight" (e.g., retrieving only neighbor IDs or a single feature). In contrast, Prompting makes a single, "heavy" call with a massive context. In practice, these different approaches balance out: the to- tal computational load of many small calls becomes comparable to that of one large, token-heavy call.
D ADDITIONAL EXPERIMENTS
In this section, we explore the sensitivity of Prompting to neighborhood depth (Section D.I. asses the impact of code-generation priors on the varying interaction modes (Section D.2). extend our evaluation to algorithmic graph reasoning tasks (Section D.3 and compare our zero-shot interaction modes against supervised baseline architectures (Section D.4.
D.1 THE IMPACT OF THE NUMBER OF HOPS ON PROMPTING
A potential consideration when comparing Prompting (Oprope) to Graph-as-Code (dcode) is the number of hops accessed. Prompting is fundamentally constrained by token limits, which upper-bounds the number of hops that can be included in the context window. In contrast, ReAct-based methods can dynamically query the graph, adjusting the exploration depth on a per-node basis to effectively manage the token budget. This varying resource exposure is not confounding variable, but rather a fundamental, defining property that differentiates these interaction modes in practice. The strict context boundary is a primary real-world limitation of Prompting, while the capacity for efficient programmatic exploration defines Graph-as CophhaasCode
Nevertheless, to provide a rigorous comparison, we introduce an iterative k-hop summary prompt variant. This approach utilizes an iterative and recursive summarization process to compress infor- mation from distant hops, enabling its inclusion without exceeding the token limit. For example, to construct a 3-hop summary prompt, the LLM first summarizes the immediate neighborhood of each 2-hop node (i.e., the 3-hop nodes). Next, it summarizes the neighborhood neighborhood of each 1 1-hop node, where 2-hop nodes are now represented by their previously generated summaries. The final prompt for the target query node thus includes the immediate neighborhood - containing these rich, summarized I-hop neighbors, which carry compressed representations of the 2-hop and 3-hop topology and features. This allows the model to access 3-hop information that we could not have included before due to token limits.
We evaluate 2-hop. 3-hop, and 4-hop summary prompts on the long-tex: text datasets in Table g Execution logs confirm that our ReAct-based modes do not query beyond 4 hops these specific datasets, ensuring that evaluating up to 4-hop summaries provides a fair comparison of maximum resource exposure.
Finding 13. Iterative neighborhood sunmarization hurts performance beyond certain number of hops.
The results in Table 8 yield several key insights. First, the 2-hop summary prompt consistently outperforms both the T-hop prompt and the 2-hop budget prompt, establishing as much stronger and more token-efficient prompting baseline.
Second, we observe a clear trend of diminishing, and even negative, returns as we extend this summarization to more distant hops. Performance peaks the 2-hop summary and then slightly declines with the 3-hop summary and 4-hop summary (Finding 13). Since we know adding hops is beneficial in these homophilic datasets, this decline indicates that the iterative summarization process is lossy. Each summary of a summary compounds the information loss, eventually outweighing the benefit of including more distant nodes.
17

---

Published as a conference paper at ICLR 2026
Table 8: Accuracy of M.---grphh interaction modes Prompting, Summary Prompting, GraphTool, and Graph-as-Code on long-text datasets. Best per-dataset results are bold, runner-up underlined.
citeseer
reddit
computer
photo
instagram
wiki-cs
O-hop prompt
68.20
47.90
65.50
69.80
48.00
74.00
1-hop prompt
68.30
59.30
86.10
85.80
56,10
Token Limit
2-hop prompt
69.40
TokenLimit
TokenLimit 86.00
TokenLimit 85.60
TokenLimit 54.50
TokenLimit
2-hop budget prompt
70.20
54.40
80.80
2-hop summary prompt
70.40
57.60
86,20
86,00
55.70
81.70
3-hop summary prompt
69.80
57.00
85.80
85.60
55.80
81.00
4-hop summary prompt
69.60
55.90
86.00
85.40
55.60
80.90
GraphTool Graph-as- -sdde
68.30
56,25
80.80
77.00
47.80
76.27
71.80
61.60
86.20
86.40
56.40
82.20
Finding 14. Graph-as-Code oulperforms multi-hop summarization prompting by adaptively retriev- ing only the necessary information.
Finally, even these stronger, multi-hop summary baselines are still outperformed by Graph-as-Code This reinforces Finding 4 confirming that the superiority of Graph-as-Cods is not merely byproduct of "seeing more" information. Instead, Graph-as-Code adaptively and surgically retrieves the optimal information required for reasoning (Finding 14). For instance, the programmatic execution allows the model to retrieve only the labels of 2-hop neighbors (which are token-cheap) while ignoring their less informative features, or vere-versa. Prompting, even with iterative summarization, remains non- adaptive, pre-computed information dump. We thus conclude that Sraph-as-Code programmatic, iterative approach allows to execute more flexible and efficient reasoning strategy, leading to its superior performance.
D.2 HAPH-AS-CCDE'S RELIANCE ON STRUCTURE VS.
PRIORS
We verify that the strong performance of Graph-as-Code (ó ) is not merely an artifact of code- generation priors with control experiment using randomly shuffled adjacency matrices. This process preserves all original node features and the graph's 's degree distribution, but destroys all meaningful structural patterns. repeat our evaluations across representative datasets and report the average accuracy over 10 independent random adjacency permutations.
Table 9: Accuracy of baselines and LLM-graph interaction modes Prompting, GraphTool and Graph-
as-Code datasets with randomly shuffled adjacency matrices.
cora
arxiv
cornell
texas
citeseer
photo
wiki-cs
Random
14.13
2.44
21.74
8.40
16.80
8.20
9.90
Label propagation
13.29
3.85
30.48
10.34
13.10
8.40
9.60
O-hop prompt
64.21
68.10
81.57
53.20
68.20
69.80
74.00
--hop prompt
58.10
61.50
74.30
47.10
60.30
63.20
Tokenl limit
2-hop prompt
44.50
47.20
60.10
Tokenl Limit
45.10
TokenI.imi
TokenI Liket
GraphTool Graph-as-Code 56.50
56.20
59.80
72.40
45,90
58.10
61.10
66.20
59.10
72.90
45.20
58.80
60.90
66.70
As shown in Table 9 the performance of all structure- methods significantly decreases when compared to their original results Tables to This confirms that they are attempting to use the neighborhood information, which now only adds noise. These additional trends arise:
The O-hop prompt's 's performance unaffected, since never queries the neighborhood. thus immune to structural noise and becomes the best-performing baseline.
2. Label Propagation is on par with a random guess. This is expected, as it relies exclusively the meaninglees structural information.
18

---

Published as a conference paper at ICLR 2026
3. Performance degradation scales with the amount of noise introduced. 2-hop prompting is impacted more severely compared to 1-hop prompting, as it ingests the largest volume of noisy neighborhood context.
4. GraphTool and Graph-as- prove more robust than the naive 2-hop prompt. This suggests that while the LLM's reasoning is confused by the noise, can still mitigate the noise better than a simple prompt expansion, likely by weighing its own (correct) features more heavily than its neighbors' (noisy) information.
These results strongly indicate that the superior performance of Graph-as-Code and other interaction modes seen in our main paper is not an artifact of code-generation priors, but direct result of effectively utilizing the true graph structure. This experiment also reinforces Finding 6 While 100% random noise hurts all methods, Graph-as- Craph-as-Code as-Code's performance degrades less severely than the 2-hop prompt. demonstrating superior ability to handle noisy or irrelevant structural information, even in this extreme scenario.
D.3 ALGORITHMIC REASONING EVALUATION
We test algorithmic reasoning capabilities, with an additional experiment on synthetic shortest path prediction task.
Synthetic dataset generation. We generated dataset of 100 synthetic Erdös-Rényi graphs, each comprising 50 nodes, with the edge probability chosen to ensure full connectivity. For each graph, we randomly sampled 100 source-targe node pairs. The regression task is to predict the integer length of the shortest path (le) between the two nodes. This setup allows us to assess predictive performance as direct function of problem complexity, represented here by the true path length k. We report the Mean Squared Error (MSE) for 2-hop Prompting, GraphTool+, and Graph-as-Code across varying path lengths in Table TO
Table 10: Mean Squared Error (MSE) of LLM LLM-graph interaction modes Prompting, GraphTool and Graph-as-Code on the synthetic shortest path prediction task across varying true path lengths (k))
Method
k ==1 1
k = 2
k
k:
4
k: 2 1
2-hop prompt
0.00
0.00
2.02
2.29
2.05
2.82
GrophTool+
0.00
0.00
0.74
2.46
2.37
2.13
Graph-as-Coon Code
0.00
0.00
0.00
0.00
0.00
0.00
As shown in Table 10 the 2-hop Prompting method achieves a perfect prediction for path lengths = and h: = 2. This is expected, as the target node naturally falls within the 2-hop neighborhood provided in the static prompt, allowing the LLM to simply "find" the answer in its context. However, for any path length & 2 3. the target lies outside the prompt's 's fixed context boundary. Lacking any mechanism to explore further, the LLM is forced guess, and its MSE increases dramatically. This confirms that static prompting is incapable of the iterative computation needed for algorithms.
Similarly, GraphTool+ performs perfectly for path lengths of & 1 and k = 2. However, its performance begins to degrade
and becomes highly error-prone for longer paths, with the MSE jumping to 2.46 at A = 4. This indicates that while GraphTTool+ can attempt to simulate Breadth-First Search (BFS) by iteratively traversing the graph via tool calls, the computational space of possible paths explodes length increases, making the LLM highly error-prone expected.
Finally, Graph-as-Code (dcode) achieves perfect prediction across all evaluated path lengths. This flawless performance occurs for a simple reason: the LLM does not need to internally simulate the path-finding algorithm step-by-step. Instead, it leverages its code-generation capabilities to write standard BFS execution code, which the environment runs deterministically. This experiment demonstrates that for pure algorithmic reasoning, delegating computation to generated code as in Graph-as-Code is vastly superior to other interaction modes.
19

---

Published as a conference paper at ICLR 2026
D.4 COMPARISON WITH SUPERVISED ARCHITECTURES
While the primary focus of our paper evaluating capabilities of LLMs inference time, re- mains meaningful to contextualize their performance against state-of-the-art supervised architectures. To this end, wc extend our evaluation on long-text datasets to compare our zero-shot interaction modes against end-to-end GNNs GCN (Kipf & Welling 2017b). GAT (Veličković et al. (0188) and hybrid LLM+GNN models- Encoder GCN (Wu et al. 2025), TAPE (He et al. 2024). LLaGA (Chen All GNN and hybrid LLM+GNN results were taken
et al. 2024). , and UniGraph (He et al.
2025
from
Wu et al. (2025).
Table 11: Accuracy comparison between classic GNNs, hybrid LLM+GNNs, and our evaluated
LLM-graph interaction modes Prompting, GraphTool and Graph-as-Code long-text datasets.
Method Type
Model
citeseer
reddit
computer
photo
instagram
GCN
70.55
61.44
71.44
69.25
63.50
Classic GNN
GAT
69.94
60.60
83.39
80.40
63.56
Encoder GCN
71.39
68.65
88.22
84.84
67.49
TAPE
71.87
62.43
89.52
86.46
66.07
Hybrid LLM+GNN
L.LGA
74.15
69.20
88.80
87.10
68.20
UniGraph
72.50
68.80
88.30
85,90
67.60
O-hop prompt
68.20
47.90
65.50
69.80
48.00
1-hop prompt
68.30
59.30
86.10
85.80
$6.10
LLM-Only
2-hop budget prompt
70.20
54.40
86.00
85.60
54.50
GraphTool
68.70
61.80
83.10
81.30
48.20
Graph-as Credh-as-Code
71.80
61.60
86.20
86.40
56.40
This comparison shows that hybrid LLM+GNN models generally outperform both classic GNNs and LLM-only methods. We hypothesize this is because they effectively combine the distinct strengths of each architecture: Classic GNNs specialized for structural processing and operate shallow, low-dimensional text features (e.g., bag-of-words, TF-IDF); LLM-Only methods excel at deeply understanding raw, rich text but are not specialized for graph-based message passing.
E AN EXAMPLE OF GRAPH-AS- -CODE REASONING
To demonstrate how the Graph-as-Code (doode) mode operates in practice, we present representative execution log in Example [ The log illustrates the practical, step-by-step reasoning process of the Graph-as-Cove -Cdee mode. shows the agent forming a hypothesis from its own features (Class 'Reinforcement Learning') and then programmatically testing that hypothesis against its neighborhood even noting the correlation between Class and Class 5. This demonstrates a more complex and interactive reasoning process than in standard static prompting.
20

---

Published as a conference paper at ICLR 2026
Example Graph-as- Codde execution log for node 133 of the cora dataset
You are solving node-based reasoning task for node 133, You have pandas DataFrame df where each corresponds to node, indexed by its hode_id
Task:
Instructions: Always begin with reasoning. After your reasoning, provide single, valid pandas command on new line.
Schema structure:
The DataFrame index is the node id: df. loc [node_id].
Textual description:
Loo inde _d, 'features'
List neighbor node IDs: df 1oc node id, "neighbors'
Integer node label (if in training set, otherwise None): df Loc.llllne. node id, "abell"
Available class labels:
0: Rule Learning,
Neural Networks,
2: Case Based,
3: Genetic Algorithms,
4: Thecry,
5: Reinforcement Learning,
6: Probabilistic Methods
Request 1: need classify 133. will start by inspecting its features and label.
df.loc [133] [[" features', "label']]
Response 1:
features: 'A Reinforcement learniig Approach to Shop Schedul ing'
1abel: None
Request 2: The label is unknown. The features strongly suggest 'Reinforcement Learning' (class 5), I will check the labels of its neighbors confirm this. df lloo [af, loc (133, "neighbors' 'label'1
Response 2:
node_1s id
label
134
5.0
135
5.0
707
5.0
1588
5.0
1589
NaN
2048
1.0
Request 3: I have 5 neighbors with known labels. Four are class 5 ("Reinforcement Learning") and one is class ('Neural Networks'). The majority class among neighbors is 5. The presence of class 1 1 ('Neural Networks') also supports this, reinforcement learning is highly correlated with neural networks. This neighborhood evidence strongly matches my initial hypothesis from the node's own features. confident in this prediction.
Answer: 5
F DETAILS ON LLM-GRAPH INTERACTION MODES
In this section, we provide the full prompt templates used for each LLM-graph interaction mode evaluated in our experiments. These templates specify the instructions, available actions, and response formats provided to the LLM during node classification tasks. Specifically, include the exact text used for the Prompting, GraphTool+, and Graph-as-Con modes, in Templates respectively, enabling reproducibility and facilitating future methodological comparisons.
21

---

Published as a conference paper at ICLR 2026
Template
Prompting
Task: You are solving node-based task. Your goal is to determine the label for node (node_id); The final answer must be submitted as integer corresponding class label. Below is the mapping from each integer index its associated label.
Available class labels:
0: (text description on of label 0)
1:
(text description of label 1
Node (noce_io has textual description (feat_id and belongs to label class (Label id) (or None).
Node [node_id] has the following neighbors 1-hop away:
Node (n_1) has the textual description [feat_n]} and belongs to label class (label_n1) (or None).
Node (n_2) has the textual description (feat_pt and belongs to label class (label_ng) [2] (or None).
Node {node_id) has the following neighbors 2-hops away: Node (m_1) has the textual description [feat_ml) and belongs to label class (label_mi (or None).
Think and end your response with: Answer:
[class_id]]
Template 2 GraphTool++
Task:
You are solving node-based reasoning task using interleaved steps. Your goal is to determine the label for node (node id). ). At each step. you may choose of several available actions to gather information submit your final prediction.
Always begin with reasoning. You may take as many steps as needed, but aim to solve the task efficiently using the fewest necessary actions. Before cach action, assess what information is available, what's 'ss missing, which action is most appropriate next, and how many steps likely remain. Then, new line, specify your chosen action using one of the formats below. It must be the final non-empty line of your response.
Instructions:
Available actions:
Action 0, answer class_ id: Submit your final answer as integer label.
Action node node id: Retrieve the list of neighboring nodes connected to the specified node. Action node node_i Retrieve the textual description (features) of the specified node. Action 3, node 1o Retrieve the label the specified node otherwisc, return None.
the training set;
Action 4, node node id, hop rum hop: Retrieve the textual descriptions (Features) of all nodes that exactly num hop hops away from the specified node.
Action 5, node node _id, hop num hop: Retrieve the labels (or None) of all nodes that are exactly num hop hops away from the specified node.
Available class labels:
0: (text description of label 0)
Now begin your reasoning Scratchpad below:
22

---

Published as a conference paper at ICLR 2026
Table 12: Statistics of all datasets
Dataset
#Nodes
#Edges
#Classes
Train/Val/Test (%)
cora pubmed
2,708
5,429
60/20/20
Short-text homophilic
19,717
44,338
3
60/20/20
arxiv
169,343
1,166,243
40
53.7/17.6/22 6/28.77
products
2,449.029
61,859,140
47
8,0/1.6/90.
cornell
191
292
60/20/20
Heterophilic
texas
187
310
60/20/20 5
washington
229
394
60/20/20
wisconsin
265
510
60/20/20 5
citeseer
3,186
4,277
60/20/20
reddit
33,434
198,448
60/20/20
Long-text homophilic
computer
87,229
721,081
10
60/20/20
photo
48,362
500.928
12
60/20/20
instagram
11,339
144.010
60/20/20
wiki-cs 11,701
215,863
10
60/20/20
Template Graph-as-Cool
Task:
You solving node-based reasoning... forr node [node id). You have pandas DataFrame d.E
where each row corresponds to node, indexed by node
Instructions: Always begin with reasoning... Schema structure:
The DataFrame index is the node id. Access row node id with: df loc [node_id]
stores each node's textual description: af loc [nodc_id, 'features' The column no neighbors stores a list of neighbor node IDs: df. los [node_id, nc 'niighoor' ]. The column label contains the integer node label if it belongs to the training set; otherwise None. You may query ANY column(s) of df using any valid pandas command that applies DataFrame named df. You may also pd. utilities with of input. The dataframe be long, you may want avoid commands that print entire table.
The column feat
Response format:
For intermediate steps: reason then on the final line output single valid pandas expression. To finish: reason then on the final line respond exactly as: Answer [class_ié]]
Available class labels:
0: (text description of label 0}
All experiments were conducted using Intel Xeon Platinum 8370C cpus.
G DATASET STATISTICS
The statistics of all datasets be found in Table 12
23


Reviews:
[{"review_id": "cB6nwPFkJq", "reviewer": "Reviewer_xRrD", "summary": "The paper benchmarks these modes on node classification tasks across homophilic vs. heterophilic graphs, short- vs. long-text datasets, and different model scales (from LLaMA to GPT-5). The authors report that Graph-as-Code consistently outperforms prompting and tool-calling due to better token efficiency and compositional reasoning.", "questions": "1. The Graph-as-Code paradigm is described abstractly. If code generation and execution are treated as a form of tool invocation, then Graph-as-Code is effectively a special case of GraphTool with compositional or batched tool calls. Both rely on external execution environments to retrieve graph information. \n2. In Prompting, neighborhood hops are truncated due to token limits. In GraphTool/Graph-as-Code, there is no equivalent limit, and the LLM can query indefinitely. This makes the modes incomparable in resource exposure: Graph-as-Code effectively sees more information than Prompting. How many tokens are used per query on average?\n3. The work attributes improvement to “graph reasoning,” but Graph-as-Code might simply exploit code-LLM priors (e.g., structural induction, algorithmic recall). There is no control experiment using the same LLM on shuffled or random adjacency matrices to test if improvements are truly structure-dependent.\n4. In GraphTool, how many reasoning steps are allowed? \n5. Could authors present an example of how Graph-as-Code behaves in practice?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Timely and high-impact topic.\n- Reproducibility transparency.", "weaknesses": "- Conceptual overlap between GraphTool and Graph-as-Code.\n- Weak theoretical framing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "np8w3yd2tn", "reviewer": "Reviewer_37Ma", "summary": "The paper presents a systematic, large-scale empirical study on the performance of LLMs in graph inference tasks—particularly node classification. The authors design three LLM–graph interaction paradigms: Prompting, GraphTool, and Graph-as-Code. They conduct controlled experiments across multiple dimensions—including dataset domains, homophilic vs. heterophilic graph structures, text length of node features, and LLM model scales—and further employ ablation analyses to reveal how different methods rely on graph structure, node features, and labels.", "questions": "> *I originally intended to give this paper a score of 7, but ICLR doesn’t provide that option. I hope the authors can further strengthen the completeness and rigor of their experiments and deepen the discussion around Graph-as-Code—especially its scalability, efficiency, and extensibility to other graph tasks.*", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Systematicity and Experimental Breadth**: This work is the first to factorize LLM-based graph reasoning capabilities across six key axes: interaction mode, dataset domain, homophily level, feature text length, model scale, and reasoning capability. The experimental design is rigorous and covers the major variables present in real-world scenarios.  \n\n- **Proposal and Validation of Graph-as-Code**: The paper introduces *Graph-as-Code* as a novel LLM–graph interaction paradigm that leverages LLMs’ code generation abilities (e.g., via pandas operations) to dynamically query graph data. It significantly outperforms traditional prompting—especially on long-text or high-degree graphs. (**This is an excellent contribution that finally provides clear research direction for the emerging field of LLMs for graphs.**)\n\n- **Challenging Prevailing Assumptions**: The paper refutes the commonly held belief that “LLMs fail on heterophilic graphs.” Experimental results show that all interaction modes perform well on heterophilic graphs, demonstrating that LLMs can effectively leverage node features rather than relying solely on neighborhood label consistency.", "weaknesses": "- All experiments are limited to **node classification**; it remains unclear whether the conclusions generalize to other graph tasks such as graph classification, link prediction, or subgraph matching.  \n- The **baselines are simplistic**, including only Label Propagation, Random, and Majority voting—without comparison to state-of-the-art GNNs or hybrid LLM+GNN models. *(Note: this is understandable given the paper’s focus on empirical analysis rather than SOTA performance.)*  \n- Experiments are **dominated by closed-source models**, which limits reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3cTuL7e2lg", "reviewer": "Reviewer_RTGA", "summary": "This paper investigates the effectiveness of LLMs for solving the node classification task through extensive controlled experiments. The authors consider various factors, including the mode of interaction between LLMs and graphs, graph domain and structural/feature characteristics, as well as LLM model size and reasoning capabilities. The experiments reveal several valuable findings, such as the strong and robust performance of the \"Graph-as-Code\" interaction mode across different graphs, even under perturbed features or structures. These insights provide a useful foundation for advancing research in both the graph learning and LLM communities.", "questions": "Please refer to Weaknesses section", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **This paper broadens the discussion on how LLMs interact with graphs** by bringing attention to the tool-use manner (GraphTool) and code generation (Graph-as-Code). These interaction paradigms have largely been overlooked in existing benchmarks for LLMs applied to the node classification task.\n\n2. **The authors evaluate a wide range of variables**, including interaction modes between LLMs and graphs, graph characteristics (domain, homophily, text-feature informativeness, etc.), and LLM capabilities (model size, reasoning ability, and series). Additionally, they investigate how different graph manipulations (e.g., truncated features, deleted edges, removed labels) interplay with these variables, providing a comprehensive understanding of performance under diverse conditions.\n\n3. **The experiments are extensive and bring many actionable insights**. Among the interaction modes, \"Graph-as-Code\" achieves strong overall performance and demonstrates robust generalization across perturbed graphs. This makes it a highly favorable method. Furthermore, even on heterophilic graphs, LLMs leverage their reasoning capabilities to deliver reliable and accurate predictions. These findings are promising to benefit the community by encouraging further exploration of the Graph-as-Code approach, both for developing new methodologies and establishing new benchmarks.\n\n4. The paper is **well-written**, with a logical flow that is easy to follow. The authors effectively use vivid figures and tables to present their findings, making the results accessible and facilitating understanding.", "weaknesses": "1. The paper could discuss the computational costs associated with the different interaction modes, e.g., token consumption. This would help readers better understand the performance-efficiency trade-offs of different methods.\n\n2. The evaluation focuses primarily on scenarios with abundant supervision, e.g., up to 60% of training data on most datasets. However, it would be valuable to compare the few-shot or zero-shot performance of LLMs, as these capabilities are a key strength of LLMs and could reveal additional insights.\n\n3. The considered tasks are limited to node classification. Expanding the scope to include other graph computational problems, e.g., shortest path, could further demonstrate the utility of the Graph-as-Code methodology, particularly in its ability to directly generate Python code for problem-solving.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j8IATAnMPP", "reviewer": "Reviewer_PyZw", "summary": "The paper systematically evaluates how large language models perform on node classification across multiple settings, including interaction mode (prompting, tool-use, and code generation), dataset type, graph structure, and model scale. The authors find that Graph-as-Code, which leverages LLMs’ code generation ability, achieves the strongest and most adaptive performance—especially on long-text or high-degree graphs. Through controlled ablation studies, the paper demonstrates that Graph-as-Code dynamically adjusts its reliance on structural, feature, and label information, offering an efficient and flexible approach to LLM-based graph reasoning.", "questions": "Q1. Can you include comparisons with GNN baselines such as GCN, GraphSAGE, or GAT?\n\nQ2. Can you report computational efficiency metrics such as runtime, token usage, or latency for each LLM–graph interaction mode?\n\nQ3. Can you provide case studies or examples showing how Graph-as-Code performs reasoning or generates code during inference?\n\nQ4. When edges are deleted, how does Graph-as-Code access feature and label information of other nodes—does it query arbitrary nodes rather than neighbor nodes?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper conducts a large-scale controlled evaluation of LLMs for text-rich graph tasks (e.g., node classification) across six key variability axes, filling the field’s gap in principled understanding of LLM-graph interactions.\n\nS2. It proves all LLM-graph interaction modes work on heterophilic graphs (challenging prior assumptions) and identifies Graph-as-Code as optimal, especially for long-text/high-degree graphs.", "weaknesses": "W1. The paper lacks comparisons with GNN baselines (e.g., GCN, GAT), making it hard to judge LLMs’ competitiveness against GNN methods.\n\nW2. The evaluation focuses mainly on accuracy, with limited discussion of computational efficiency, latency, or interpretability trade-offs, which are critical for practical use.\n\nW3. The paper lacks case studies or examples illustrating Graph-as-Code’s internal reasoning or generated code behavior, which would help clarify its mechanisms beyond aggregate metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)