GraphOmni: A Comprehensive and Extensible Benchmark Framework for Large Language Models on Graph-theoretic Tasks
Authors: Hao Xu
Xiangru Jian
Xinjian Zhao
Wei Pang
Chao Zhang
Suyuchen Wang
Qixin Zhang
Zhengyuan Dong
Joao Monteiro
Bang Liu
Qiuzhuang Sun
Tianshu Yu
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Benchmark and Evaluation
Prompt Optimization
Primary area: datasets and benchmarks

Abstract: This paper introduces GraphOmni, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs on graph-theoretic tasks articulated in natural language. GraphOmni spans diverse graph types, serialization formats, and prompting schemes, substantially extending upon prior efforts in both scope and depth. Through systematic evaluation, we uncover critical interactions among these dimensions, revealing their decisive impact on model performance. Our experiments show that state-of-the-art closed-source models such as Claude-3.5 and o4-mini consistently lead overall, yet still leave considerable headroom, while open-source models display pronounced sensitivity to various design choices. Beyond the standard scope, larger graphs, real-world graphs, and additional NP-hard tasks are further discussed. We further analyze efficiency via output token usage, highlighting cost–accuracy trade-offs, and introduce a reinforcement learning-based optimizer that adaptively selects factor combinations, reducing evaluation cost by 75\% while retaining strong accuracy. This flexible and extensible benchmark not only deepens understanding of LLM performance on structured graph reasoning but also establishes a robust foundation for advancing model design and evaluation.

Published as a conference paper at ICLR 2026
GRAPHOMNI: A COMPREHENSIVE AND EXTENSIBLE BENCHMARK FRAMEWORK FOR LARGE LANGUAGE MODELS ON GRAPH-THEORETIC TASKS
1 Hao Xu', Xiangru Jian 11 "t Xinjiaan Zhao Wei Pang", "Chao Zhang, "Suyuchen Wang, $ Qixin Zhang, 'Zhengyun uan Dong, Joao Monteiro, 'Bang Liu, "Qiuzhuang Sun, 'Tianhhu Yu
1 The Chinese University of Hong Kong, Shenzhen, UUniversity of Waterloo
3 Université de Montréal Mila Quebec AI Institute, City University of Hong Kong
5 Autodesk, School of Mathematics and Statistics, University of Sydney
Project Page:
https://ithub.com///A/ http:///haa..oooooooooooooonnnna Grniipphhhh/////// http://wuggnngaaaeeooodddeeeee/
ABSTRACT
This paper introduces GRAPHOMNL, a comprehensive benchmark designed evaluate the reasoning capabilities of LLMs on graph-theoretic tasks articulated in natural language. GRAPHOMNI spans diverse graph types, serialization formats, and prompting schemes, substantially extending upon prior efforts in both scope and depth. Through systematic evaluation, we uncover critical interactions among these dimensions, revealing their decisive impact model performance. Our experiments show that state-of-the-an closed-source models such as Claude-3.5 and o4-mini consistently lead overall, yet still leave considerable headroom, while open- source models display pronounced sensitivity to various design choices. Beyond the standard scope, larger graphs, real-world graphs, and additional NP-hard tasks are further discussed. We further analyze efficiency via output token usage, highlighting cost-accuracy trade-off's, and introduce reinforcement learning-based optimizer that adaptively selects factor combinations, reducing evaluation cost by 75% while retaining strong accuracy. This flexible and extensible benchmark not only deepens understanding of LLM performance structured graph reasoning but also establishes a robust foundation for advancing model design and evaluation.
INTRODUCTION
Large Language Models (LLMs) have emerged as transformative force natural language process- ing (NLP), demonstrating state-of-the-as performance in tasks such open-ended text generation, summarization, and problem-solving (Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020; Lewis et al., 2020). However, their application to structured reasoning on graph-based data remains relatively underexplored. Graphs, defined by their nodes and edges, encapsulate complex relation- ships that are crucial many real- wolldwrlld applications, including social network analysis (Easley et al., 2010), recommendation systems (Wu et al., 2022; Jian & Wang, 2023), out-of-distribution detection (Fang al., 2025a), and drug discovery (Gaudelet et al., 2021; Wang & Zhuang, 2025).
Traditional approaches to graph analysis primarily rely on Graph Neural Networks (GNNs) that are designed with specialized representations and training paradigms tailored for tasks such node classification (Wu et al., 2020), link prediction (Zhang & Chen, 2018), and community detection (Su al., 2022). In contrast, LLMs are trained on vast quantities of unstructured or semi-structured text and excel at reasoning about entities and relationships described linguistically, as evidenced
"Equal contribution. X Corresponding to: xiangru,jiane uwaterloo.ca, yutianshu @ @cu.k.edu.cn
I Project Lead

---

Published as a conference paper at ICLR 2026
Evalustion Pipaline
Graph Thearetic Questions
Adjustablc Settings
LLM
Answers
Local Propertics
Graph types
Ac are D 3-8 the conminc eycie Thheee Intenglee Arthe longeet tharteet tn U1 to whics takes dstess on
Are the Kalkwwigg two rodes connodied
Desornins lowiing
cydee.
Global Propertes
D: Counn number of stangla thi undiocttdigggh
Formals
Componation on
Commuiis sumoker quach find durited heee node à nodee sertorm a beracth-tret tnowrol
ABC.C
CA.5
0101101
A110211
A46O
08
3.
chooee Sohw "ee bumplen
Led thit by ripp CChiin
D-0
Rength
w Dis
Vura
Set
Arriee
Ac
E
Discussions
Factora
Performance Hostman
Enror calse soud
Fin Rgg
Grampeplec pourtal a acciinglo countings
Find kg Higghsist Bi and
shenple of
Al leee carking endees
Bund
unsmmition Hingll hemuls.
sungle anch
Figure 1: GRAPHOMNI Evaluation Pipeline. We convert graph-thewr theorett tasks into text- based questions about local and global properties. In the adjustable settings, we vary three dimensions, i.e., graph type, serialization format, and prompt scheme, and then generate every possible combination.
by benchmarks like MMLU (Hendrycks et al., 202 and MATH (Hendrycks et al., 2021b). This discrepancy raises a pivotal question: Can LLMs be effectively harnessed to understand and manipulate graph-theoretic concepts when graphs articulated in natural language?
To address this question, multi-dimeenionnl evaluation is required rather than tuning a single knob. Prior work has examined individual components in isolation, including prompting strategies (Wang et al., 2023; Fatemi et al., 2024), textual graph serialization (Xypolopoulos et al., 2024), or specific graph families (Zhang al., 2024b), but this piecemeal view obscures how these choices interact. We therefore vary three interacting dimensions jointly. First, graph type: different graph types exhibit distinct structures, so we use synthetic generators (ER, BA, scale-free, bipartite) to produce them, which in turn affects how readily a text description can capture these structures. Second, serialization format: the same graph written as an adjacency list or matrix, an edge set, or a richer schema can help hinder model reading. Third, prompt scheme: the way the question is posed (zero-shov few-shot, instructive, algorithmic, chain-of-thought) can shift answers even with identical inputs. As summarized in Table 1, previous studies do not vary these dimensions together, so they cannot determine whether gains come from the model, the representation, or the instruction, nor explain why setting that benefits model may harm another. Consequently, still lack comprehensive and robust understanding of LLM capabilities in graph reasoning.
Table 1: Comparison of existing graph-related benchmarks for LLM with our GRAPHOMNI. We evaluate their inclusion of different types of graphs, serialization formats, and prompt schemes, noting gap between recent works and ours. Additionally, GRAPHOMNI is the only work with random baseline well as modularized and expandable framework design. More related works included in Detailed Related Works in Appendix F.
Benehmarks
Graph Sources
Serializations
Prompi Schemee
Evalustion Framework
* Samples
# Graph Types' Node Size
Multiple Types
Types Mulúple Types
Types Random Buseline Modularized
LLM4DyG (Zlagg et 20246)
900 per tosk)
to 20
X
Gruphinstract (Lao 2024b)
N/A
S 35
MAGMA at
NLGraph (Wang et 2023)
GPC (Thi
NIA
OriphWiz (Uhen 20040)
3,600
2o
GPP4Grrph (fan etal. 20144)
N/A
10 20
/
6
Grophrren (Tang ctal. 2015)
10,000
NIA
5 501
X (oniy via
X
GriliQA ('unemi
2,300
S
NLGiit (Zhasa
Graph
al., 2025)
GRAPHOMNE
241,736
Nove that Graph Typos targetad for syntheri datasets and reflocts the mumber of types of ruedom graph generators. The range for all non civial taaks, excluding nearest neighbur and sbortest distance.
2

---

Published as a conference paper at ICLR 2026
Easy
Medium
Hard
Cnt.
Cnt.
Cy.
Cnt.
Llama-3.1 (8B)
Mistral (7B)
<db 2
Phi-4 (14B)
Dia.
BFS
Dia.
BFS
Dia.
BFS
Qwen2. (7B)
Llama-3 (8B)
Qwen2.5 (728)
Vani
Zoanrd
Etemat Vina
Qwen3 (8B)
Short.
Shart.
Tri.
Short.
Tri.
Easy
Medium
Hard
Cy.
Cnt.
cy.
Cnt.
Cy.
Cnt.
Claude-3.5
Gemini-2.0
GPT-4o
Dia.
0.00
BFS
Dia.
BFS
Dia.
BFS
GPT-4o-mini
04-mini
Tosmed Viei
Itooned Gwel
Reooee Vin
Short.
Tri.
Short.
Tri.
Short.
Trl.
Figure 2: Radar charts comparing the performance of open-source (top row) and closed-source (bottom row) LLMs across six canonical graph reasoning tasks at three difficulty levels.
To address this gap, propose GRAPHOMNI, unified benchmark with extensible framework, summarized in Figure 1. represents the most comprehensive graph-theory-biie framework developed to date, compared with all related works in Table 1. spans various graph types, serialization formats, and prompt schemes, surpassing previous works in scope and granularity. Furthermore, our framework designed as an extensible and flexible evaluation system. Researchers can easily incorporate new graph generators, scrialization methods, and prompt strategies, thereby ensuring that the benchmark remains current with evolving methodologies both LLM research and graph theory. A random baseline is then implemented to ensure fair evaluation.
evaluation
With the help of GRAPHOMNI we clearly demonstrate that no single serialization or prompt works best for all models and accuracy varies widely across graph types, serializations, and prompts, which validates the need for our multi-dimensional design and per-task configuration. Additionally, model performance requires further improvement overall: Claude-3.5 and 04-mini lead across tasks and difficulty levels, yet even they fall short the near- cear-perfect accuracy a non-specialist human evaluator could achieve on 5-30 node problems given sufficient time. To verify the robustness of the evaluation results, we extend the analysis to larger graphs, NP-hard tasks, and conduct a representativeness check real-world graphs, all of which yield the same trends. Motivated by these results, introduce simple RL-inspired selector that chooses the optimal settings (prompt + serialization) for each task, thereby improving accuracy minimal extra cost. We summarize contributions as:
Novel benchmark: We introduce GRAPHOMNT, the most comprehensive benchmark to our knowledge for evaluating greph-theoretic reasoning in L.MSs, covering wide range of synthetic graph types, diverse serialization formats, and varied prompt schemes.
Comprehensive evaluation framework: We design flexible and extensible evaluation framework that allows for the seamless addition or removal of graph generators, serialization methods, and prompt schemes, ensuring adaptability to future research developments. We also include extended studies on larger graphs (30-50 nodes), real-world datasets, and NP-Hard tasks, which together confirm the robustness and transferability of our conclusions.
Insightful empirical observations: State-of-the-ar models still exhibit considerable room for improvement overall. Our experiments reveal substantial performance variance, with notable accuracy differences across different serialization and prompting configurations, emphasizing the need for comprehensive evaluation across all dimensions to provide fair and trustworthy understandings.
Practical methods inspired by observations: Motivated by the above observations, we develop an RL-based adaptive mechanism that dynamically selects the optimal factors, achieving near-optimal performance with only small exploration cost.
3

---

Published as a conference paper at ICLR 2026
2 GRAPHOMNI
Overview and Statistics. GRAPHOMNT rigorously evaluates LLM performance on graph reasoning by examining the interplay between graph structure, textual representation, and prompt formulation. comprises four key components: Benchmark Tasks, Graph Types, Prompt Schemes, and Serial- ization Formats. Figure I illustrates how these four components form our end-to-end evaluation pipeline. Benchmark Tasks cover canonical graph problems that test both local and global reasoning. Graph Types are defined by diverse synthetic datasets generated by different random graph genera- tors, including stochastic, scale-free, and bipartite models. Prompt Schemes incorporate various query designs such algorithmic, chain-of-thought, k-shot, instructive, and zero-shot approaches. Serialization Formats convert graph data into text using methods like adjacency lists, matrices, and the GMoL. Moreover, we have designed three difficulty modes for all graph-related tasks, determined by the number of nodes: Easy (5-10 nodes), Medi um (10-20 nodes), and Hard (20-30 nodes). This unified and extensible framework distinguishes itself by integrating multiple dimensions of graph reasoning into a single evaluation platform, thereby providing comprehensive insights into LLM performance on complex, structured data. The basic statisties of GRAPHOMNI are presented in Table 2, while token statistics for different combinations are shown in Figure 3. In summary, our dataset contains total of 2,1.726 queries. More detailed statistics are in Appendix B.
Graph Tasks. We consider 6 canonical tasks that cap- ture both local and global properties of graphs, thereby requiring diverse reasoning capabilities from LLMs. Connectivity involves determining whether a path exists between two designated nodes, testing the model's under- standing of local linkages. Cycle detection requires ver- ifying the presence of any cycle, which probes the model's 's ability to recognize recurring patterns in connectivity, Di Diameter calculation demands calculating the max- imum distance between any two nodes, thereby challeng- ing the model to grasp the global network structure. BFS order tests the ability to generate an ordered sequence of nodes as encountered in a breadth- -readt search, assess- ing sequential output and structured reasoning. Triangle counting requires precise numerical enumeration cycles, blending quantitative precision with structural in- sight. Shortest path tasks compel the model to identify the most efficient route between nodes. Collectively, these tasks provide robust measure of performance across both binary decisions and nuanced numerical analyses. For more details on the design of the graph task, please refer to Appendix A.3, where we further discuss the rationale behind the task selection and analyze the distinct capability demands of each task in Appendix 1. We also include NP-hard tasks for extended discussion, elaborated in Appendix C.4.
3500
3000
2500
2000
1500
1000
see
OvAlgarithes
GMol MMa EL At Al Serialisation Formatt
o-instruct
Algarithm
Pomppe Scheme
CoT
Köhht
3-
Figure 3: Token usage for prompt- serialization combinations by GPT-4 tokenizer. More detailed statistics are included in Figures 6a and 6b.
Graph Generators (Types of Graphs). To mirror the diversity found in real-world net- works, our benchmark incorporates a broad array of graph families of 7 types, each pre- senting unique structural characteristics that challenge LLM reasoning. ER Graphs are gen- erated by random sampling from the space of all graphs with vertices. Within this fam- ily, ERM employs a fixed edge count m, randomly chosen between and m(n-1), while ERP uses a probability-based approach with an edge probability drawn uniformly from [0, 1]. Extending these models to capture structured varia- tions, Bipartite ER Graphs (denoted as BERM and BERP) impose bipartite constraints that yield additional topo- logical diversity. To reflect the power-law distribu- tions prevalent in real-world networks, we include Barabási-Albert Graphs tializing a complete graph of my vertices (with mo randomly chosen up to ") and sequentially adding nodes that form m = mp + connections via pref- erential attachment. Recognizing that many practical
2
Table 2: Statistical summary of GRAPHOMNI over tasks at all difficulty levels. More statistics are in Table 7.
generated by ini-
Difficulty
Easy
Medium
Hard
Numbers
88956
87318
65452
Avg. Nodes
8.01
14.70
26.61
Avg. Edges
11.70
34.51
77.60
4

---

Published as a conference paper at ICLR 2026
networks are hierarchical or tree-like, extend BAG
to Barabási-Albert Forests (BAF) by enforcing an acyclic topology. Moreover, our framework features Scale-Free (SF) Graphs generated via a degree-weighted' random connection strategy, which can yield multiple disconnected components, offering a complementary perspective to BAG. A detailed description of each type of graph be found in Appendix A.4, where we also provide the detailed rationale for this selection and empirical evidence showing that even within the 5-30 node range, the chosen generators yield statistically distinct and representative structural regimes in Appendix 1.4.1.
Recognizing that the formulation of query prompts critically influences LLM reasoning, our benchmark systematically evaluates 9 distinct prompt schemes that vary in the degree of explicit guidance provided. The k-Shot prompts supply multiple exemplars from simpler graph instances to prime the model with relevant examples. The Algor thm prompts (Wang et al., 2023) explicitly delineate well-known algorithm (such as BFS or Dijkstra), offering clear procedural instructions. In contrast, Chain-of-Thought (CoT) prompts (Wei et al., 2022) encourage the model to articulate intermediate reasoning steps, thereby exposing its internal thought process. The Instruct prompts directive language tailored for iirruction-basse models to elicit focused responses. All three types above come with few-shot examples. For cases requiring minimal intervention, the @-Shot (i.e., plain) prompts pose bare questions without supplementary cues. To further investigate the impact of reasoning visibility, we also include variants without few-shot examples, such as 0-CoT, 0- Innttruct, and 0-Algori thmm which deliberately restrict the exposure of intermediate solution steps, as well as LTM prompts that employ ceast-to-most prompting. The detailed design process and some examples of the prompt program are shown in Appendix A.5.
Prompt Schemes.
Serialization Formats. Since LLMS operate on textual inputs, the method by which graphs are serialized has profound impact on the clarity and accessibility of structural information. Our benchmark examines 7 distinct serialization formats that offer varied representations of graph topology. The Graph Modeling Language (GMoL) provides a structured, tag- representation that mirrors hierarchical data organization. In contrast, the Adjacency Set and Edge Set formats offer succinet listings of node neighbors and edges, respectively, emphasizing compactness. The Edge List format, which may incorporate additional details such as edge weights, serves more verbose alternative. Moreover, the Adjacency Matrix and Adjacency List formats balance detail and conciseness differently depending the graph density, and the Graph Markup Language (GMaL) (Brandes et al., 2013) is an XML-based file format used to describe graph structures, including nodes, edges, and their attributes. Specific examples of graph serialization formats are in Appendix A.6.
EXPERIMENTAL SETTING
We evaluate the graph reasoning capabilities of various LLMs a diverse set of tasks and difficulty levels. Our protocol highlights the impact of different dimensions in Section 2 on model performance.
Random Baselines. To assess the intrinsic graph reasoning ability of our models, include random baseline for each task. Appendix A.2 shows its detailed design process. These baselines provide clear reference point for evaluating how much the LLMs improve upon chance performance when reasoning about traph-theoretic properties expressed in natural language.
Models and Configurations. We evaluate a diverse suite of LLMs spanning both open-source and closed-source categories. Our open-source models include LI Llama-3, Llama-3. 3. 1, Mistral, Phi-4, 5, and Qwen-3, while our closed-source models consist of Claude 5, Gemini-2. GPT-4o, GPT-4o-mini and o4-mini (versions and sources of the LLMs applied can be found in Appendix A.1). The model selection here is designed to provide coverage of the widely used LLMs of different sizes, reasoning types, and whether they are open-sourced or not, based on the budget and availability of models the time of the work. We also try our best to include models with better performance on GRAPHOMNI than comparable alternatives to make our conclusion convincing. In all experiments related to few-shot examples, five exemplars are prepended to the prompt (i.c., k=5). More implementation details can be found in Appendix A.
Evaluation Metrics. Evaluation of LLM responses is conducted using predefined binary accuracy metrics, assigning an output of for correct responses and 0 for incorrect responses. For qualitative tasks. such as Connectivity ty verification and Cycle detection, correctness is determined by identifying and verifying key phrases in the model's output (e.g., "yes, there cycle" or "yes, there
5

---

Published as a conference paper at ICLR 2026
is path") against the ground truth (GT). For numerical tasks, such as Triangle counting and Diameter calculation, correctness is assessed by extracting numerical values that follow specific key phrases (e.g., "the number of triangles is" "the diameter is") and directly comparing these numerical outputs to the corresponding ground truth values. For tasks with multiple valid solutions, specifically BFS order and Shortest path, evaluation is conducted using a rule-based function. This evaluation process involves identifying key phrases, such as "The BFS traversal starting from node X is" "The shortest path from node X node Y is,," " to extract the model's response. Based on this extraction, we evaluate the model's ss response using a task speci-ic verifies solutions for tasks and assigns score of when the response matches one of the correct answers. The detailed rationale for the choice of the metrics is included in Appendix C.6.
algorithm that
Table 3: Benchmark Results of Representative LLMs Across Tasks (Meant95% CI Margin). Bold orange Underlined blue/ / Light purple highlights indicate best/second-bbett best//hiird-boe per- formance in its category. The complete results are included in Table 13.
Dilicutt
Llams-31 (BB)
Mistral
Pli-4
Open-suucce Monich
Quean225
Queas23.5
Qwea-3 (SB)
Clasde-35
Chasct-surce Modch
o4-mini
137521 46
1500877322
754413345
21:4684255
00.8735.39
$1.4883222
90.3622 30
95 BELAGE77
1000
BFS crier
5.27a00 MS
4.3680:4
12.4923.20
80ta1.41
41
$5.0144.50
6000
79 S03221
79 3M
26:2948.00 56
00:3411 89
15 14
97.3743229
18 88
91 92
Ms 201883
67:00
Couectiviy
70:75
74 64.5822.6
74.775246
596
02.5922.07
47.99gl
55 -91A-1O
45:2585 FM
3400225-44 34
02 1961855
30.3062.33
$2.5625.89 89
85.0882
02.3023
97.9724733
50,00
Cycle
33.71an
44.3685.4
02.072180
90.0023334
00-7566.76
86.8133.29
41 227--2211
28.5938.22
42.4133M K6
78.5041.36
222662277
832281125
53.9922.10 19
78.18
11:31
Dtumator
15:174257
38-4938199
21-18311
617113.38
48-3242
72848183
8:20
17.
36 75655555 $1
31.1858.45
42618838
98/322:37
47 27.665-75
94 35352233
90.17A1.01
41.7504 TO
95
50,00
Stacies
17:3.03
3047744
12.2121.96
266085.21
39
50
74.9644117
14.7211333
18.8781.22
12:498245
36 26.7755-40 401
18.581125 1244
23.1211554
36.3281
30-3822-31
84.5420155
2:33
Thanal:
8.55.00122 922
5.858873
14.222285
1,62
4.5
19, 14.44.1.34
15955077
13.8160.80 88
[5,35el
17.53e1.45
4 RESULTS AND ANALYSIS
4.1 MAIN RESULTS
We evaluate model performance comprehensively across four main dimensions: model overall ca- pability, graph type. effectiveness of prompting strategy, and impact of serialization format. This multifaceted evaluation offers a comprehensive understanding of the most effective approaches for graph algorithm tasks. Our analysis systematically considers each task at varying difficulty levels (easymeddun/hard). To isolate each dimension, we control for other variables when assessing particular aspect and calculate the mean accuracy with 95% confidence interval (Mean+95% CI Margin) across all combinations of the remaining factors. For example, when evaluating model capability, we compute statisties across all combinations of graph types, prompts, and serialization formats while holding the model constant. The evaluation results from the model capability perspec- tive are presented in Table 3 and Figure 2. To provide a comprehensive view, we present additional experimental results in Appendix 1. examining prompt schemes, serialization formats, and graph types across collective results (Tables 14, 15, 16), open-source models (Tables 17, 18, 19), and closed-source models (Tables 20, 21, 22). These controlled evaluations yield complementary insights summarized across multiple perspectives. Additionally, example input/output pairs are provided for clarity in Appendix E.5.
Result O: High variability underscores the need for comprehensive evaluation across all benchmarking dimensions. Detailed analysis reveals substantial variability in LLM performance across different combinations of serialization formats, prompting schemes, and graph types. This variability highlights the need for a comprehensive evaluation across all benchmarking dimensions. The performance heatmaps, presented in Appendix E.2, illustrate the accuracy of different prompt schemes and serialization formats across tasks, models, and difficulty levels. The performance heatmaps show that no single serialization or prompting strategy consistently outperforms others across all tasks and difficulty levels. Instead, optimal results require careful and adaptive selection of serialization cettiizaaton-rropppl combinations, explicitly tailored to task characteristics such as structured graph- theoretic reasoning tasks. For instance, in the case of GPT-4o, depicted in Figure 4, accuracy gaps of up to 40% occur when varying input representations within the same task and model, indicating
6

---

Published as a conference paper at ICLR 2026
significant sensitivity to input formatting, which is also observed in other domains, like evaluation of vision language models (VLMs) (Feizi et al., 2025). These observations emphasize that evaluating LLMs comprehensively across interconnected dimensions, i.c., serialization formats and prompting schemes, is essential for fairly assessing their capabilities in graph reasoning tasks.
Result @: Model performance still has considerable room for improvement. Models generally demonstrate reasonable performance across tasks, underscoring their inherent potential in graph reasoning when appropriately guided. Notably, 04-mini delivers remarkable performance, frequently surpassing other closed-source models across most tasks and setting a new benchmark overall. However, the performance gap remains large on the hard difficulty tasks, particularly BFS order, Diameter calcul ct,oo,, and Triangle counting, which require full, global information of the graph. Here, even 04-mini's performance drops to as low as 32.45% on BFS order (Hard), 34.61 % on Di Dimeter calculation (Hard), and 17.53% on Triangle counting (Hard), underscoring the remaining challenge in holistic graph reasoning. Therefore, substantial room for improvement persists relative to ideal human- outcomes, primarily due to the scarcity of structured graph-theoretic content in typical web corpora used for LLM pretraining. Among open-source alternatives, Qwen-3 remains the top performer but continues to lag behind leading closed-souree models, such as 04-mini and 3.5, suggesting a meaningful room for advancement in open- spen-source solutions.
Result 0: Common Errors Reveal Fundamental Gaps in Graph Reasoning. Our error analysis highlights representative categories of errors commonly observed in incorrect L.M responses: A. Misinterpretation of serialization formats: Models occasionally struggled accurately interpret serialized graph representations, resulting in misunderstandings of the underlying graph structure, such BFS order case I, Connectivi ty case 1, and Triangle counting case in the Appendix: B. Incorrect reasoning about graph- misunderstandings of basic graph definitions and mooblem-solving methods. In the error cases Triangle counting case 1, incorrect responses inaccurately estimated the number of triangles as approximately one-third of the number of nodes. For the error cases Diame ter cal culation on case 1, some models erroneously identified the diameter as the length of the longest path, rather than correctly defining it as the length of the longest shortest path between any two nodes, These representative errors underscore critical areas for improvement in the graph reasoning capabílities of current LLMs. Additional error cases and analyses are provided in Appendix E.4.
concepts: LLMs frequently exhibited fundamental
Easy
Med
Hurd
0-Algorithm
0.725
0.500
0:Go7
9.529
0.789
0.433
1.532
0.054
0.525
0.348
0.252
8:543
0.452
0.500
0.335
0 0.29
0.404
Promet Schemea
0-Inafruct
0.648
0.548
3.707
3:535
5.555
5.641
6.623
0.523
0.531
0.815
0.205
0.500
0.547
0.562
0.454
0.455
0.454
0.232
0.23
0.512
0.505
0-Shot
9.541
8.758
0:523
0.30
0.510
0.256
Algorihi
CoT
0.711
9.617
3.660
0:495
0.501
0.529
6.683
0.648
0.555
0.860
0.368
0.402
0.6 6.64
0. 5.500
0.555
0.11
0.667
0.367
0.22
0.53
5.50
Instruct
0.809
0.833
3.885
3.553
0.551
8.645
0.504
0.652
0.582
0.852
0.418
0.445
0.522
8.573
0.555
0.470
0.815
0.440
0.222
0303
0.577
K-Shot
8.200
3.854
3.543
0.525
0.625
0.565
8.621
0.482
0.79
0331
8.551
0.571
0.500
0.542
0.345
0.198
LTM
AM
AS
EL
65
GMal GMol.
AL
AM
A5
EL
E5
GMal
GMol.
AL
AM
AS
EL
E5
GMaL. GMol.
Serilization Formate
Serinkzation Formats
Sarinibatiion Formats
0.5
0.0 Accurscy Range
0.7
0.8
0.3
0.4 Accuracy Range
0.5
0.5
3.7
02
0.3
0.4 Accurscy Range
0.6
0.6
Figure 4: Performance heatmaps for different prompt schemes and serialization formats on Diameter calculation of GPT-4o. The color intensity represents the accuracy, with darker colors indicating better performance. The red solid and yellow dashed line indicates Best and Second Best Performance, respectively.
4.2 FINE-GRAINED EMPIRICAL FINDINGS ON MODEL PERFORMANCE
In this section, we dive deeper into our empirical results, identifying detailed performance patterns and revealing nuanced interactions across various evaluation dimensions. We present here the two most critical findings, while additional observations are available in Appendix E.6.
Finding O: Domain-specific knowledge significantly improves model performance on graph- theoretic tasks. Algorithm-bast based prompts, explicitly detailing teeh-ttheoretic algorithmss consis- tently improved model accuracy in structured reasoning tasks such BFS order and Di ammetrr calculation (Table 14). This result highlights the value of incorporating explicit domain knowledge into prompts, particularly when tasks require step-by-step algorithmic reasoning. From Di ameter
7

---

Published as a conference paper at ICLR 2026
Parformance by Prompt Schemes
Performance by Prompt Schemas and Tasks
1,0
Diameter
Shorteat
Mean Acc
0.6
Mean Acc
0.6
Triangle
0.4
0.4
0.2
Indo ud
CoT
Kathn
Algarehin
D-Algerthin
G-shot
ITM
einitrect
B-Cot
Algoritii
G-Cot
Cot
intrrd
e-intruu
Alggrrett a-shat
R-shot
Prompt Schemes
Prompt Schemes
(a) Open-source models.
(b) Closed-source models.
Figure 5: Accuracy of open-source versus closed-source models with different prompt schemes. (a) and (b) show the average performance with a 95% confidence interval for open/closed-soute source models across various prompt schemes and tasks, with z-axis sorted by mean accuracy.
cal clluaaiion case I and Triangle counting case 1, it shows that when employing plain prompts, the LLM's response does not accurately reflect the appropriate method for solving the relevant task.
Finding @: Scaling raises the floor, while reasoning lifts the ceiling. A targeted comparison of Qwen-2.5 (7B), Qwen-2.5 (72B), and Qwen-3 (8B) (Table 11) highlights complementary effects. Scaling within the same family (7B to 72B) yields consistent improvements splits, such as BFS order, Shortest path, and Diameter calculation (Easy/Medium). By contrast, reasoning model comparable size, i.e., Qwen-3 (8B), delivers larger gains the hardest regimes that require multi-step exploration and combinatorial checks, including BFS order, Di ameter calculation and Triangle counting (Harrd). Together, these results indicate that scaling predominantly improves robustness on simpler instances, while reasoning-centric design is more effective for pushing the upper bound of graph reasoning ability (details in Appendix C.5).
easier tasks and
Finding @ Divergent impacts of prompt schemes Open-source models benefit from multi-shot exemplars, whereas they do not help closed-source models much. In Figures 5a and 5b, the open-source model achieves the highest average accuracy with prompt schemes that incorporate shots. However, for the closed-souree model, prompt schemes show more complexity. Only considering prompt patterns, 0-CoT performs second to best, 0-Algori thm worst, but both surpass k-shot. However, adding shots improves Algori thm's overall accuracy, suggesting that shots enhance the model'ss 's understanding of Algorithm-basee -basch prompts. Yet, this effect is not universal: shots may hinder comprehension in particularly challenging tasks, as noted in Finding Appendix E.6.
4.3 EXTENDED STUDY AND DISCUSSION
Scaling to Larger Graphs (Beyond 30 Nodes). We extend the evaluation to graphs with 30-50 nodes, sampling 50 graphs per generator and ~33 test cases overall (details in Appendix C.1). As the results in Table show, the performance degrades as graph size increases, particularly for tasks with sequential combinatorial requirements: accuracy BFS order and Triangle counting drops sharply, reflecting the added difficulty of maintaining frontiers or enumerating subgraphs over longer horizons. By contrast, tasks such as Connective vi ty and Cycle detection remain relatively stable, consistent with their reliance on local connectivity checks. Importantly, despite the absolute drop in scores, the relative ranking of models and the performance gap between open- and closed-source systems remain nearly identical to the 5-30 node Hard split, confirming that the benchmark's 'ss conclusions are robust under further scaling of graph size.
Representative Check on Real-World Graphs. We further test whether our synthetic setup transfers to real data by evaluating on two representative domains: IMDB-MULTI (social/interaction) and ogbg-molhiv (molecular), yielding ~3.66 samples across six tasks (details in Appendix C.2). Results Table 9 corroborate our findings: (i) Connectivity and Cycle detection are consistently easiest; (ii) ordered-pan tasks (BFS order, Shortest path, Diameter calculation) remain substantially harder, dominated by serialization and memory errors; and (iii) Triangle counting is the most challenging. However, because many public graphs are sparse and connected, specific tasks become easier than in our synthetic regime (e.g.. Connectivity saturates near 100% for strong models). This shows that real-world graphs alone under-stress graph reasoning. Together with prior works
8

---

Published as a conference paper at ICLR 2026
that adopt synthetic-only designs (Fatemi al., 2024; Chen et al., 2024a; Luo et al., 2024b), our results validate real graphs as a sound check, but reaffirm that synthetic graphs provide a systematic evaluation with balanced structural coverage, controllability, and contamination-fiee conditions. The detailed rationale is elaborated in Appendix C.3.
Exploration on NP-Hard Tasks. As complementary stress test, also consider two classical NP-hard problems, Hamiltonian cycle detection and Max-Cut (details in Appendix C.4). Results in Table 10 show accuracy patterns aligned with our six canonical tasks: open-souree models remain near random, while closed-source still imperfect scores. This indicates that the core conclusions of GRAPHOMNI naturally extend to NP-hard problems, Interestingly, however, LLMs do not exhibit the same graded difficulty separation between polynomial-time and NP-hard tasks as human solvers: accuracy tends to collapse uniformly across NP-hard regimes just like polynomial tasks. Thus, while useful as complementary check, NP-hard tasks do not add progressive challenge the same way as our tractable yet demanding suite, reinforcing why the latter remain the centerpiece of GRAPHOMNI.
maasoning-orientee
models attain noticeably higher but
Efficiency-accuracy trade-off. Besides accuracy, we also analyze inference efficiency by measuring the number of output tokens produced across models (details in Appendix E.7). The results reveal a clear trade-off: accuracy gains often come at the cost of longer responses, but models navigate this balance differently. Closed-soure models (c.g.. GPT-4o, Claude-3.5) reach high accuracy with compact generations under 300 tokens, while 04-mini relies on very long chains of thought (over 1.6K tokens) achieve similar accuracy (Figure 32). By contrast, open-source models such Llama- 3.1 and Qwen-2.5 (7B) must generate substantially longer outputs to achieve high performance, whereas shorter responses are correlated with lower accuracy. These trends persist across difficulty levels, task types, serialization formats, and prompt schemes (Tables 23-26). Overall, efficiency, measured by output length, emerges as an additional axis of divergence across LLMs, reinforeing the importance of evaluating not only correctness but also the cost of achieving it.
4.4 REINFORCEMENT LEARNING (RL)- BASED PROMPT SEARCH INSPIRED BY GRAPHOMNI
Our benchmark evaluates three key dimensions, graph type, serialization format, and prompt scheme, to underscore the critical role of transforming graph structures into textual inputs for LLM inference. While GRAPHOMMiprovides comprehensive insights into how different dimensions affect LLM inference, we still face concrete, actionable question: Given many interacting dimensions, which configuration is best for a specific graph reasoning task? In this section, we want to identify the optimal combination strategies (serialization format; prompt scheme, etc.) that enhance the effectiveness of textual representations, thereby improving LLM performance in graph reasoning and understanding tasks. We define the process of converting graph structures into textual inputs tailored to a specific task as the serialization process, To operationalize this serialization process, we introduce an RL-based search method as diagnostic tool within our benchmark, enabling automatic selection of effective serialization strategies.
Specifically, RL transforms optimizing the serialization process into a sequential decision- problem for each type and difficulty of the task. There are T decision epochs, and each decision epoch determines one component of the serialization strategy. Then we provide a predetermined order to specify a sequence of action spaces (AA We set the initial state so as the specific type and difficulty of the task. Then at decision epoch action a, E A, based on the previous actions a1,.... 12....,a-1. a--1- Then the
(e.g., A, can be all candidate prompts).
t= 1,..,,, T. we choose state 8, consists of the task type and difficulty (initial state sy) together with the previously selected serialization components. This corresponds to a policy ne : Su A1 X ... X A + Ae, where Su is the state space of the initial state 8p- For any instance s (e.g., query for Connectivity task in easy mode for specific graph), a binary reward, denoted by r(s, (1,- , ar), decision epoch, which is set to if the LLM correctly answers the specific query under the selected serialization strategy (a) (aj...., .T) and otherwise. For each type and difficulty of the task, our objective is to maximize the expected reward of choosing the serialization strategy dI,..., ar:
is incurred at the end of the
max { Jest.......
E[r(s, a1...., |sol,
where the expectation is taken with respect to the problem instance and the (random) answer output by an LLM (affected by the randomness of the LLM, e.g., the temperature parameter). Note that (i) 8o is part of the instance information 8, and (ii) we fix the type and difficulty of the task, and the

---

Published as a conference paper at ICLR 2026
only randomness in terms of is from graph generation. To approximate this objective function, we generate N different graphs for each type of query. We assess the performance RL using the average reward across the N graphs, which essentially is the accuracy of the serialization strategy for a specific graph-related task across these N graphs.
Consider the problem of dealing with high-dimensional, complex state spaces in the serialization process, we employ the Deep Q-Network (DQN) (Mnih et al., 2013) to implement RL. which employs neural network
function approximator for the Q-function. Specifically, we use a neural network (s0 .i...... 0e; 0x) parameterized by O, to approximate the corresponding Qr(sp, ai...., ae) for the actions or factors considered in serialization process. Each Q-network is modeled
three-layer multilayer perceptron with ReL.U activations. Training minimizes the mean squared error loss, and action selection follows e-greedy policy, where linearly decays from 1.0 to a minimum of 0.01. Then we design the RL-Opt (RL-guided Optimal Serialization Selection) experiment, leveraging existing benchmark data apply RL evaluate computational cost and validate the effectiveness of the derived optimal strategy. Additionally, we introduce the RL-Scale (RL Scalability in Serialization Expansion) experiment to analyze how RL's computational cost scales when incorporating additional factors in the serialization process. All detailed information can be found in Appendix D.
In RL. the serialization process involved three key factors based on benchmark's s results: serialization format, prompt scheme, and the choice of open-source language models. To evaluate the effectiveness of RL in identifying the optimal combination, we employ two key metries: Cost and Rate. To evaluate RL's effectiveness in finding the optimal combination, we use two metrics: (a) Cost is the ratio of explored combinations: Cost K where is the number of explored combinations, and K is the total number of combinations; (b) Rate = ses., where acc, is the accuracy of RL's best-found combination and aCCmax is the highest accuracy in the benchmark data, Results are in Table 4. The results demonstrate that, at only 25% the original cost, the RL--based method maintains an average success rate of 0.9, indicating its ability to significantly reduce the time required to search for optimal combinations while preserving the quality of the outcomes.
acC
Table 4: Performance summary of RL-Opt, averaged across all instances of a specific experimental case, reducing the cost to about 25% of the original, maintaining an average success rate of 0.9.
Task
Mode
Avg Cost
Avg Rate
Task
Mode
Avg Cost Avg Rate
Easy
0.2203
0.9740
Fasy
0.2244
0.9883 0.9875
BFS order
Medium
0.2251
0.9045
Connectivity Medium
0.2263
Hard
0,2279
0.7812
Hard
0.2238
0.9871
Easy 0.2229
0.9757
Fasy 0.2263
0.9728
Cycle
Medium
0.2263
0,9833
Diameter
Medium
0.2181
0.9541
Hard
0.2203
0,9584
Hard
0.2235
0.9471
Easy 0.2244
0.9636
Easy 0.2276
0.9061
Shortest path Medium
0.2159
0.9856
Triangle
Medium
0.2206
0.8456
Hard
0.2187
0.9073
Hard 0.2235
0.7321
5 CONCLUSION
We introduced GRAPHOMNI, a comprehensive benchmark framework for systematically evaluating the graph reasoning capabiliiies of L.MMs. By analyzing critical dimensions, including graph types, serialization formats, and prompt schemes, we provided extensive insights into the strengths and limitations of current LLMs. Our empirical findings emphasize that no single serialization or prompting strategy consistently outperforms others. Motivated by these insights, we propose a reinforcement learning-basee approach that dynamically selects the optimal serialization-promee pairings, leading to significant improvements in accuracy. GRAPHOMNI's 's modular and extensible design establishes a robust foundation for future research, facilitating advances toward general- purpose graph reasoning models.
10

---

Published as a conference paper at ICLR 2026
Ethics Statement We confirm that this research complies with all applicable ethical guidelines and poses no ethical issues.
Reproducibility Statement We have taken extensive measures ensure the reproducibility of our work. The source code and data resources are released at https:/gai--ommmuity www.i/uuuuntttggtuiio.. github.io/ Graph-Omni/.
Our experimental setup, including model configurations and evaluation protocols, is fully described in Section 3 and Appendix A. For transparency, we provide comprehensive coverage of oututt examples (Appendix E.5) and error cases (Appendix E.4), enabling a thorough understanding and verification of the reported results.
Together, these resources support faithful reproduction and further exploration of our findings.
REFERENCES
W. Aiello, F, Chung, and L. Lu. A random graph model for massive graphs. In Proceedings of the 32nd Annual ACM Symposium on Theory of Computing (STOC), pp. 171-180, 2000. doi: 00.1145/355305.333320
R. Albert and A.-L. Barabási. Emergence of scaling in random networks. Science, 286(5439): 509-512, 1999. doi: 10.1166/sciemee28..5499...
Ulrik Brandes, Markus Eiglsperger, Jürgen Lerner, and Christian Pich. Graph markup language (graphml). 2013.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Von Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. arXiv preprint arXiv:2310.05845, 2023.
Deepayan Chakrabarti and Christos Faloutsos. Graph mining: Laws, generators, and algorithms.
ACM Comput. Surv., 38(1):2-cs, June 2006. ISSN 0360-0300. doi: 10.922 URL https://di..rr//10. 1145/1 132952.1 132954.
Nuo Chen, Yuhan Li, Jianheng Tang, and Jia Li. Graphwiz: An instruction language model for graph computational problems. In Proceedings of KDD 2024, 2024a. doi: 10. 15.154//637528. 3672010.
Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. Llaga: Large language and graph assistant. arXiv preprint arXiv:2402.081777 2024b.
Xinnan Dai, Haohao Qu, Yifen Shen, Bohang Zhang, Qihao Wen, Wenqi Fan, Dongsheng Li, Jiliang Tang, and Caihua Shan. How do large language models understand graph patterns? benchmark for graph pattern comprehension. arXiv preprint arXiv:2410.05299, 2024.
Debarati Das, Ishaan Gupta, Jaideep Srivastava, and Dongyeop Kang. Which modality should i use text, motif, or image? Understanding graphs with large language models, 2024. 1tXiv:231.0098622
David Easley, Jon Kleinberg, et al. Networks, crowds, and markets: Reasoning about highly connected world, volume 1. Cambridge university press Cambridge, 2010.
Erdōs and A. Rényi. On the evolution of random graphs. Publications of the Mathematical Institute of the Hungarian Academy of Sciences, 5:17-61, 1960.
11

---

Published as a conference paper at ICLR 2026
Xiang Fang, Arvind Easwaran, Blaise Genest, and Ponnuthurai Nagaratnam Suganthan. Adaptive hierarchical graph cut for multi-gra mualti-raaullrity ofto----iittibttionn detection. IEEE Transactions on Artificial Intelligence, 2025a.
Yi Fang, Dongzhe Fan, Sirui Ding, Ninghao Liu, and Qiaoyu Tan. Uniglm: Training one unified language model for text-attributed graph embedding. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining (WSDM), 2025b. doi: 10.1145/3701551. 3703586.
Bahare Fatemi, Jonathan Halerow, and Bryan Perozzi. Talk like a graph: Encoding graphs for large language models. In Proceedings of ICLR 2024, 2024.
Aarash Feizi, Sai Rajeswar, Adriana Romero-Soriano, Reihaneh Rabbany, Spandana Gella, Valentina Zantedeschi, and João Monteiro. Pairbench: A systematic framework for selecting reliable judge vlms. arXiv preprint arXiv:2502.15210, 2025.
Yifan Feng, Chengwu Yang, Xingliang Hou, Shaoyi Du, Shihui Ying, Zongze Wu, and Yue Gao. Beyond graphs: Can large language models comprehend hypergraphs? In The Thirteenth Inter- national Conference on Learning Representations, 2025. URL https:/oppenevie..eefffmmmm 28q0Qwjuma.
Thomas Gaudelet, Ben Day, Arian R Jamasb, Jyothish Soman, Cristian Regep, Gertrude Liu, Jeremy BR Hayter, Richard Vickers, Charles Roberts, Jian Tang, et al. Utilizing graph ma- chine learning within drug discovery and development. Briefings in bioinformatics, b2,6),bbab159 2021.
E. N. Gilbert. Random graphs. The Annals of Mathematical Statistics, 30(4):1141-1144 1959. doi: 10.1214/aam//11777666999
Jayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. 2024a. arXivv2305. 150666...
Kai Guo, Zewen Liu, Zhikai Chen, Hongzhi Wen, Wei Jin, Jiliang Tang, and Yi Chang. Learning on graphs with large language models (Ilms): A deep dive into model robustness. arXiv preprint arXiv:2407. /2068, 2024b.
Shuo Han, Yukun Cao, Zezhong Ding, Zengyi Gao, S Kevin Zhou, and Xike Xie. See or say graphs: Agent-driven scalable graph structure understanding with vision-language models, arXiv preprint arXiv:2510.1676 2025.
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. G-retriever: detrieval-augmented generation for textual graph understanding and question answering. Advances in Neural Information Processing Systems, 37:132876-132907, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https://opemreriew.ee/ffoumu Xttpr//eeweee.eeoomooomdda/////a Kdd=dd
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fffth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021b. URL http:://weeeeeeeeeeeee//o//d////
Weihua Hu, Matthias Fey, Marinka Zitmik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: datasets for machine learning on graphs. In Proceed- ings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829544
Yuntong Hu, Zheng Zhang, and Liang Zhao. Beyond text: A deep dive into large language models' ability understanding graph data. arXiv preprint
2023.
12

---

Published as a conference paper at ICLR 2026
Xiangru Jian and Yimu Wang. InvGC: Robust cross-modal retrieval by inverse graph convo- lution. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 836-865, Singapore, December 2023, As- sociation for Computational Linguistics. doi: 10.18653/11/0223. htwt:////020..nnnnn-eeem...... URL
Xiangru Jian, Zhengyuan Dong, and M. Tamer Özsu. Interacsparql: An interactive system for sparql query refinement using natural language explanations, 2025a. URL https://anxiv.or//bb/5551 02002.
Xiangru Jian, Xinjian Zhao, Wei Pang, Chaolong Ying, Yimu Wang, Yaoyao Xu, and Tianshu Yu. Rethinking spectral augmentation for contrast-based graph self-supervised learning. Transactions on Machine Learning Reseurch, 2025b. ISSN 2835-8856. URL http:///ponnereiieeeeffoormmmn/ HjpD5kpfa3.
Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han. Large language models on graphs: A comprehensive survey. In Transactions on Knowledge and Data Engineering (TKDE), 2024a.
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. Graph chain-of-thought: Augmenting large language models by reasoning on graphs. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 163-184, 2024b. ACL 2024.
Lecheng Kong, Jiarui Feng, Hao Liu, Chengsong Huang, Jiaxin Huang, Yixin Chen, and Muhan Zhang. Gofa: generative one-for-all model for joint graph language modeling. arXiv preprint arXiv:2407.09709 2024.
M. Latapy, C. Magnien, and N. Del Vecchio. Basic notions for the analysis of large two-mode networks. Social Nenworks, 30(1):31-48, 2008. doi: 10.1016/j.socnet.200 2007.000000.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising lequence-to-sequeee pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10. (t.18653/v1/2220.ce wtt.://....lmmi........ 703. URL https://ccannhologg.or////// acl- main. 703/.
Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, et al. Graphreader: Building graph- agent enhance long-context abilities of large language models. arXiv preprint arXiv:2406. 14550, 2024a.
Xin Li, Weize Chen, Qizhi Chu, Haopeng Li, Zhaojun Sun, Ran Li, Chen Qian, Yiwei Wei, Zhiyuan Liu, Chuan Shi, Maosong Sun, and Cheng Yang. Can large language models analyze graphs like professionals? benchmark, datasets and models. In Proceedings of the 38th Conference Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks, 2024b. URL: http://hhhhbbooo///UUGGGGAAAAAA
Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, and Jeffrey Xu Yu. survey of graph meets large language model: Progress and future directions. arXiv preprint arXiv:2311.12399 2023.
Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. Zerog: Investigating cross-datase zero-shot transferability in graphs. In Proceedings of the KDD Conference, 2024c. doi: 10.1
3637528.3671982.
Yuhan Li, Peisong Wang, Xiao Zhu, Aochuan Chen, Haiyun Jiang, Deng Cai, Victor W Chan, and Jia Li. Glbench: A comprehensive benchmark for graph with large language models. Advances in Neural Information Processing Systems, 01442399-42366 2024d.
Yuankai Luo, Hongkang Li, Qijiong Liu, Lei Shi, and Stao-Ming  Wu. Node identifiers: Compact, discrete representations for efficient graph learning- arXiv preprint arXiv:2405. /6435, 2024a.
13

---

Published as a conference paper at ICLR 2026
Zihan Luo, Xiran Song, Hong Huang, Jianxun Lian, Chenhao Zhang, Jinqi Jiang, and Xing Xie. Graphinstruct: Empowering large language models with graph understanding and reasoning capabillity 2024b. 24.442240..04483v2
Haitao Mao, Zhikai Chen, Wenzhuo Tang, Jianan Zhao, Yao Ma, Tong Zhao, and Neil Shah. Position: Graph foundation models are already here. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs, 2020. URL
hutps//aariiv.rg//bb/20000..... 006663.
Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami AI-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for llms, arXiv preprint arXiv:2402.05862, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), January 2020. ISSN 1532-4435.
Clayton Sanford, Bahare Fatemi, Ethan Hall, Anton Tsitsulin, Mehran Kazemi, Jonathan Halcrow, Bryan Perozzi, and Vahab Mirrokni. Understanding transformer reasoning capabilities via graph algorithms. Advances in Neural Information Processing Systems, 37:78320-78370, 2024.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features prompt design or: How learned to start worrying about prompt formatting. arXiv preprint arXiv:2310. 1324, 2023.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design. In International Conference on Learning Representations (ICLR), 2024.
Chengshuai Shi, Kun Yang, Zihan Chen, Jundong Li, Jing Yang, and Cong Shen. Efficient prompt optimization through the lens of best arm identification. arXiv preprint 2XXi:22422.09233 2024.
Rok Sosic and Jure Leskovec. Large scale network analytics with snap: Tutorial at the world wide web 2015 conference. In Proceedings of the 24th International Conference on World Wide Web, WWW '15 Companion, pp. 1537-1538, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450334730. http:://oii.rr// 14/11114/77490.822222
doi: 10.1 001145527400008.24447000 URL
Xing Su, Shan Xue, Fanzhen Liu, Jia Wu, Jian Yang, Chuan Zhou, Wenbin Hu, Cecile Paris, Surya Nepal, Di Jin, et al. A comprehensive survey community detection with deep learning. IEEE transactions on neural networks and learning systems, 3584)2468--470, 2022.
Yanchao Tan, Zihao Zhou, Hang Lv, Weiming Liu, and Carl Yang. Walklm: A uniform language model fine-tuning framework for attributed graph embedding. In Proceedings of NeurIPS 2023, 2023.
Yanchao Tan, Hang Lv, Xinyi Huang, Jiawei Zhang, Shiping Wang, and Carl Yang. Musegraph: Graph-oriented instruction tuning of large language models for generic graph mining. arXiv preprint arXiv:2403. 04780, 2024.
Jiabin Tang, Yuhao Yang. Wei Wei, Lei Shi, Lixin Su, Sugi Cheng, Dawei Yin. and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 491-500, 2024.
14

---

Published as a conference paper at ICLR 2026
Jianheng Tang. Qifan Zhang, Yuhan Li, Nuo Chen, and Jia Li. Grapharena: Evaluating and exploring large language models on graph computation. In The Thirteenth International Conference on Learning Representations, 2025. URL http:://pppmnerieeww/////om/////
Alexander K. Taylor, Anthony Cuturrufo, Vishal Yathish, Mingyu Derek Ma, and Wei Wang. Are large-language models graph algorithmic reasoners?, 2024. arXiv:2410.2259v1.
Heng Wang. Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov.
Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 04.30840-3086 2023.
Jianing Wang, Junda Wu, Yupeng Hou, Yao Liu, Ming Gao, and Julian McAuley. Instructgraph: Boosting large language models via graph-centric instruction tuning and preference alignment. arXiv preprint (rXXi:2402.087885 2024a.
Rui Wang and Chunlin Zhuang. Graph neural networks driven acceleration in drug discovery. Acta Pharmaceutica Sinica B, 15(12)66163-617) 2025. ISSN 2211-3835. doi: https://doi.org/1000000 URL
j.apsb.2025. 10.011.
Yu Wang. Ryan A Rossi, Namyong Park, Huiyuan Chen, Nesreen K Ahhed, Puja Trivedi, Franck Dernoncourt, Danai Koutra, and Tyler Derr. Large generative graph models. arXiv preprint arXiv:2406.05109 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys, 55(5):1-37, 2022.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1) 1)44 24, 2020.
Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. arXiv preprint arXiv:2403.01 2024.
Yuhao Xu, Xinqi Liu, Keyu Duan, Yi Fang, Yu-Neng Chuang, Daochen Zha, and Qiaoyu Tan. Graphfm: A comprehensive benchmark for graph foundation model. arXiv preprini arXiv:2406.088 083 10, 2024.
Christos Xypolopoulos, Guokan Shang, Xiao Fei, Giannis Nikolentzos, Hadi Abdine, Iakovos Evdai- mon, Michail Chatzianastasis, Giorgos Stamou, and Michalis Vazirgiannis. Graph linearization methods for reasoning on graphs with large language models. arXiv preprint arXiv:2410. 19494, 2024.
Hao Yan, Chaozhuo Li, Ruosong Long, Chao Yan, Jianan Zhao, Wenwen Zhuang. Jun Yin, Peiyan Zhang, Weihao Han, Hao Sun, et al. A comprehensive study on text-attributed graphs: Bench- marking and rethinking. Advances in Neural Information Processing Systems, 36:17238-17264, 2023.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018
Qifan Zhang, Xiaobin Hong, Jianheng Tang, Nuo Chen, Yuhan Li, Wenzhong Li, Jing Tang, and Jia Li. Gcoder: Improving large language model for generalized graph reasoning. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM '25. pp. 4149-4159, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400720406. doi: 10. 00.1145/7446552.37666666
URL httpsh/ts////ooorr/ 10. 145/3746252.376 (166.
Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. Can Ilm graph reasoning generalize beyond pattern memorization? In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 2289-2308 2024a.
15

---

Published as a conference paper at ICLR 2026
Zeyang Zhang, Xin Wang, Ziwei Zhang, Haoyang Li, Yijian Qin, and Wenwu Zhu. Llm4dyg: Can large language models solve spatial-temporal problems on dynamic graphs? Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining (KDD 24). ACM, 2024b. doi: 10.1 /41114//677588867777077
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. arXiv preprint arXiv:231000108. 2023.
Xinjian Zhao, Wei Pang, Zhongkai Xue, Xiangru Jian, Lei Zhang, Yaoyao Xu, Xiaozhuang Song, Shu Wu, and Tianshu Yu. The underappreciated power of vision models for graph structural understanding. In The Thith Annual Conference on Neural Information Prucessing Systems, 2025. URL
Xinjian Zhao, Wei Pang. Zhixuan Yu, Xiangru Jian, Xiaozhuang Song, Yaoyao Xu, Zhongkai Xue, Dingshuo Chen, Shu Wu, Philip Torr, and Tianshu Yu. When vision meets graphs: A survey on graph reasoning and learning. February 2026. doi: 10.36
URL http://dx.doiiioommo org.:///dooooo///////////////// 10..:6277/eccnxiii.117
Lyuyi Zhu, Qixin Zhang, Xiangru Jian, and Yu Yang. Graph convolutional network for traffic incidents duration classification. Engineering Applications of Artificial Intelligence, 151:110570 2025. ISSN 0952-1976. doi: https://doi.orgg/ 1016
httproddictttooossssimeettttttip http://oo........oooooooop....a. www. 100500. URL https://www.
Lyuyi Zhu, Qixin Zhang, Xiangru Jian, Yu Yang, and Lishuai Li. Spatio-temporal traffic accidents detection via graph based generative adversarial network. Engineering Applications of Artificial Intelligence, 165:1 13488, 2026. ISSN 0952-1976. doi: https://dooi..rg/10.. 101 wt..g//aaaaaiii.0000....
113488. URL http:///www//oooocooooooooootooo
16

---

Published as a conference paper at ICLR 2026
Table of Contents for Appendix
Page
A. Experimental Details
A.1 LLM Versions
18
A.2 Parameter and Random Baseline Settings
18
A.3 Graph Tasks
19
A.3.1 Rationale for Selection Tasks
19
A.4 Graph Types
20
A.4.1 Rationale for Generator Selection
21
A.5 Prompt Format
22
A.6 Serialization Format
22
A.7 Data Examples
24
B. Benchmark Statistics
27
B.1 Basic Statistics GRAPHOMNI
27
B.2 Token Statistics of GRAPHOMNI
C. Extended Discussion and Ablation Study of GRAPHOMNI
29
C.1 Study Larger Graph (Beyond 30 Nodes).
29
C.2 Study on Real-Wordd Graphs: Representative Check
30
C.3 Considerations on Real-Wor World Graphs vs. Synthetic Graphs
C.4 Exploration on NP-Hard Tasks
31
C.5 Scaling vs. Reasoning: Disentangling Their Effects Graph Reasoning
32
C.6 Rationale for Binary Metric over Partial Score
34
RL-based Prompt Search Inspired by GRAPHOMNI
D.I Background and Serialization Process
34
D.2 Details for RL-Opt Setting
35
E. Comprehensive Experimental Results
38
E.1 Fine-grained Results Across Dimension
38
E.2 Performance Heatmaps across Tasks
42
E.2.1 Heatmaps for BFS order
42
E.2.2 Heatmaps for Connectivity.
47
E.2.3 Heatmaps for Cycle detection
E.2.4 Heatmaps for Diameter calculation
53
E.2.5 Heatmaps for Shortest path
56
E.2.6 Heatmaps for Triangle counting
59
E.3 Graph Type Sensitivity Analysis
E,4 Error Analysis
69
E.5 Input/Output Examples
77
E.6 More Findings from Evaluation Results
102
E.7 Analysis on Efficiency via Number of Output Tokens
103
F. Detailed Related Works
105
F.1 LLM Applications Graph Data
106
F.2 Benchmarks LLM Application Graph Data
107
G. Limitations and Future Directions of GRAPHOMNI
107
H. Additional Ablation Studies
108
H.1 Performance v.s. Time Complexity of Tasks
108
H.2 Scaling Beyond 50 Nodes
109
H.3 Robustness Check under Prompt Noise (Perturbation)
110
17

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
A.1 LLM VERSIONS
Table 5 provides an overview of the diverse suite of large language models (LLMs) evaluated in our study. Open-source models hyperlinked to their respective documentation, while closed-source models are identified by their version numbers. Note that we only uniformly sample 25% of data when evaluating Qwen-3 due the limited time after its release, its result will be only included in the model-wise statistics, i.e. Table 3 for refernece.
Table 5: Overview of evaluated LLMs. Open-source models are linked, while closed-source models list their version.
Model
Model Link/Version
Llama-3
Meta-Llamaa-3888 (Link)
Llama-3.1
Llama-3.1-8D (Link)
Mistral
Mistral-7B-v0.3 (Link)
Phi-4
Phi-4-14B (Link)
Qwen-2.5 (7B)
Qwen-2.5-7B-Instru. (Link)
Qwen-2.5 (72B)
Qwen-2.5-72BB-nntrree (Link)
Qwen-3 (8B)
Qwen-3-8B (Link)
Claude-3.5
claude3355-ommet-2224100
Gemini-2.0
gemini-2.0-fllahhe (Version
GPT-4o
901-40-2024-8--66
GPT-4o-ming
gpt-4o-mini-2024-07
04-mini
04-mini--225-044-2
A.2 PARAMETER AND RANDOM BASELINE SETTINGS
Parameter setting. We have studied various methods of representing graphs as text based on diverse set basic graph problems. This appendix details the parameter setting and the design of the graph input text. For the parameter setting, the temperature is set to 0.7, following the parameter selection in Wang et al. (2023). The nucleus sampling (top-p) set to 0.9 for open-source models, while for closed-source models, the default top-p value is used.
Random Baselines setting. For Cycle detection, the random bascline simply selects an answer from (True, False --yilldng expected accuracy of 50%. Since the GT obtained through the design function has a high proportion of True labels, we iterate through all queries, assuming the given answer is True. We then GT for evaluation, leading to the final baseline based on this assumption. For tasks that require generating numerical outputs (e.g., Diameter cal culatiionn inlation and Tri angle counting), the random baseline corresponds randomly choosing numerical solutions derived from the graph's 's structure. For the Diameter calculate clluaaiion task, the random baseline is determined based on the number of nodes in the graph for each query. Specifically, we sample a random integer from the range N.] where N is the number of nodes in the graph, and compare it with the ground truth compute the baseline performance. For the Triangle counting task, the random baseline is derived from the estimated upper bound on the number of triangles in the graph. We compute the maximum possible number of triangles based on the number of nodes and the task difficulty level, take the smaller value between these estimates, and sample a random integer from the range [1, M). where M the determined upper bound. The sampled value is then compared against the ground truth obtain the random baseline performance. In contrast, for tasks that require generating sequences (c.g., BFS order), the number of possible combinations is combinatorially large, so random baseline would yield an accuracy that is approximately 0%.
of the valid
18

---

Published as a conference paper at ICLR 2026
A.3 GRAPH TASKS
We conducted a comprehensive study on a diverse set of fundamental graph problems, including BFS order, Cycle detection, Connectivi ty,, Diameter calculation, Shortest path, and Triangle counting. The input text for each task is provided below, where the italicized variables X, y denote generic node numbers corresponding to the specific problem under consideration.
Graph Tasks
BFS-ORDER: Give the bfs traversal order starting from node X.
CYCLE: there cycle this graph?
CONNECTIVIT:: there path between node and node Y?
DIAMETER: What the diameter of this graph?
SHORTEST PATH: Give the shortest path from node X to node Y.
TRIANGLE: How many triangles are in this graph?
A.3.1 RATIONALE FOR SELECTION OF TASKS
The six core tasks in GRAPHOMNI are deliberately selected to span qualitatively different reasoning capacities. Their difficulty increases as the model must move from local checks to global traversals, maintain more intermediate states in working memory, or perform exhaustive combinatorial enu- meration. Beyond reasoning capacities, variation also arises from how well LLMs internalize task definitions and from the complexity of output formats. Together, these factors explain the accuracy gaps observed in Table and highlight why the chosen tasks form a balanced and challenging suite.
Aspect 1: Reasoning capacities required. These tasks are grouped according to the type and depth of reasoning they demand, ranging from simple global checks to multi-layered traversals and full combinatorial cnumeration.
Here follows detailed elaboration on these three aspects.
1. Reachability verification (Connectiv ty, Cycle detection). These tasks require a global traversal but only simple decision condition, such as whether the graph connected or whether cycle is present. Most errors stem from serialization misunderstandings (c.g.. assuming a missing edge exists, in Appendix E.4.3). Once the format is parsed correctly, accuracy is high.
2. Ordered-path reasoning (BFS order, Shortest path, Diameter calculation),
These tasks demand that the model keep frontier distance map and then output compare those ordered distances. For BFS order, the model must list nodes level-by-leved In the error case in Appendix E.4.7, failures occur when it forgets whether two previously visited nodes are connected. Shortest path and Di Diameter calculation add a final aggregation step: the former selects the minimum path, the latter the maximum among all shortest paths. The common mistakes are also mostly about losing track of some vital information while exploring the graph. Like the one in Appendix E.4.2 for Diameter calculation iillaiion, the model forgets two important edges, so the path length is wrong, Accuracy here for those three tasks is lower than the first type of tasks because the model must track ordering information across multiple expansion layers.
3. Combinatorial enumeration (Triangle counting).
Triangle counting is the most challenging: the model must evaluate every three-node subset and make sure each sub-traverse correct. Even given correct execution of the emu- meration, the counting should be accurate to produce the correct final result. Appendix E.4.6 and E,4.8 document the dominant errors on cnumeration over each possible triangle in the graph (like missing edge wrongly assuming one). We also spotted cases that fail on the counting at the end, too. In sum, performance is strongest when only reachability is tested, drops when ordered path reasoning is required, and falls sharply when complete combinatorial enumeration comes into play.
Aspect 2: Task understanding and definition knowledge. LLMs sometimes rely on heuristics rather than precise textbook definitions, particularly for less common tasks. For example, some models
19

---

Published as a conference paper at ICLR 2026
confuse diameter with the longest simple path, producing inflated results (Appendix E.4.1). Others apply shortcuts such as "triangles & n/3"" (Appendix E.4.5), ignoring the need for all three edges to be present. Such misinterpretations highlight that accuracy depends not only raw reasoning ability but also on task comprehension. Our coverage of tasks enable the evaluation on these knowledge of each model and does reflect in the results as the error cases mentioned.
Aspect 3: Output format. The output formats of the tasks chosen are also very diverse. Some tasks here need only short answer, i.e., "Yes/No" for Connectivi ty or single number for Triangl: counting, so there is little room for formatting errors. Meanwhile, BFS order is different: the model must print long, strictly level-by-level list of node IDs, and one extra or missing node makes the whole response wrong. The coverage of different output formats brings challenges to the models.
In summary, these systematic differences validate that the GRAPHOMNI task suite probes diverse reasoning skills over graphs and exposes where current LLMs struggle most.
A.4 GRAPH TYPES
A primary distinguishing aspect of our benchmark is the inclusion of multiple graph families, each possessing unique structural properties. All 7 types of graph are highlight in bold:
1. Erdōs-Rényí (ER) Graphs are randomly sampled from the space of all possible graphs with n vertices, making them well-suited for capturing a wide range of topological and connectivity properties within a fixed number of vertices.
To enhance the diversity of random graphs, we consider two sampling methods: m-edge sampling and probability- sampling. referred to Erdös-Rényi M-Edges (ERM) (Erdös & Rényi, 1960) and Erdös-Rény: Rényi Probability (ERP) (Gilbert, 1959) respectively.
ERM: Generates graphs with n vertices and a fixed number of edges m, where m is ran-
domly chosen between 1 and m(n-1) ensuring that all possible edge counts are considered.
ERP: Constructs graphs with n vertices but an unfixed number of edges, where the edge probability is randomly sampled as dloating-poin value between 0 and 1.
Additionally, we extend these models to bipartite settings:
Bipartite Erdós-Rényí M-Edges (BERM) and Bipartite Erdös-Rényí Probability (BERP) graphs (Latapy et al., 2008) are generated using the ERM and ERP sampling strategies but constrained to bipartite structures.
These bipartite graphs introduce additional variations in topology and connectivity that standard ERM and ERP graphs, which are inherently undirected, may not capture.
Barabási-Albert Graphs (BAG) (Albert & Barabási, 1999) exhibit power-law degree distri- bution, where a small number of nodes (hubs) have significantly higher degrees, while most nodes have relatively few connections. Such structures frequently appear in real-world networks, including social and biological systems.
While ER graphs, being randomly sampled, may occasionally exhibit power-law degree distributions, BAGs explicitly model this phenomenon due to their practical prevalence.
BAGs are constructed by starting with a complete graph of mo vertices and incrementally adding nodes.
Each new node forms m connections, where m is proportional to the degrees of existing nodes (preferential attachment).
In our dataset, mo is randomly sampled with an upper bound of g. and is to mo 1.
Although BAGs generally capture power-law degree distributions, they do not always represent tree-like structures such as citation networks or hierarchical systems. To address this, we introduce Barabási-Albert Forests (BAF) (Albert & Barabási, 1999), which follow the same generation process as BAGs but enforce an acyclic structure, ensuring that the result is a forest (a set of trees) rather than single connected graph.
20

---

Published as a conference paper at ICLR 2026
3. Scale-Free (SF) Graphs (Aiello al., 2000) Another class of power-law networks that BAGs may not fully capture is general scale-free free (SF) networks. While all BAGs are SP, not all SF graphs are BA.
BAGs typically consist single connected component, whereas SF graphs can contain multiple disconnected components,
To represent SF graphs more comprehensively, we introduce a distinct SF graph generation process, different from BAGs.
Unlike BAGs, which are constructed through incremental growth and preferential attachment, SF degree-weighted graphs are generated using degree- weighted random connection strategy:
All vertices are created at once.
Edges are formed probabilistically, where the probability of connection is proportional to node degrees.
These fundamental differences in growth dynamics and edge formation result in SF graphs and BAGs capturing distinct topological properties. By including both, we enhance the diversity of our dataset.
These families challenge LLMs to adapt their reasoning across numerous topological extremes, from sparse bipartite graphs to highly connected ones. Although future expansions may include small-world graphs or others, this current selection already covers a rich array of structural profiles as elaborated in the next section.
A.4.1 RATIONALE FOR GENERATOR SELECTION
The seven generators in GRAPHOMNI are deliberately selected to provide the most comprehensive structural coverage possible within the 5-30 node range. Each generator encodes distinct mo- tif/structure observed in real-world networks, i.c. random connectivity, scale-free growth, bipartite affiliation, hierarchical trees other tendencies, ensuring that the benchmark spans all major regimes of graph organization. Even at this scale, the underlying generative biases remain evident and produce meaningful differences in task difficulty and model behavior. By relying controlled synthetic generators, GRAPHOMNI achieves balanced representation across families while isolating structural effects without the confounding noise of empirical data.
To be specific, the selected generators cover wide range of canonical structures:
1. Erdős-Rényi M-Edges (ERM) & Probability (ERP). Serve as canonical baselines for random connectivity, yielding binomial/Poisson degree distributions used extensively the study of biological and technological networks.
2. Bipartite ERM (BERM) & Bipartite ERP (BERP). Capture two-mode affiliation struc- tures, such as author paper and user-item systems, which exhibit realistic clustering and degree properties.
3. Barabasi-Albert Graphs (BAG). Model scale-free networks with hubs emerging via preferential attachment, mirroring the structure of the Internet, citation graphs, and social networks.
4. Barebási-Albert Forests (BAF). A specialization of the BA process that produces acyclic scale-free trees, modeling hicrarchical taxonomies such as phylogenies and organizational charts.
5. Scale-Free (SF) Graphs. Configuration-style models generate prescribed power-law de- gree sequences, often producing disconnected components akin to regional transport or communication subnetworks.
To further validate that these generators produce graphs with statistically distinct and meaningful properties, we conduct two empirical studies. First, we sample 1,000 graphs of 30 nodes each from the same Barabási-Albees
(BAG) and Rédōs-Rényi (ERP) generators used in GRAPHOMNI. As summarized in Table 6, the two models exhibit clearly different structural characteristics: BA graphs form hubs with high maximum degree and short paths, while ER graphs display uniform randomness
21

---

Published as a conference paper at ICLR 2026
with lower clustering and longer paths. Second, shown in Table 7 in Appendix B.1, even when node counts are fixed, the edge counts (and thus average degrees) vary substantially across generators, providing strong statistical evidence that the structural characteristics of these graph families are fundamentally distinct. Together, these results confirm that the design of GRAPHOMNI captures the essential structural diversity needed probe LLM reasoning.
Table 6: Comparison of structural statistics for 1,000 sampled graphs with 30 nodes. BAG graphs exhibit hub formation with high maximum degree and short paths, while ERP graphs display more uniform randomness.
Type
Max Degree
Clustering Coefficient
Avg Path Length
Barabási-Albes (BAG)
19.28 + 2.15
0.397 0.042
1.76 0.014
Erdös Erdös-Rényi (ERP)
10.43 + 1.35
0.199 0.044
2.07 0.099
A.5 PROMPT SCHEMES
The process of converting graph into a textual representation is referred to as the serialization process, which involves two primary considerations our study: the choice of serialization format and the selection of the prompting method. we employ total of nine distinct prompting methods: Algori thm, COT, k-shot, Instruct, 0-Shot(i.e. plain), 0-CoT, 0-Instruct, 0-Algori thm, and LTM. As outlined in the main text, the pairs Algori thm and 0-Algori CoT and 0-CoT, k-shot and 0-Shot, and Instruct and I-Instruct share a common structural format, with the first element in each pair incorporating additional 5 examples. A detailed description of the design for each of these prompting methods is provided below. In particular, for the algorithmic description components of Algori thm and 0-Algori thm, we primarily draw upon established methodologies in Wang et al. (2023) and illustrate them with an example derived from the BFS-order task.
Prompt format
0-CoT: Let's think step by step:
LTM: Let's break down this problem:
0-INSTRUCT: Let's construct graph with the nodes and edges first:
0-ALGORITHM: To determine the BFS (Breadth-First Search) traversal order, you need follow these steps: 1. Initialize: Start by choosing a starting node enqueue into queue. 2. Mark visited: Mark the starting node as visited to avoid reprocessing. 3. Traverse: While the queue is not empty: Dequeue node and add it to the traversal order. For each unvisited neighboring node of the dequeued node, enqueue it and mark it visited. 4.Continue the process until all reachahle nodes visited.
A.6 SERIALIZATION FORMATS
This study utilizes seven distinct yet commonly used graph representation formats: Adjacency Matrix, Adjacency List, Adjacency Set, Edge Set, Edge List, Graph Modeling Language (GMoL), and Graph Markup Language (GMaL). For the same graph, even when the underlying information remains consistent, the representation varies across different serialization formats in textual form. The following section presents specific examples of the same graph depicted in various serialization formats.
Adjacency Set
[0: (1), 1: (0, 2). 2: (1), 3: (4), 4: (3, 5), 5: (4))
Edge Set
[(0, 1), (4, 5), (1, 2), (3, 4))
22

---

Published as a conference paper at ICLR 2026
Edge List
1
3 4
4 5
Adjacency Matrix
C[e
0
0]
[1
8
8
0]
[0
1
8
8
0]
8
01
8
11
[0
8
03]
Adjacency List
[0: [1). 1: [e, 2]. 2: [1], 3: [4], 4: [3, 5], 5: [4]}
GMaL
<?xml version='1 1,0' encoding='utf-8'?>
<GMaL httne:hhttpp//lbaa..rrraaiririir
hll.shaaubbaatiioerttttte///aaaa .rgiinng.or///oogoooooooooooo xmlns org/xmlns
http://dchll..raanddaairneeeee/r 0/GMaL. xsd">
<graph edgeddefaull=""nddiretted""
<node id="0" />
<node id="1" />
<node id="2"
<node id="3"
<node id="4"
<node id="5" />
<edge source="Ø" target="1" />
<edge source="1" target="2"
<edge source="3" target="4" />
<edge "4urce="4" target="5" />
</graph>
</GMaL>
23

---

Published as a conference paper at ICLR 2026
GMoL
graph [
node
id
label "0"
]
node
id
label "1"
]
node
id
label "2"
]
node
id
label "3"
]
node
label "4"
]
node
id
label "5"
]
edge
source
target
]
edge
source 1
target
]
edge
source 3 target
]
edge
source 4
target 5
]
]
A.7 DATA EXAMPLES
In order to better show the input example, select the BFS order task in the serialization format the Adjacency List the complete prompt example, due to space reasons, the middle of the excessively long part will use Each of the following examples is randomly selected from the source data.
O-Shot
Given graph, your task is to determine the bfs traversal order of this graph starting at node And the graph representation of: Adjacency List is (0: [1], [0, 2, 3, 5, 6). 2: [1, 4J. 3: [1], 4: [2). 5: [1. 7J, [1]. 7: [5. 8), 8: [7]]
Q: Give the bfs traversal order starting from node 4.
A:
24

---

Published as a conference paper at ICLR 2026
O-CoT
Given graph, your task is to determine the bis traversal order of this graph starting at node 7. And the
graph representation of: Adjacency List is {1: [0, 2). O: [1, 3, 4. 5, 6J. 2: [1], 3: [0], 4: [0, 8J. 5: [0, 7],
6: [0]. [5], 8: [4]]
Q: Give the traversal
starting node
A:
Let's think step by step:
O-Instruct
Given graph, your task is to determine the bis traversal order of this graph starting at node 6. And the
graph representation of: Adjacency List is (1: [0, 21, 0: [1, 3, 4, 7, 8), 2: [1], 3: [0], 4: [0, 51, 5: [4, 6),
[5], [0], (01}
Q: Give the bfs traversal order starting from node
A:
Let's construct a graph with the nodes and edges first:
O-Algorithm
To determine the BFS (Breadth- First Search) traversal order, you need to follow these steps:
1. Initialize: Start by choosing starting node and enqueue it into queue.
Mark visited: Mark the starting node visited avoid reprocessing. Traverse:
to traversal order. For each unvisited neighboring node of the dequeued node, enqueue and mark it as visited.
While the queue is not empty: Dequeue node
4. A.Continue the process until all reachable nodes are visited.
Given graph, your task is to determine the bi's traversal order of this graph starting at node 7. And the graph representation of: Adjacency List (0: [7, 3,
71, 6), 5: [2, 6, 7, 8, 4, 1, O}, [1, 0, 4). 7: [0, 3, 5, 4, 21, [4,
5]. [4, 3, 5, O, 2:
[7. 0, 8, 1, 6, 4], 4: [1, 8,
6, 5. 3, 7, 0}}
Q: Give the bfs traversal order starting from node 7. A:
LTM
Given graph, your task to determine the hfs traversal order this graph starting at node 4. And
graph representation of: Adjacency List is [0: [4, 5, 7], 1: [2, 3, 4, 5, 6), 2: [1, 3, 4, 6), 3: [1, 2, 5, 6), 4:
[0, 1. 5, 6). 5: [0. 3, 4, 6, 7], 6: [1, 2, 3, 4, 51. 7: [0, 5]]
Q: Give the bfs traversal order starting from node
Let's break down this problem:
25

---

Published as a conference paper at ICLR 2026
Algorithm
To determine the BFS (Breadth- First Search) traversal order, you need follow these steps: 1. Initialize: Start by choosing a starting node and enqueue into a queue. 2. Mark visited: Mark the starting node as visited to avoid reprocessing. 3. Traverse: While the queue is not empty: Dequeue node and add it to the traversal order. For each unvisited neighboring node of the dequeued node, enqueue it and mark visited.
it as visited. 4.Continue the process until all reachable nodes Given a graph, your task is to determine the BFS traversal order of this graph starting at node And the graph representation of Adjacency List is : {0: [1. 2, 3, 4, 6]. 1: [0, 2. 8J, 2: [0, 1, 5]. 3: [0], 4: [0,
71. 5: [2]. 6: [0]. 7: [4). 8: [1]]
Q: Give the BFS traversal order starting from node 7.
A: Dequeue node 7. The neighbors are [4], so enqueue node Dequeue node 4. The neighhors of
17. 0]. Node is visited, so enqueue node O. Dequeue node 0. The neighhors of O are [1, 2, 3, 6, 4]. Node 4 is already visited, so enqueue node 1. 2, 3, and 6. Dequeue node 1. The neighhors of are [0, 2, 8). Nodes 0 and 2 are visited, enqueue node 8. Dequeuenode 2. The neighbors of 2 are [0. 5J. Node 0 visited, so enqueue node 5. Dequeue node The neighbors of 3 are [0]. Node 0 is visited, new nodes. Dequeue node The neighbors of are [0]. Node visited, so no new nodes. Dequeue node 8. The neighbors of are [J]. Node is visited, so no new nodes. Dequeue node S. The neighbors of are [2]. Node 2 is visited, so no new nodes. All its neighbors have been visited, so the traversal ends. The BFS traversal order starting from node 7 is 7, 4, 0, 1, 3, 6, 5.
no
Given graph, your task is to determine the hfs traversal order of this graph starting at node 3. And the graph representation of: Adjacency List is [0: [3, 7, 8, 5, 6, 1, 4], 1: [4. 10, 0, 3, 9, 21, 2: [10, 9,4, 7. 6, 11. 3: [0, 5, 1, 4, 10, 7, 2: (2,
10, 6, 0J. [9, 0, 7,
10), 6: [10. 0, 7,4, 1,
2]. 7: [8, 10, 0, 5, 6, 2, 8: [9. 7, 0, 2, 10), 9: [8, 10, 2, 5, 6J. 10: 19,
Q: Give the bfs traversal order starting from node
A:
CoT
Given a graph, your task is to determine the BFS traversal order of this graph starting at node And the graph representation of Adjacency List is : [0: [1. 2, 6], [0, 2. 8). 2: [0, 1, 5], 3: [0], 4: [0.
7J. 5: [2]. 6: [0]. 7: [4]. 8: [I]]
Q: Give the BFS traversal order starting from node 7.
BFS, we visit each node level by level, starting from the node we begin at (node 7). Here's the step-by-step breakdown: Start at node 7. The first node in the BFS traversal is 7. Visit the neighbors of 7. The neighbors of node 7 are just node 4 (since 7 4 is an edge). So, we enqueue node 4. Visit the neighbors of node 4. The neighbors of node 4 are node 0 (4 0), so we enqueue node Visit the neighbors of node 0. The neighhors of node that order. Visit the neighbors
A: The BFS traversal starts node
nodes 1, 2, 4 and 6.
Since node has already been visited. we enqueue nodes 2, 3, of nodes 1, 2, 3, From this point, the BFS continues by visiting any remaining unvisited neighbors of these nodes in the same manner, reaching node 8 last. Thus, the BFS traversal order starting from node 7 is 7, 4, 0, 1, 2, 3, 6, 8, 5. The BFS traversal order starting from node 7 is 1,2,0,1,,,3,66885.
Given a graph, your task is to determinc the bfs traversal order of this graph starting at node 28, And the graph representation of: Adjacency List is (0: [1, 2, 3, 4, 31J, 1: [0], 2: (0, 17, 22, 33], 3: [0, 5, 7, 8,
9. 10. 13, 15, 16, 19, 22, 23, 24, 26, 27, 28, 31], 4: [0, 6, 8, 10, 14, 15, 24, 27, 30], 5: [3, 8), 6: [4], 7:
[3. 11, 12, 25], [3. 4, 5, 10, 29], 9: [3], 10: [3,
33J, 11: [7, 18, 20], 12: [7. 13: [3], 14: [4].
15: [3, 28, 33), 16: [3], 17: (2, 24), 18: [11, 32), 19: [3. 171, 20: [11], 21: (12), [2, 31, 23: (3).
24: [3,4, 17], 25: [7), 26: [3), 27: [3, 33), 28: [3, 15), 29: [8), 38),30 [4), 31: (0, 3), 32: [18], 33: [2, 10.
15, 27]]
Q: Give the bfs traversal order starting from node 28.
A:
26

---

Published as a conference paper at ICLR 2026
K-Shot
Given a graph, your task is to determine the BFS traversal order of this graph starting at node 7. And the graph representation of Adjacency List is {0: [1. 2, 4, 6J. [0, 2. 8J. 2: [0, 1, 5J, 3: [0], 4: [0,
7J. 5: [2]. 6: [0]. 7: [4]. 8: [1]]
Q: Give the BFS traversal order starting from node 7.
A: The BFS traversal order starting from node is 7,4,0, 1,,,6,1,2.3,,,885
Given graph, your task is to determine the hfs traversal order of this graph starting at node 2. And the graph representation of: Adjacency List is O: [6]. 1: [6]. 2: [6], 3: [6]. 4: [6], 5: [6]. [0. 2, S]
Q: Give the bis traversal order starting from node
A:
Instruct
Given a graph, your task is to determine the BFS traversal order of this graph starting at node And
the graph representation of Adjacency List is {0: [1, 2, 4, 6J. [0, 2. 8), 2: [0, 1, 5], [0], [0,
7]. (2). 6: [0]. 7: [4), 8: [1]]
Let's construct a graph with the nodes and edges first.
Q: Give the BFS traversal order starting from node 7.
A: The BFS traversal starts node BFS, we visit each node level by level, starting from the node
we begin at (node 7). Here's the step-by-step breakdown: Start at node 7. The first node in the BFS traversal is 7. Visit the neighbors of The neighbors of node 7 are just node 4 (since 7 4 is edge). So, enqueue node Visit the neighbors of node 4. The neighbors of node 4 are node 0 (4 0), so we enqueue node Visit neighbors node 0. The neighbors node are nodes 1. 2, 3, and Since node has already been visited, we enqueue nodes 1, 2, 3, and that order. Visit the neighbors of nodes 1, 2, 3, 6. From this point, the BFS continues by visiting any remaining unvisited neighbors of these nodes in the same manner, reaching node 8 last. Thus, the BFS traversal order starting from node The BFS traversal order starting from node 7 is 1,2.0,1,2.336688,5.
7 7, 4, 0. 1,
Given a graph, your task is to determine the bfs traversal order of this graph starting at node 10. And the graph representation of: Adjacency List is (0: [4, 14, 1, 11, 5, 13, 2, 12), 1: [12, 4, 10, 2, 0, 3, 14,
111. 2: [8,
13, 11, 12, 15, 5, 0J. 3: [10, 1, 11, 7, 4:
4: [0, 1, 15, 11, 6. 10), 5: [14, 6. 11, 0, 7J, 6: 5J. 8: [2, 15,
[5. 11, 14]. [14, 12, 9, 13,
12, 14, 13J, 10: 6, 10, 131, 12: [1, 13, 7, 14,
12, 10, 3, 13], 9: (2.
[3, 1. 8, 15, 11, 6]. [14, 0, 12,
01, 13: [12,
7,9, 11, 81, 14: [7, 0, 1, 12, 8, 9, 1, 6), 15: [4, 9, 8, 2, 10])
Let's construct a graph with the nodes and edges first.
Q: Give the bfs traversal order starting from node 10.
B BENCHMARK STATISTICS
This section presents the statistical characteristics of GRAPHOMNI, focusing on the graph families and token usage. We first detail the statistical properties of graph families used in our benchmark in Section I, followed by an overview of token consumption associated with various prompt schemes and serialization formats in Section B.2.
B.1 BASIC STATISTICS OF GRAPHOMNI
Table 7 offers detailed statistical overview of the diverse graph families employed in GRAPHOMNL. The table reports the average number of nodes and edges for each graph family across tasks such as order, Connectivity, Cycle detection, Diameter calculation, Shortest path, and Triangle counting. These statistics are presented for three difficulty levels: easy, medium, and hard, which reveal the inherent structural complexity differences introduced by the various synthetic graph generators. The selection of graph families is guided by their unique topological properties so that each task is evaluated on graphs that best reflect the challenges encountered in practical applications. In addition, some graph families are omitted from certain tasks because of their intrinsic structural characteristics; for instance, graphs produced by the BAF and all bipartite graphs are excluded from triangle detection when they are structurally incapable of forming triangles.
27

---

Published as a conference paper at ICLR 2026
Table 7: Statistics of Different Graph Types
Easy
Medium
Hard
Task
Graph Type
#Avg Nodes
#Avg Edges
#Avg Nodes
#Avg Edges
#Avg Nodes
#Avg Edges
BFS-order
BAF
8.11
5.78
15.14
11 11.06
27.86
20. 14
BAG
8.19
11.36
13.92
23.28
26.55
82.14
Bipartite-FRM
8.03
6.50
14.44
19.28
27.23
59.36
Bipartite-ERP
7.94
5.44
14.42
16.72
28.86
57.82
ERM
8.22
16.06
13.72
51.92
25.32
135,09
13.14
14.33
50.17
24.59
121.77
SF
8.11
9.00
14.81
19.00
27.73
38.00
Connectivity
BAF
6.21
10.67
31.17
Bipartite-ERI
8.07
7.43
15.33
23.00
30.83
63.00
Bipartite-ERR
8.14
7.57
13.67
20.00
28.17
59.33
ERM
8.07
9.71
13.83
36.67
27.50
102.17
ERP
8.11
10.56
17.83
66.17
26.33
98.00
Cycle
BAG
7.93
9.90
14.12
25.10
27.82
59.04
Bipurtite- ERMi
8.12
7.60
15.62
20.38
28.54
$7.96
Bipartite-FF
8.29
7.12
15.17
17.55
30.25
44.89
ERM
8.19
11.43
15.10
36.43
26.46
ERP
8.07
9.71
15.40
26.36
26.07
71.18
8.05
8.07
12.71
14.24
25.04
29.71
Diameter
BAG
7.98
9.73
14.30
29.91
26.81
92.24
ERM
8.00
19.73
15.22
65.48
25.90
139.79
8.16
18.61
15.17
70.36
25.19
130.79
SF
7.95
9.14
15.41
19.91
28.48
39.10
Shortest-Path
BAF
7.83
6.11
14.17
11.56
25.62
21.71
BAG
7.97
10.72
14.72
31.19
25.29
88.33
Bipartite-ERa ERM
8.06
8.97
14.61
30.58
25.50
94.71
Bipartite-HH FER
9.72
28.86
25.58
89.00
ERM
8.00
17.42
15.47
67.89
25.96
179.21
ERP
18.92
15.42
63.14
25.25
165.46
8.03
9.42
15.50
20.03
25.54
35.21
Triangle
BAG
8.16
13.12
14.09
25.48
27.72
55.65
ERM
8.06
17.44
13.39
30.81
28.80
62.60
ERP
7.94
16.05
14.16
31.11
27.22
55.28
8.14
9.59
15.61
20.88
28.35
38.58
Note: Graph types selectively excluded from certain tasks based their structural properties: (I)
Connectivity excludes BAG as they are inherently connected by construction: (2) Diameter calculation
excludes BAF and Bipartite-ER ERReeERR graphs due to potentially disconnected components leading to infinite distances:
(3) Triangle counting excludes BAF and Bipartite graphs as they are structurally incapable of forming triangles; (4) Cycle detection excludes BAF as they are acyclic by definition.
28

---

Published as a conference paper at ICLR 2026
B.2 TOKEN STATISTICS OF GRAPHOMNI
Figure 6 offers an overview of token consumption across different dimensions. We use GPT-4 tokenizer here. Token usage is impacted by the choice of prompt scheme and graph serialization format, interactions between them can further influence the overall token count.
Token Usage by Prompt Schome and
Token Usage by Serialiation Fermat and Task
1000
3000
Takene
2000
2100
Thooll
3000
2530
2005
1800
3000
500-
E-dipurties
Chat 4he Seriall serioliaation Format
Plimpe Schama
ratburt
Prompt Schame
Serialization Format at
DM
(a) Token usage across different (b) Token usage across different sc-(c) Token usage for prompt- prompt schemes and tasks.
rialization formats and tasks
scrialization format combinations.
Figure 6: Analysis of token usage patterns across different dimensions. (a) shows how token usage varies across different prompt schemes for each task. (b) illustrates token consumption patterns for different serialization formats across tasks. (c) provides a 3D surface visualization of the interaction between prompt schemes and serialization formats regarding token usage. Error bars in (a) and (b) represent the standard error of the mean.
C EXTENDED STUDY AND DISCUSSION OF GRAPHOMNI
C.I STUDY ON LARGER GRAPH (BEYOND 30 NODES)
Our benchmark design centers on graphs with 5-30 nodes. While modest compared to real-world networks, this range is both deliberate and effective. First, it aligns with the context length limits of current LLMs and matches the scale used nearly all recent graph reasoning benchmarks (see Table 3), ensuring comparability. Also, the scale enables us to generate tens of thousands of diverse queries per task, providing statistically robust performance estimates and clearly separating different models apart, like open-source from closed-source models. In this sense, the 5-30 node regime is not a limitation, but a well-calibrated testbed for probing the boundaries of LLM graph reasoning.
To further validate our considerations, we conduct additional experiments on graphs with 30-50 nodes. We sample 50 graphs evenly across all seven generators and evaluate four representative models, yielding approximately 3k new test cases with varied prompt and serialization settings. Results are reported in Table 8. As expected, larger graphs further stress performance, especially on BFS order and Triangle counting. Nevertheless, the relative ranking and accuracy patterns remain consistent with the 5-30 node Hard split, reinforcing the robustness of our findings.
Table 8: Preliminary results on 30-50 node graphs (EH= Extra Hard). Results on the 5-30 node Hard split are shown in parentheses. Bold orange Underlined blue/ Light blue highlights indicate best/second best performance in its category.
Open-source Model
Closed-souree Models
Task
Diffientty
Llama-3.1 (8B)
Mistral (7B)
Phi-4 (14B)
Qwen-2.5 (7B)
Claude-3.5
o4-mini
BPS order
EH
0.7020 36(0.63
0,2720.23 (0.34)
2.5511.08 (2.65)
1. 11950.52 (1.38)
16,0742.48 (26.80)
35 35.39112661 (32.43)
Connentivity
FH
80.33+3 74(74
14+11+1.200 26 (74.77)
$1.8248.23 (48.39)
N5.01+3.06211.
97.92+1.18 (96.99)
4847 (92.08)
Cycle
ElI
56.6213 16152 10)
22.62233.2 (51.6))
19.3818.00 09 (10.64)
61.3812.61 (62.27)
68.5345.11 18 (78,22)
71.66111.33 (93.06)
Diameter
EH
15.3953 83(1663)
6.8992.00(6977)
16.44:2.89 (17.71)
11,6742.51 (15.27)
48.7864.76 (56,
96 (34.61)
Shortest
Elt
15.5616. 76/23 (3)
1.4812.03 (12.21)
16.3047.66 (26. 60)
11.8516.6 62 (28.31)
$7.0614.46 (87.88)
58.08:4.52 (88, 62)
Triangle
EH
926.999224.951
2.4150.65(2555)
4,35+1.40 (4.38)
4.4020.80 (4.45)
12.3120.82 (15.92)
8.28a4.35 (17.533
29

---

Published as a conference paper at ICLR 2026
C.2 STUDY ON REAL- WORLD GRAPHS: REPRESENTATIVE CHECK
To assess whether our synthetic design translates to real data, we run a focused representative check on two widely used real-world graph suites from complementary domains: a social/interaction dataset IMDB-MULTI (Morris et al., 2020) and a molecular graph dataset (oghg-molhiv) (Hu et al., 2020). We sample 20 graphs per difficulty per dataset (60 graphs per task in total and thus 3.6k evaluated samples across tasks with prompuscrialization variants) and test four representative open-source models plus two flosed-source models. Table 9 reports the experimental results.
Finding 1: Conclusions remain consistent. Across all six tasks and difficulty levels, accuracy patterns on IMDE-MULTI and bgbg-molhit closely track the synthetic results: (i) reachability (Connectivity, Cycle detection) is the easiest regime and exhibits high accuracy once serial- ization is parsed: (ii) ordered-path tasks (BFS order, Shortest path, Diametes calculation remain substantially harder, with error modes dominated by lost ordering or forgotten edges; and (iii) Triangle counting remains the most difficult due to exhaustive enumeration and arithmetic reliability. The relative ranking of models is stable, and the gap structure between open- and closed- source models mirrors the results from standard GRAPHOMNI, In short, the representative real-world runs perfectly corroborate our synthetic-only conclusions rather than overturning them.
Finding 2: Real graphs often simplify certain tasks. Because many public real graphs are connected and sparse within the selected ranges, some tasks become easier than in our synthetic distribution. For example, connectivity saturates for strong models (near 100% on Easy/Medium in Table 9), and cycle detection displays uniformly higher means than in matched synthetic settings. This because the uneven data distribution of real graphs means that nearly all graphs are connected and contain at least one cycle. This case does not invalidate the benchmark, but it shows that using real-world graphs alone can under-stress the tasks that are critical to graph reasoning.
In sum, we include IMDB-MUIE and ogbg- molhiv as a representative check, which validates that our conclusions persist on real graphs from two major application families (social interaction and molecular science). However, consistent with both our evidence and prior community practice, we retain synthetic graphs in GRAPHOMNI as the default for comprehensive structural coverage, fine- -raiigr interpretability and control, and contamination-free evaluation.
Table 9: Benchmark results of LLMs across tasks (Mean+95% CI Margin) on real-world graphs. Results on the standard setting (i.e. GRAPHOMNI) are shown in parentheses.
Bold orange Underlined blue Light purple highlights indicate hest/second best per- formance in its category.
Task
Difficulty
Open-source Models
Closed-source Models
Llama-3.1 (8B)
Mistral (7B)
Phi-4 (14B)
Qwen-2.5 (7B)
Claude-3,5
04-mini
E
65.6345.22 22 (18.69)
48.0613318 (13.75)
41,9018.66 (33,03)
41,9887.40
(21.46)
97.1410.11 (91.42)
95.46E1 .37 (95
BFS onler
M
12.14 11 (5.27)
10.24 (3.36)
17,86:3.89 (12.49)
10.3212.40 222222299900 (6.05)
76,9812.74 79 (68.25)
86.9933.94 (79.37)
254480)90
0.2480.27
4.76:1.77 (2.65)
0.956077611 (1
41.3563.76 (26.80)
44.65=9399 (32.45)
E
94.21:1.47779
93.57a| 89 (79.90)
61. 61.59:9.07
(56.29)
96.3510.92 (88.10) 10)
99.92a0.16 16 (98.
100.00:0.00 (98.23)
Concenivity
M
88.73x2.37 (79.47)
$8.81
(50.60)
53.25:8.09 (54.38)
93,10:167
(87.23)
99.9210.16
(99.11)
H
89.44a2.01 (74.58)
87,3022 32 (74.77)
55.24 7.67 (48.39)
89.76=2104 (81.19)
98.65=0.07
(96.99)
95 05.00233.004 (92.02)
E
56.75+2.67
(55.49)
51.43el
(55.44) (4)
$1,03e6 82 (45:25)
59.3762.00 (62_19)
80.48+4.81
(83.56)
94.51x2.14 (97.97)
Cycle
M
56,0512.46 (54 (55.69)
49.9221.18
(53.71)
48.1715.83 (44.25)
55.63s1.79 (62.07)
76.51a5.06
(50.80)
92.1923.78 (97.75)
H
51,0342.21 (52.40)
49.84e1.31 (51.64)
44.6845.27 (40.64)
54.0512.200 (58.88)
71,7533.87 (80.10)
89.65-3.40 (95.61)
E
25.4844.20 20 (41.27)
20,05+4.18 (28.55)
501,8855.69 (42.81)
47.86+3 57 (45.08)
83.33+1.13
(83.71)
97.30=f1.90 (98.88)
Diameter
M
15.48.3.51 (27.20
8.81 (15.17)
25.16:3.92 (28.49)
23.02 29 (27.11)
58.3362.82 (71.22)
34.37&3.89 (72.84)
8.9743.00
6.1922258 (6.97)
(17.71)
12.5611.99 (15.27)
13.0242.52 (56,70)
(34.61)
E
39.2145.877 (38.75)
28.65e4.69 (31.18)
43.89+8 50 (42.61)
50.188:9.77 (47.46)
598 (9.44911..2 29 (94.35)
V8.59=1.44 (95.08)
Shortest
M
30. 16116644,7 (28.84)
21.8363.61
(19.89)
34,44+7.972 (33.)))
37.14:7.44 (35.53)
05.71+2.09
(91.27)
U8.39=1.72 (92.60)
II
22,0513.73
14,0512.51 (12.21)
30.16:6.43 (26.60)
29.6815 65 (28.31)
92.1 1.80 (87.88)
95.7952.91 (88.63
E
11.19833005 (14:97)
5.32e1.00 09 (11.87)
25.81 (12.88)
24.44:3.65 (18.56)
63,8943316 (43.41)
81 81.30e4.110 (84.54)
Trinngl
6.51+2.03 (8.56)
1.8340.84 (6.66)
14.44+3.61 (7.54)
11.83+2.11 (9
37.3042..4 (24.00)
N2. 12.2001..6 (48.13)
11
5.0512.17 (4.95)
1.35u00.6 643500664000 (2.55)
3.26 (4,38)
9.52 40 (4-45)
34,8412.80 (15.92)
65.29 75 (17.53)
30

---

Published as a conference paper at ICLR 2026
C.3 CONSIDERATIONS ON REAL- WORLD GRAPHS VS. SYNTHETIC GRAPHS
In designing our benchmark, we considered several possible choices of evaluation substrate, including both real-world and synthetic graphs. After careful consideration, we opted primarily use synthetic graphs, for the following methodological reasons:
1. Coverage and controllability Our seven classic generators are selected to span the principal structural motifs (random/Poisson, scale-free, bipartite, hierarchical, small-world), and they support fine-grained parameter control (e.g.. p in Erdōs-Rényi, attachment in BA) (Chakrabarti & Faloutsos, 2006). This control enables balanced, modular ablations and isolates causal factors of failure, which typical public real-graph suites do not provide.
2. Representativenesss vs. noise in public repositories. Real-graph repositories such as SNAP (Sosic & Leskovec, 2015) skew toward specific domains (social/web) with narrow size and density bands. Also, many graphs are connected and share similar sparsity patterns. This induces structural narrowness and domain bias, and it can reduce task hardness (e.g., connectivity becomes trivial). Mixing such graphs into a general-purre purpose reasoning benchmark, therefore, risks adding noise without broadening structural regimes.
3. Zero contamination. Fully synthetic construction guarantees no overlap with pretraining corpora, avoiding inflated scores due to memorization or leakage (Hendrycks et al., 2021a), Given rapidly evolving LLMs and opaque training mixtures, contamination-fiee evaluation is essential for credible comparisons.
Mcanwhile, synthetic-only evaluation is also standard in prior work. This design choice is not unique to GRAPHOMNI. Several foundational studies adopt the same "synthetic only" paradigm to ensure interpretability and controlled analysis: GraphQA (Fatemi et al., 2024) and GraphInstruct (Luo et al., 2024b) both rely solely on synthetic graphs to probe LLM reasoning, while Graph (Chen et al., 2024a) demonstrates that synthetic graphs can data. These precedents highlight that synthetic construction is widely accepted in the community as the most principled way to study graph reasoning in LLMs. At the same time, we note that the real-graph domains we choose MIMDB-MULTI and ogbg-molhiv) align with recent works such as LLM4Hypergraph (Feng et al., 2025), which employ citation networks and protein structures, respectively. Thus, our real-graph ablation covers representative application families, while our synthetic benchmark remains the default for comprehensive coverage and methodological clarity.
serve as effective fine-tuning
C.4 EXPLORATION ON NP-HARD TASKS
To complement our six canonical tasks, we further probe LLM performance on two classical NP-hard graph problems: Hamiltonian cycle detection and Max-Cut. This evaluation serves ablation rather than a core component of GRAPHOMNI, allowing us to test whether the conclusions from tractable tasks extend to settings of higher computational complexity.
Experimental setup. We retain the three difficulty splits by node size: Easy (n E [0, 10),, Medium (n E 20)), and Hard (n (20, 25)). Compared to the main benchmark, the Hard regime uses slightly smaller graphs due to the exponential growth in search space. For Hamiltonian cy- cle, structural imbalance makes several generators unsuitable (e.g., SF, Biparti SereiteEER,, BAF, and rarely admit cycles). We therefore restrict the task to ERM, ERP, and BAG, with ground-truth labels balanced 50/50 between existence and non-existence of Hamilronian cycle. For clarity, we also report Hamiltonian cycle results on the positive cases separately, since these are strictly harder: a correct answer must not only assert existence but also return complete and valid tour (metric mentioned below). For Max- Cul, all seven graph families are included (SF, ERM, ERP, BAG, BERP, BAF, and BERM). We sample graphs per split for Hamiltonian cycle and 14 per split for Max-Cut, yielding just over 6,000 queries across prompt and serialization variants.
Evaluation metrics. As with the canonical tasks, we apply strict binary scoring. For Hamiltonian cycle, a prediction is marked correct only if: (i) the model explicitly affirms or denies the existence of a cycle in line with the ground truth, and (ii) when the ground truth is True, the model additionally outputs concrete cycle, which verify with dedicated checker. Omitting producing a non non-verifee verifiable tour results in O. For Max- we extract both the predicted maximum cut size and the corresponding bipartition (from phrases such as "the maximum cut size is."".. A
explicit decision or
31

---

Published as a conference paper at ICLR 2026
custom validation function checks whether the reported cut matches the ground truth. Full correctness requires both size and partition to be correct, while partial matches or con-extracttble answers are scored 0.
Results and insights. As summarized in Table 10, performance patterns closely resemble those of the six canonical tasks: open-source models hover near random, while closed-sourg reasoning models achieve substantially higher, but still imperfect, scores. Thus, the core conclusions of GRAPHOMNI generalize naturally to NP-hard settings. More interestingly, these results highlight how LLMs perceive task difficulty differently from humans. Whereas human solvers experience a sharp jump in difficulty between polynomial- uniform collapse in accuracy across NP-hard tasks. In other words, scaling to NP-hard does not introduce a progressive "step up" in challenge for models as it does for humans. This suggests that including NP-hard tasks may not meaningfully enrich the evaluation landscape, and reinforces our focus on tractable yet diverse tasks as the primary design of GRAPHOMNI.
and NP-hard problems, current LLMs instead exhibit a nearly
Table 10:
Benchmark Results of LLMs Across Tasks (Mean+95% CI Margin).
Bold orange Underlined blue Light purple highlights performance in its category.
indicate
Task
Difficulty
Open-suurce Models
Cloved-suuce Models 04-mini
Llamm-3.1 (8B)
Mistral (7R)
Phi-4 (14B)
Qwen2.5 (7B)
qwen38
E
57. $7.14=2.97
55,4722.85
57.6746.2 25
23.02-2.66 87
93.74:1.45
97.09:00888
Hamilton eycle
M
54 94:3 40
46 3.55
50.0916.55 59
60. 14:2.90 95
24.3
71.1
9413.61
46.38+3.00 04
51.85e6.99 98
60:23-3.08 08
79.98+2.64
55.5662.48
E
45. 6844.6
63.67+7.09
$4.6746.04 04
23.02=5.14
89 29.5992..51
95.24
Hamitoo eyele (Posirive Samples)
M
50. 50.62=6.90 90
64.75+7.61
60,8548. 39
21 21.60=5.3 30
79.37=4.15
49.56.4.32
47.09s6688
58.3846.81
56.7348. 13
61.55e6.30
70.90:4.61
18.4623.64
E
15.10:3.29
11.6342.44
23:881:1.9
18.37.2.71
27.9614.56
61.94g:.27
Max cut
M
5.20u1 1.66 1.66
5.10u1144
6.4311.81 81
10.41=1.66
16:94:2.33
28.16a 180
1.22:0.70
0.6110.47
2,04+1.02
1.1240.61
9.0842.17
34.74:3.64
C.5
SCALING VS. REASONING: DISENTANGLING THEIR EFFECTS ON GRAPII REASONING
To contrast model scaling with reasoning centric improvements, we isolate three Qwen variants: Qwen-2.5 (7B) as the baseline, Qwen-2.5 (72B) to represent scaling up within the same family, and Qwen-3 (8B) as reasoning model at a comparable parameter budget. Table 11 subsets the main results (Table 3) to these three columns.
Finding 1: Sealing lifts the floor. Relative to Qwen-2.5 (7B), Qwen (72B) yields consistent improvements across nearly all tasks, particularly on the easier splits. For example, accuracy on BFS order (Easy) rises from 21.46 to 71.41, an absolute gain of nearly 50%, while Shortest path (Easy) improves from 47.46% to 90.03%, margin of over 42%. Similarly, Diameter cal culatioo (Easy) increases by more than 33% (45.08% + 78.50%). Even on Connectivity, ty. which is already natu-saturated, scaling provides modest yet consistent lifts (E/M/H: +2.14%, +2.45%, +2.90% respectively). In contrast, on the most combinatorial regime, Triangle counting (Hard), the gain is negligible (4.45% - 4.73%), suggesting that sheer scale does little overcome the inherent difficulty of exhaustive enumeration.
Reasoning lifts the ceiling. When holding parameter count roughly constant, Qwen-3 (8B) substantially outperforms both Qwen-2.5 (7B) and, on several hard splits, even Qwen-2.5 (72B). For instance, on BFS order (Hard), performance improves from 22.03% to 29.53% compared to Qwen-2.5 (72B), relative advantage of more than 7%. On Di Diametrr calculation (Hard), the margin widens further: 39.83% versus 29. 29,5%% an absolute gain of over 10% The effect is most striking on Triangle counting (Hard), where Qwen-3 (8B) achieves 19.54%, far surpassing the 73% of Qwen-2.5 (72B). These results indicate that architectural and optimization changes targeted at reasoning are more effecttiee in extending the upper bound of graph reasoning ability than scaling alone.
Finding 2:
Implication. Scaling and reasoning improve different aspects of performance. Larger models predominantly strengthen robustness on casier instances, lifting the floor, whereas ceassoninn-orientte models better capture multi-hop dependencies and complex subgraph structures, lifting the ceiling. A
32

---

Published as a conference paper at ICLR 2026
Table 11: Isolating scaling vs. reasoning effects. Baseline: Qwen-2.5 (7B). Scaling: Qwen-2.5
(72B). Reasoning: Qwen-3 (8B). Bold orange
Underlined blue
Light blue
highlights indicate
bestsecoond-best/hhitee performance.
Open-souree Models
Task
Difficulty
Qwen2.5 (72B)
Qwen2.5 (7B)
Qwen3 (8B)
E
71.41
21.46:4.26
65.87+5.59
BFS order
M
47.8245.30
6.05+1.41
53.30:5.42
H
22.03+4.39
1.38+0.37
29.53+4.25
E
90.24+1.88
88.10#1.44
97.17
Connectivity
M
89.68 1.56
87.23+1.60
96.87+1.16
H
84.09+1.98
81.1942.02
92.89+2.07
E
74.02+3.34
62.19+1.85
90.30+2.33
Cycle
M
71.9943.34
62.07+1.80
89.66#2.07
H
68.40+2.73
58.88+2.14
86.81+2.27
E
78.50+1.16
45.08
77.56+2.77
Diameter
M
52.32+2.00
27.3143.16
61.7142.28
H
29.59+2.48
15.2742.47
39.83+2.67
E
90.03+2.27
47.4648.76
77.6945.17
Shortest
M
81.17+3.03
35.5346.80
69.6045.50
H
72.53+4.29
28.3145.50
64.28+5.60
E
36.57+4.40
18.56+1.24
41.36+4.63
Triangle
M
14.52+2.63
9.18+0.73
26.95+2.44
H
4.73+1.58
4.45+0.58
19.54+1.34
33

---

Published as a conference paper at ICLR 2026
balanced recipe, i.e. moderate scaling combined with reasoning-orientedd oriented objectives, appears most promising for closing the persistent gaps in BFS order, Diameter calculation, and Triangle counting.
C.6 RATIONALE FOR BINARY METRIC OVER PARTIAL SCORE
Evaluating graph reasoning outputs with partial credit is appealing in theory, but defining a consistent and objective scheme across six tasks, seven graph types, seven serializations, and nine prompt schemes is exceptionally difficult. In practice, two approaches exist: assigning credit based on the degree of correctness in the final answer, or rewarding intermediate steps and sub-outputs. Both approaches introduce major challenges. For final answers, it is often ambiguous how to compare partially correct results (e.g., overcounting triangles by one preferable undercounting by onc?). Such ambiguity undermines the credibility of fine-grained scoring. For intermediate steps, reliably extracting and interpreting model outputs at scale is infeasible, since formatting and reasoning styles vary widely across models and prompts.
By contrast, binary accuracy against a known ground truth provides a clear and unambiguous evaluation signal. With the extensive and diverse evaluation set in GR GRAPHOMNI, binary scoring captures performance gaps robustly and fairly across models and tasks. While finer-grained metrics such as edit distance, subtask scoring, partial correctness may be valuable for training objectives like reinforcement learning, they extend beyond the present study's 's evaluation focus. Incorporating such measures represents promising avenue for future work.
D RL-BASED PROMPT SEARCH INSPIRED BY GRAPHOMNI
D.1 BACKGROUND AND SERIALIZATION PROCESS
Our benchmark evaluates three key dimensions graph type, serialization format, and prompt scheme-to to underscore the critical role of transforming graph structures into textual inputs for LLM inference. In this section, We want identify the optimal combination strategies (serialization format; prompl scheme, etc.) that enhance the effectiveness of textual representations, thereby improving LLM performance in graph reasoning and understanding tasks. Prior research indicates that while particular serialization format or prompt scheme may yield optimal performance in isolation, their combination does not necessarily lead to the best results, highlighting complex interactions among various factors. Furthermore, the final performance of L.MS may be influenced by additional factors that were not systematically examined in our benchmark (e-g., those in Appendix D.3), underscoring the intricate nature of the graph-to-text transformation process, which extends beyond the scope of single-factor analysis. This makes finding the optimal serialization strategy complex. We define the process of converting graph structures into textual inputs tailored to specific task as the serialization process. Similar prompt processes are used in NLP. For example, Shi et al. (2024) formulated prompt formanting a multi-armed bandit problem; Sclar et al. (2023) employed Thompson sampling to determine the optimal strategies. For LLM-based graph reasoning. however, previous studies predominantly focused on single-factor variations. The multiple factor considered in our study significantly complicates the serialization process-one particular factor is determined, others are influenced in complex and often unpredictable ways.
Due to these complexities, it is computationally enormous to find the optimal serialization strategy by enumerating all possible combinations (termed as grid search in our study). To mitigate this computational challenge, we propose using RL to find high-quality serialization strategy under a spaces
limited LLM cost, because of RL's ability learn oparmal strategies in high-dimensiin through exploration and feedback. In the context of RI., we assume that all benchmarking results (e.g., those in Section 4.1 and 4.2) are not available. Instead, we will repeatedly choose various serialization strategies, test their performance, and use the results for RL.
Specifically, RL transforms optimizing the serialization process strategy into a sequential decision- making problem for each type and difficulty of the task. There are 7' decision epochs, and each decision epoch determines one component of the serialization strategy. For example, the decision horizon is T = 3 when we aim to identify the optimal combination of the serialization format, prompt scheme, and LLM. In the T = 3 decision epochs, we sequentially determine the prompt scheme, serialization format, and LLM. Such order of optimizing the components of a serialization strategy
34

---

Published as a conference paper at ICLR 2026
is predetermined, and will investigate its impact on the optimization results in future studies. This predetermined order specifies a sequence of action spaces {=1).= (e.g., A, can be all candidate LLMs). We set the initial state so the specific type and difficulty of the task.
Then at decision epoch = 1, ......,II T, we choose an action az E Ar based on the previous actions d1:-..; d4-1- This corresponds
policy x, So A1 x... x At-1 r At. where Sq is the state space of the initial state 8p- For any instance s (e.g., a query for Connectivity task in easy mode for specific graph), a binary reward, denoted by (l.... ,r), is incurred at the end of the decision epoch, which is set to if the LLM correctly answers the specific query under the selected serialization strategy (ai...., ar) and otherwise. For each type and difficulty of the task, our objective is to maximize the expected reward of choosing the serialization strategy d1,...,
max (o..........
E[r(s, a1 (l....,,r))........... (so),
where the expectation is taken with respect to the problem instance and the (random) answer output by an LLM (affected by the randomness of the LLM, e.g., the temperature parameter). Note that (i) So is part of the instance information s, and (ii) we fix the type and difficulty of the task, and the only randomness in terms of s is from graph generation. To approximate this objective function, we generate N different graphs for each type of query. We assess the performance of RL using the average reward across the N graphs, which essentially is the accuracy of the serialization strategy for specific graph-related task across these N graphs.
We use the Q-learning approach to solve this optimization problem. Let Qr .,,...,, a,) be the Q-function at decision epoch I,...,T, T, which represents the optimal reward-to-go if actions a1, ag have been determined at decision epoch t given the initial state 8o- These functions satisfy the Bellman recursion:
Qo(80: 01 ... 0e) max Q141 a1.................... " .....), t= 1= -
a0+1EAr-1
with terminal condition
Qr a1,....,ay ar) = E[[r(s, 01 E/r(s,,,,.......... ar....,)))))s)
This terminal Q-function can approximated by the accuracy of the LLM answer across the N generated graphs
Consider the problem of dealing with high-dimensional, complex state spaces in serialization process, we employ the Deep Q-Network (DQN) (Mnih et al., 2013) to implement RL, which employs a deep neural network as a function approximator for the Q-function. Specifically, we use a neural network (So, ....... ae; 0e) parameterized by 0r to approximate the corresponding Qr(so, a1...., at) for the actions or factors considered in serialization process. Each Q-network modeled as three-layer multilayer perceptron with ReLU activations. Training minimizes the mean squared error loss, and action selection follows an e-greedy policy, where linearly decays from 1.0 a minimum of 0.01. The detailed algorithm for each initial state so is provided in Algorithm 1.
We design two experimental settings, RL-Opt and RL-Scale, to assess the effectiveness of our approach. RL-Opt focuses on a T = 3 serialization prompt scheme, and L.LM model and evaluates both LLM cost and the accuracy of identifying the optimal configuration. RL-Scale extends the scope to include additional factors beyond those in GRAPHOMNI, investigating the scalability of the RL method for more complex serialization tasks, with emphasis on LLM cost.
selecting the serialization format,
D.2 DETAILS FOR RL-OPT SETTING
In RL-Opt, we apply RL to find high-quality serialization strategy under limited LLM cost. The serialization process in this case involved three key factors: serialization formats (in total 7), nine prompt schemes (in total 9), and five open-source language models (including LLaMA3, LLaMA 3. Mistral, Phi-4, and 2.5n. The total number of possible combinations in our search space is given by E = x 9 x 5 315. To find high-quality serialization strategy, we set the total training episodes to M 80 and initial learning rate to 0.001 during RL training. We evaluate the RL performance 6 tasks in three different modes, resulting in a total of 18 experimental cases.
35

---

Published as a conference paper at ICLR 2026
Algorithm 1 RL Framework of GRAPIIOMNI
Input: Action spaces {Ai] number of training episodes M; exploration rate e: initial state so Initialization:
Generate N graphs according to the initial state 80
Initialize Q-nttwrrks {Qr .1....................... (c;00))) fTm with random initialized weights 8 for episode to MM do
for / 1 to 7" do
Choose action:
With probability 6, select a random action a₂ E AA
Otherwise, set f- arg max Qr(5o. 01,...,.4-111 02) aEA:,
Execute action and obtain
state:
Update state: {8n, al....,at 0t) < {0o, (l....,a--))- Q-network update:
If T, set to be the accuracy of the LLM answer across the N generated graphs Otherwise, set y
max (01.,,0,,,,
aEAr+1
Perform a gradient descent step on (y Pe(a,,..,,,;;pp, ar:))) with respect to 0g
end for
Decay exploration rate: é +- e decay rate end for
For each case, based on our numerical results in Sections 4.1 and 4.2, we know which combination (serialization format; prompt scheme; LLM) performs the best for each specific graph-related task. Hence, we can compare the serialization strategy obtained by RL with the ground-truth optimal strategy.
Specifically, we employ two key metrics. Search Cost: Given that RL explores different combina- tions during the training process, we define Cost le where & depends the number of training episodes and K is the total number of combinations. Rate: Let acc, be the accuracy achieved by the best combination found by RL, and mcc be the highest accuracy in Sections 4.1 and 4.2. The results are displayed in Table 4. can be seen that, with an approximate 25% reduction in cost, the RI. method still maintains an average rate around 0.9, indicating its ability significantly shorten the time required for the search for optimal combinations while ensuring the quality of the results. This outcome underscores the notable advantages of RL in the serialization process problem rapidly finding high-quality solutions, thereby substantially reducing computational resources and time costs. Moreover, this approach does not rely heavily on extensive manual expertise, enhancing the automation of the optimization process. As result, is applicable not only to the factors considered in this study but also to other factors warranting further investigation.
Then we define Rate acc.
acCmax
D.3 RL-SCALE
In RL-Scale, we examine the scalability of our RL method by incorporating additional four factors into the serialization process. Different from RL-Opt that optimizes the LLM model, we fix the model Qwen-2.5 and test the performance additional four factors are inspired by Sclar et al. (2023), which are the delimiters between sentences, the capitalization style of each sentence, the delimiter used to introduce questions and answers, and the delimiter between words. As before, we still optimize the prompt scheme and the serialization format. Details of the 6 factors implemented in the serialization process shown below.
the Diameter calculation task in easy mode. The
36

---

Published as a conference paper at ICLR 2026
factors implemented in serialization process
SPACESI: delimiter between sentences. include:
<sep> '," '. An '.' An', 'Nt',' Cs: delimiter used to introduce questions and answers, include: Anxt', 1 An At', ::'.'
\n', 11
:::
SPACES2:
delimiter between words, include: ',' ','At'
CASE FUNCTION: overall capitalization mode of each sentences, include:no change, title, upper, lower
PROMPTS SCHEME: include: O-shot, 0-Co'T, O-Instruct, O-Algorithm, LTM, Algorithm, CoT, k-shot, Instruct
SERIALIZATION FORMAT: include: GMol,, Adjacency Set, Edge Set, Edge List, Adjacency Matrix, Adjacency List, GMal.
Since the optimal combination in RL-Scale is un- known, only focus the cost of RL finding the near optimal combination. In addition, tak- ing into account the L.LM cost problem and the performance stability, for each combination, we took fixed evaluation of 30 samples to get ac- curacy and set the temperature to O. Ultimately, we compare the costs of RL and grid search der conditions where the serialization process involves 2-6 factors, with the results presented in Figure 7. For step counts of 2, 3, 4, 5, and 6, the number combinations explored for RL. is 40, 121. 182, 300, and 632, respectively. In contrast, the number of combinations for Grid Search is 100, 300, 1200, 8400, and 75600 for the same step counts. It shows that RL exhibits highly promising trend in terms of the cost asso- ciated with searching for optimal combinations. When considering two-step factors, the cost of
Comparison of RI. and Grid Search
50000
78000
Polynomial
68000
Combinations
Exponential
58000
RL Trend
40000
500
34000
20000
10000
5
Steps
Figure 7: RL and Grid Search
RL is comparable to that of grid search. However, as factors or steps increase, the cost growth of RL significantly lower than that grid search. This finding suggests that for serialization process tasks, RL can adaptively adjust its strategy to better accommodate complex environments, highlighting its broader potential for application.
Table 12: 2-6 factors, top-3 combinations and corresponding reward from RL-Scale.
Grid research
Combination
Edge List.-sbbr, Q: Ac, Ia DO Edge Listt0- Q; Ar, ln,
0.3000
100
0. 0.2666
Edge List,0-shot, Q; Ar.... ... no
0.2000
Edge List,0-shot, Q: A;, II, 't, Edge List00 shot,
0.3667
300
0.3333
Edge List,O.sho
0.3333
Edge List,0.hoo Q;
upper
Edge List.0-sbor, Q: At, Il..: lower
lower
0.4000
Edge List.O-sbor, Q: Ac, u. upper
0.3667
Adjucency O-sho, Q:, A:, <sep>.. ln, title
0.5667
8400
Dhbo,, Q: A:, la, M, :.: upper
0.5333
G-oo, Q; ; \n. Mt 12 upper
0.5333
Adjacency Set, Algorithm, Q. Il.. lower
0.6333
75600
Adjacency Set. Algorishm. Q. Il.... Adjacency Matrix shot, Q: in, fitle
0.6333
37

---

Published as a conference paper at ICLR 2026
E COMPREHENSIVE EXPERIMENTAL RESULTS
In this section, we include all the experimental results and additional analysis of the GRAPHOMNI as a reference to support our claims and findings mentioned in Section 4.1. We first present ane-grained experimental results broken down across main evaluation dimensions in Appendix E.I, followed by detailed performance heatmaps for all tasks and models in Appendix E.2. Finally, we provide a comprehensive error analysis with representative cases in Appendix E.4.
E. FINE-GRAINED RESULTS ACROSS DIMENSION
In this subsection, we present detailed performance results across model capabiliit, graph type, prompting schemes, and serialization format impact. Based on the complete evaluation results in Table 13, we further analyze the results from multiple perspectives, including overall performance across all models, separate analyses for open-source models, and specific results for closed-source models. These comprehensive results provide additional evidence supporting our main findings discussed in Section 4.1.
Table 13:
Benchmark Results of LLMs Across Tasks (Mean+95% CI Margin). best/third-dtttteeeeeeee
Bold orange Underlined blue/ Light purple highlights performance in its category.
indicate
Talk
Liuead (B: Laaaddi
Stared
PS-4 (148)
(maad
Clasdedd
GPT-4paad
Gentai28
Radioa
133511
214454
13
03
1334037
35554435
39.563399
$25000
15511120
12AL14G
67.3
74.88287
54321217
46.35
53.441775
50.03
412441.53
524461.47
51246000
73466330
56.66199
55616123
50.53
13.
41
3
16
43003
1133
6906125
132143000
15252
53.56297
135
335548444
18.13.1111
31186400
42312335
M35-200
56.2
9
14.56155
11836132
13.56225
30336231
213
Trkalle
E.I.1 OVERALL RESULTS
Here we present a comprehensive analysis of the overall performance across all evaluation dimensions. While our main findings in Result O highlight the moderate performance of models with considerable room for improvement, the detailed results in Tables 14, 15, and 16 reveal several noteworthy patterns:
Task-specific Performance Variation:
The performance varies significantly across different tasks and difficulty levels. For instance, in Connect iimettive tivitttityytt tasks, models generally achieve higher accuracy (80%-90% for easy level) compared more complex tasks like Triangle counting (20%-30% for hard level). This suggests that while LLMs can handle basic graph properties well, they struggle with tasks requiring more sophisticated reasoning and counting.
Difficulty Level Impact: There is a consistent and non-linearly sharp decline in performance as task difficulty increases. With larger graphs, models face challenges in both processing longer contexts and conducting more complex reasoning tasks, which typically require longer reasoning paths, more precise intermediate steps, and more comprehensive exploration of the graph structure, The sharp performance drop on larger graphs suggests that current LLMs struggle to maintain reliable reasoning capabilities when faced with extended multi-step graph operations.
The performance gap between closed-source and open-source models is particularly evident in complex tasks. For instance, GPT-4o and Claude-3.5 consistently outperform other models by a significant margin 115%-20% in tasks like Diameter calcuttiion and Tr Triangle counting especially at higher difficulty levels. This reinforces our observation about the current limitations of open-source models in complex graph reasoning tasks.
Model Type Performance Gap:
Graph Type Impact: The evaluation reveals distinct performance patterns across different graph types, with certain structures showing clear advantages for specific tasks. Our analysis shows that
38

---

Published as a conference paper at ICLR 2026
bipartite graphs (BERM, BERP) tend to exhibit higher performance in connectivity and clustering-related tasks (Connectivity)) potentially due to their explicit partitioning node sets, which simplifies certain connectivity relationships for LLMs. For shortest-path (Shortest path) tasks, hicrarchical structures like BAF often show higher accuracy, as the tree-like paths may align well with reasoning processes for pathfinding. In local pattern identification tasks such triangle counting (Triangle counting), simpler graph structures like SF often perform better, possibly because they reduce the complexity of identifying local patterns. These observations suggest that the interplay between graph types and task characteristics can significantly influence LLM reasoning behaviors.
Table 14: Benchmark Results of Prompt Schemes Across Tasks (Mean+95% CI Margin of All Underlined blue highlights indicate best/sccond- -bett performance.
Models). Bold orange
Tisk
Difficully
0-Algorilhm
0-CoT
0-Instruct
U-Shot
Algociibm
CoT
Instruct
K-Shol
LTM
E
52.20+7722
46.5448.20
48,4248.22
49.494.334
63,3967.10
63.35+6.23
62,93+6.16 16
56.51+6.45
48 48.1248.1
BFS order
M
33.6366.44
33.3746.89
33.3646.76
33,9446722
43,4847.23
58.9966.32
38.6166.0
33.08+6.09
32.89+6.67
H
13:5763.45
14.46+3.58
13,98+3.00
14. 58
19.3724.46
14,01+3.22
13,2743.27
11.68+3.01
13,95+3.55 50
E
85.51=3001
83.2745.06
8686+2.45
$2.7945.40
98.7942.32
92.3761.57
87.46+2.34
83.00+4.70
Connectivity
M
86.34e:3.5
63.5014
83.80+3.26
83:3615.45
88. 98.9942.00
91.98e1
92.077.1.45
89. 29.24+2.05
83.65e4.17
H
81,7663.90 99
78.344487
74.21+4.00
79,0045335
82 $2.89+2.52
85.6822.0 70
86.41=2.19
85. 16+2.77
78.35+4441
71.75+3.91
64.7345.07
70.2343.67
64.9345.34
71.2864.34
73.01+3.62
71,66+3.77
08.8943.90
Cycle
M
69.16+4.07
64,5044.73
67.75+3.99
63.9446.37
70.0224,77
71.59t4.16
10116
70.40+4.244
67.58+1
H
66.2764.04
62/7544.47
62.1444.60
62.5445.20
67. 67.38
69.12+4.20
67.95b4.13
68.35+4.14
66.3543.73
51.5246.46
$2.6616.81
52.5547.06
53.6516.46
62.32:4.95
64,4244.01
64.651-1009
Diameter
M
34,41 35
M66865.50
34.3445.62
57,5465.07
50.69+2.97
46.8844.66
48.0064 06
47.64e3.95 93
38.8346.37
1I
19.53a3.90
22.2813.81
20.5914.16
22.9213.71
32.1333.29
30,8043.78
32.5413.49
31.8913.29
21.191,1009
E
67.5845.66
57.0648.17
55.8948.48
67.1846.33
24.7946.01
74.7346.10
75.6245.76
72,7546.37
57.17+7.99
Shortest
M
59.3226.24
50.9748.02
50.60t8.144
58.5246.58
65 0926.339
66.7326.09
66.0246.19 19
63.98a6.26
51,8067.73
II
53.1416.66 66
48:4017.89
46.8818.05
52,5116.82
56.1816.72
57.2716.41
57,1016..3
54,0316.43
47.9717.67
E
28.81
29.9545.04
28.0045.04
31,2245.08
$2.3964.75
$4.81+5.10
38.00444668
33.3964457
30.52+4.95
Triangle
M
12
17,0712.83
15.8812.93
16.9713.10
16.6212.86
18.0713.15
18.6742.77
18:5212.75 7.78
17.11
8.85e1.53
10.14:1.68
8.55+1.56
9,42s1.63
8.2121.56
9,48&1811
9.54&1.58
9.10&1.47
Table 15: Benchmark Results of Serialization Formats Across Tasks (Mcan=95% CI Margin of All
Models). Bold orange Underlined blue highlights indicate best/second- performance.
Task
Difficulty
AL
AM
ES
GMaL
GMoL
E
63,2726.63
49.10=7.02
62.5426.66 63
50.40=6.27
51.6266.37
38.8626.42
47.54:5.57
BUS order
47.1326.74
27.55+5.27
48.18
31.39=5.17
29.5724.96 96
39.5545.88
29.56:2491
23.9214..37
5.19:1.16
23.5924.24
11.40=2.7
9,0621.69
15.5842.75
11.50+2.31
E
89.4913.25
80.9243.10
89.57+3.20
87.02+3.42
88.50s3.2
88 28.4412..6
84.58+2.31
Connectivity
88.7543.23
82.6312.85
89.04s3.09
86.5113.48
86.7243.31
88.2743.04
86.87+2.52
85.5243.48
68.3742.46
85.68
82.49=3.65
81.03+3.52
83.8343.70
82.8743.09
E
64.30+3.53
65.7543.61
64.41+3.50
71.54=3.59
75.38+3.40
76.09:44.26
72.2243.38
Cycle
63.3543.58
62.9043.51
63.23+3.52
70.55=3.81
72.51+3.66
73.93+44.3
71.70+3.94
61. 61.00+3.59
59.06+2.94
60.20+3.52
69.64=3.95
68.8243.68
71.42+4..2
70.96:4.24
58.31
58.6344.95
61,33+5 09
54.95=5.24
54.51=5.33
62.28::4.87
58 38.68855.1
Diameter
42.89+4.77
39.6743.83
45.69+4.68
37.78=4.05
35.60=4.11
44.5244..55
42.98+4.449
H
27.6824.00 .00
23.6543.00
29.61+3.82
23.26=2.76
20.03+2,70
29.7743.63
27.90+3.44
75.8945.76
54.1445.9
76.60=5.61
72.00=5.63
68. 68.9925.88 81
02.8547.49
68.35
Shortest
69.65+6.00
40,94+5
69.14=5..0 68
64.5745.83
58.30=5.90
52.38+7.34
59.60+5. 25
H
65.3146.09
28.2244.13
65.79+5.96
55.82+5.90
52.05+5.83
47.8847.06 06
53.20=5 42
32.0324.41
27.61+4.08
31.8224.48
31.7024.21
30.6424.06
34.30::4.44
32.89+4.6
Triangle
17.50=2.56
13.50+1.94
17.61=2.60
11.65=2.66 66
16.45=2.440
18.8342.71
17.95+2.89
H
8.78#1442
6.61+1.05
10.35#1.51
9,7741.38
8.75+1.22
10.34+1.48
9.50=1.59
1.1 RESULTS OF OPEN-SOURCE MODELS
Open-source models exhibit several distinct characteristics compared to the overall results. In terms of prompting schemes (Table 17), more structured approaches show clear advantages: CoT and
39

---

Published as a conference paper at ICLR 2026
Table 16:
Benchmark Results of Graph Type Across Tasks
(Mean+95%
CI Margin).
Bold orange Underlined blue highlights indicate best/second- performance. the graph type is not applicable for that task.
indicates
Task
Difficulty
BAF
BAG
BERM
BERP
ERM
ERP
SF
43.8243.15
44.93=3.06
53.30+2.77
52.6842.75
43.49:2.76
47.82+2.88
48.2013.08
BFS order
M
35.0613.03
29.38=2.68
24.9612.33
34.5612.58
21.76#1.97
22.48+2.05
26.63+2.57
27.58:2.66
7.1741.17
6.67a0.95
13.79
4.03x0.70
8.40x1.06
7.40x1.24
E
77.04
88.03 1,48
84,29 152
87.022
86.97+1.54
Connectivity
M
78.99+1.8
86.31+1.52
86.60=1.54
84.61=1.51
16.11+1.56
65.93+1.775
84.18+1.75
80.68#1.74
85.28
E
64, 64.90=1.66 60
15.98#1.49
65,02+1.35
68.32+1.70
69.25=1.57
61.08#1 1,0
Cycle
M
60.18=1.73
1.621
11.41+1.54
65,23=1 86
66.08+1.70
59. 19.10#1.55
55.20=1.56
66.666991
65.58+1.72
61.97=1.82
62.99=1.81
54.98#1.4
47.3 47.35=2.100
44.95a2.25
48. 96:22222
56.82+2.08
Diameter
M
33.38&1.79
30.64
35.81+2.21
37.41=1.65
22.78=1.88
20.70=1.99
26.76*2.11
1..1
66.73s2.98
60.06=2.82
57.52+2.82
61,01+2.88
55. 55.12+2.88 80
59.53+2.87
61.12+2.80
Shortest path M
58.11=3.01
52.88+2.84
48.95+2.76
48.72+2.72
45.83=22.7
51.5552.75
57.08+2.88
55.62+3.15
46.19 16.997
42.47+2.69
39.6742.85
39.54=2.66
43.36+2.66
48 48.93+2.89
25.25=1.53
12.54=0.79
17.17=
41.20=2.03
Triangle
M
16. 16.75=1.11 11
7.38+0.45
9.55+0.56
18.30=1.12
8.99=0.77
5, 5.48+0.42
7.56+0.56
8.22=0.58
Instruct prompts consistently outperform simpler schemes like 0-Shot and LTM across most tasks. This is particularly evident in Connectivity tasks, suggesting that open-source models benefit more from explicit reasoning guidance.
For serialization formats (Table 18), open-source models show strong preference for concise
representations. Adjacency List (AL.) and Adjacency Set (AS) formats consistently perform better than more complex formats like GMal. and GMol.. This contrasts with the overall results.
Regarding graph types (Table 19), while the general pattern of sask-spec advantages remains similar to overall results, open-source models show more pronounced performance gaps between optimal and sub-optimal graph types. For instance, in Triangle counting tasks, SF significantly outperforms other graph types with wider margin compared to the overall results.
Table 17: Benchmark Results of Prompt Schemes Across Tasks of Open-source Models (Mean+95% CI Margin). Bold orange
Underlined blue highlights indicate best/second-bee performance.
Task
Difficully
0-Algorilhm
B-CoT
0-Instruc
0-Shol
Algorithm
CoT
Instruct
K-Shol
LTM
E
29.7265.04
2012515.44
22.1915.76
29.6415.43
44.3416.33
49.2416.100
48 1R8415538 85
41 41.6245548
22.501 82
BFS order
M
15.251:1.15
14.551183
145611188
15.9811.70
25.3115.91
28.8815.78
25.3115.56 56
19.5211.83
14.2611173
H
7.26
7.76+3.14 14
7.243304
7.98+3111
10.51#3.54
9.47+3.55
9.18+3.40
6.55+2.61
7,28+3.05
78.11+2.96
74.2145.84
81.3142.35
74,8346.44
85.70+2.46
80.25+2.10
89.46:1.64
8442242222
74,19+5.9
Connectivity
M
79.1313.66 60
74,0815.58 55
15.5613.19
74,5416.48
85.3912.12
88.4012.16
88.9811.42
86.5212.17
75.15
H
74.5264.2
69.13+5.34
62.30+4.72
71,4166222
80.14+2.19
81,68+2.33
83.14:1.6
82,85+2.4
69.05+4.51
63.3113328
53.6211.73
63.6613.11
55.3815.11
61.3211011
66.3333.33
64.6713.10 10
60.9013.266
Cycle
M
61,0243.55 52
13.114413
59.7543.36
53.7845.11
61,5544.57
63.8143.73
62,44+3.63
64,06643388
59.66+3.12
H
58,4263.61
52.683384
51,534388
52.4245.14
58.2584.44
61,20+3.21
59.90 00.80
61.39+3.53
58.60+2.91
36.4115.44
37.1215.55
37.3616.51
40.6745.67
66.33:2.63
56.6314.76
60.39
59.93x3.51
39.13x5.9p
Diameter
M
22.6843.00 90
25.2644468
62
27.6344.24
44,44+2.65
40.30e4.44
42.93+43.59
41 41.45+3.45
24.25+4.38
H
11,6942225
15.2542.78
12.8443.17
16.7142.83
24.70+2.52
25.5043.29
27.9843.00
27.2242.93
13.5743.00
E
55,81+5.44
54. 34.3447.21
$2.3947.55
$2.7346.28
62.0346.43
68.7246.36
64.5546.3 34
63.0446648
$4.3846.34
Shortest
M
43.89a5..0 43
28.2166.58 58
28.2586.95
42.44a5.99 92
50.31a6.24
$3.78a6.20
$2 22.9146.29
50.08a6. 15
29 29.4466.13
36.2645.51
26.5146.31
24.5046.42
35.58+3.00
40.6746.12
39.9945.99
E
14.74e1.99
16.9143.09 09
14:2362.52
19.8843.43
21 21.98+3.28
2733244522
28.14:3.84
26. 20.5543.34
17.7963.07
Triangle
M
1.9
10.24a1.73
8.3721.58
9.96
9.5721.97
129412281
14.13:2.43
13:0129.22
10.2841.83
5.14
4.50s1.19
5.96s11.31
7.35+1.99
2.49+1.57
5,57+1.08
40

---

Published as a conference paper at ICLR 2026
Table 18: Benchmark Results of Serialization Formats
Across Tasks of Open-source Models
(Mean+95% CI Margin).
Bold orange Underlined blue
highlights indicate best/second- per-
formance.
Task
Difficulty
AL
AM
AS
EL
ES
GMal.
GMMll.
42.12+5.89
25.68+5.22
41.02+5.77
30.04:4.94
30.01=4.73
39.54+5.
31.41+4.67
BFS order
27.35+5.90
11.92+3.59
24.79+5 40
15.90=3.81
14.02+3.31
22.74+4.95
15.40+3.31
H
15.45+4.36
1,99=0.72
14.60=4.1 10
5.87+1.72
4.63+1.27
8,99+2.41
5,41=1.55
83,3343.8
74.63+3.8. .50
83.54=3.79
80.79=3.99
82.7623.78
82.97+3.43
81.23+2.45
Connectivity
M
82.3643.78
76.8543 .22
82.90=3.60
79.50=4.01
79.40=3.71
02.8993.56
82. 222.13
H
78.43+3.98
66.16:2.66
78.89
74.26=4.04
72.53=3.73
77.01=4.25
77.11
58.7243.30
59.20+3
59.18=3.31
62.89=3.39
66.65=3.27
65.24+4.14
64.64+3.20
Cycle
M
57.70:3.29
55.7343 .00
57.8243.24
60.81:3.56
63.40:3.40
62.44+4.28
61. 8613.88
H
54.64+3.29
52.54+2.56
54.50=3.25 25
58 28.8043.54
39.95=3.40
60.041:.00
59.54
47.2515.
49.01s4.53
49.78s4.91
45.0662.88
43.82+4.78
53.681:4.55
48.9515.18
Diameter
M
32.2814.19
31.9113.38
34.8314.23
30.61
28.0623.68
35.40:4.099
33.50aa.16
18.13+3.07
19.70s2.24
20.57+3.23
19.65s2.700
16.12+2.30
22.44+2.89
19.87+2.82
61,3846.13
39.4645.99
62.90a6.02
57.9 $7.99=5.93
54.26+ .99
30.5245.99
55.1445.54
Shortest
M
52.45+5.93
27.69a4.83
54.6345.79
47.29
41.24+5.35
25.75+5.41
45.99+5.22
87.7945.86
18.7243.74
48.52a5.72
38.00=5.16
34.34a4.78
22.69455.
36.89+4.81
E
20.03+3.00 0.00
17.87+2.49
18.98+2.82
20.52=2.70
21.13+2,77
24.76+33.7
22.42+3.55
Triangle
M
10.37#1.79
8.81+1.35
10.41#1.83
11.17=1.65
10.72=1.70
12.92+2.19
11. 11.28+2.06
5.37=1.34
5.45=1.07
6. 6.76+1.55
5.84
6.44+1.25
7.07=1.39
5.50=1.22
Table 19: Benchmark Results of Graph Type Across Tasks of Open-source Models (Mean+95% CI Margin). Bold orange Underlined blue highlights indicate best/second-best performance. "." indicates the graph type is not applicable for that task.
Task
Difficulty
BAF
BAG
BERM
BERP
ERM
ERP
SF
31.18+2.30
31.34+2.18
41.92+2.08
41.16=2.00 04
30.59+1.90
34.53+2.01
34.43+2.24
BFS order
23.15+2..3
1999
17.92+1.79
24.76=1.87
14,89+1.41
15.08+1 1.1
18.27+1.8
H
18.06=2.00
7.31+1.11
6.36+0.99
10.85=1.16
4.08+0.83
6,70=1.00 1.00
7.38=1.13
72.16+1.43
81.79=1.41
84.65+1.40
14.10.1.45
Connectivity
74.84+1.4 143
83.02+1 12
84.28=1.45
80.99=1.33
82,86 .44
H
60.68+1.43
82.25+1.62
79.76=1.55
7..94=1.55
81.89+1 1.59
60.70+1.34
62.16+1.36
63.53=1.36
63.61=1.44
64.51:1.39
59.65+1.33
Cycle
M
16.82+1.36
62.69:143
59.75=1.40
60.91=1.49
62.21+1.44
1..4
H
12.15=1.27
19.50=1.52
61.04=1448
17.73+1.49
59.09 1.6
53.36+1.22
45.50+1.84
45.74=2.04
48.574
L..
Diameter
M
32.72+1.55
28.87+1.60
1..85+1.76
36.16+1.45
10.95+1..6
16.46*
20.54+
20.90:11.00 1.04
59.6342.63
51.54+2.37
49.30=2.39
52.61=2.46
46.04=2.32
52.97442.46 50.0442..7
Shortest
M
49.10+2.53
42.3242.25
38.87+2.22
39.58=2.23
36.20s2.12
41.70+2.1 18
46.9812.36
46.3742.59
34.2812.16
33.3622.10
30.18=2.18
30.50s
38.55+2.22 33.90x2.06
19.1741.27
11.6840.95
13.8041 .02
35.37+1.73
Triangle
M
11.1540.85
7.6240.72
8.66+0.87
15.28
5.00+0.49
5.67440.6
5.78+0.53
8.13+40.79
41

---

Published as a conference paper at ICLR 2026
E.1.3 RESULTS OF CLOSED-SOURCE MODELS
Closed-source models exhibit notably different characteristics compared to their open-source coun- terparts. For prompting schemes (Table 20), these models show more robust performances across different prompting methods, with even simple prompts like O-Shot achieving competitive results. This is particularly evident in Connectivi ivitytiityyyy tasks, where performance remains consistently high across most prompting schemes, suggesting less reliance on explicit reasoning guidance.
The serialization format results (Table 21) reveal another key distinction: closed-source models handle complex formats more effectively. While they perform well with concise formats like AL, and AS, they also show strong performance with structured formats like GMaL, especially in tasks requiring sophisticated reasoning like Cycle de detection and Diameter ter calculation. This contrasts sharply with open-source models' preference for simpler formats.
Regarding graph types (Table 22), closed-source models demonstrate more balanced performance across different graph structures. For instance, in Triangle counting tasks, while SF still performs best, the performance gap between different graph types is notably smaller than in open-source models, suggesting more robust graph structure processing capabilities.
Table 20: Benchmark Results of Prompt Schemes Across Tasks of Closed-source Models (Mean*95% Underlined blue highlights indicate best/second-bel -bst performance.
CI Margin).
Bold orange
Task
Difficulty
0-Algorilha
0-CoT
O-Instruct
0-Shol
Algorithm
CoT
Instruct
K-Shol
LTM
E
83.6613.70 70
83.34.3.61 61
8S 15.1551300 08
82.07.3.81
90.0613.02
83.07.3.53
82.66.3.000 60
77,3514.91 9.22
83.99.3.20 24
order
59,37+4992
59,6764.6 68
50,0945.30
68.91+5.06
57.35e4.97
57.1764.22
52.05+5.35
58.9844.71
H
22,41
23.85e3.11 19
23,413310 10
23.6233.61
31,79a4.30
20.38+2.78
19,0142.68
18 2,9999990
23.29:3.12
E
95.86+1 105
95.95+1.18 18
94.64+1.4 41
93.92+1.00
93.10+1.77
96.72+0.94
96.4340
91.15+2.01
95.34+1.27
Connectiviyy
96.22+1.01
95.34+1.00 00
95.71+0.96
94.01=1.35
96.99H0.74
96
93.06e1154
95.56i+1226
H
91.91H22.12
21.24+2.07
90,8942.19 19
91,55+2.04
86.76+2.74
91,28+2,34
90.98+2.27
88.4042:57
91,57+2,23
E
82.1763.67
80.26+3.22
79.42+3.52
78.90+3.73
81.09+3.95
81,58+3.22
81,4443.27
81.01+3.54
80.083 46
Cycle
80.54e3..5
79.75+3.45
789941.56 56
76.1563.55
87.8743883
N2.4963..3
20.25-3/73
79.27+4003
786761 54
H
77,26 40
76.8443.4
76.9943.62 62
76.7063.54
80,15+388
80.20+33.1
79.38+3.73
78.08+4.07
77.19+3.54
E
72.6864.82 82
74,4245 02
73.8245. 02
71.8245.77
75.8013.36
70.29+4.77
70.07:4.65
71.25+4.47
73.3925.1 1.7
Dinmeter
50.83+5.11
22.6644.72
51,1264885 85
51,3764.68
59,43:2.32
55,5244.35
55,0964 23
56.2943.86
52.054711
H
50.49
32.1 12.41 10
31.4364.30
31,62a4.03
42.53x2.94
38.2143.90
38 38.9363.00 70
38.4423.31
31.8564.34
E
84.0743.82
88.8712.80
88.7942.88 82
87.4213.00
92.6512.03
87,3111.55
91.11 21
86.3411.99
89.0812.76
Shoetest
M
80.93433.00
82.8343.74
81.91+3.79
81.03+3.90
85.7753.19
84.8699.11
84.3433.27
83.4543.41
83.0943.71
1I
76.76x:1.53
79.052:448
78.22a/1.56
76.281:1.86
77,901:1.12
75.691:1.5
76.051:1.22
73.70u.1.00
78.23+:1.81
E
48.50:44.79
48.20444.65
47.2944.80
47.11+5.09
46.9644.77
45.2945.08
44.68e5.
43.2545.25
48.3364.55 58
Triangle
M
27.2552.82
26.6242.82
26.3842.91
26.9243.44
26.4752.58
25.2512.98
25,0212.66 66
24.9842.90
26.67a2.88 88
H
14,04.1.23 22
18.54x1.20
14.23x1.14
14.23A1
13.11a1.26
12/4611.29
11.34a1.11
12/41a1.38
14.0411.25
E.2 PERFORMANCE HEATMAPS ACROSS TASKS
In this section, we provide detailed visualizations of model performance through heatmaps, extending the example shown in Figure 4. These heatmaps illustrate the interaction between prompting schemes and serialization formats across different tasks and difficulty levels, offering a comprehensive view of how various methodological combinations affect model performance.
E.2. HEATMAPS FOR BFS order TASK
As shown in Figure 8 (featuring Claude-3.5, GPT-4,, GPT-4o-mini, Gemini-2.0), Figure 9 (featuring Llama-3 (8B), Llama-3.1 (8B). Mistral (7B), Phi-4 (14B)), Figure 10 (featuring Qwen-2.5 (7B), o4-mini), the heatmaps compare different prompt strategies and graph serialization formats under easy, medium, and hard difficulties for the BFS order task. The color intensity encodes accuracy (darker higher), and solid/dashed boxes highlight best/second best combinations, respectively.
42

---

Published as a conference paper at ICLR 2026
Table 21: Benchmark Results of Serialization Formats Across Tasks of Closed-source Models (Mean+95% CI Margin), Bold orange Underlined blue highlight: second best performance
best performance, Underlined and blue
Difficulty
AL
ES
GMaL
GMoL
E
92.88=1.83
81.89+3.88 82
92.68+1 81
78.90=3.25
82.02=3.12
85,90+2.55
70.12+3.57
BFS order
74.53+3.82
49.4344.43
43.71+3.56
53.07=3.89
51.34+3.83
63.09+3.86
49.3944.33
35.7943.30
9.67+1.14
36.16+3.18
19.14=1.89
15.26=1.50
24.81+2.21
20.02+2.24
E
98.11=0.54
89.74#1.12
98.03+0.54
95.74=1113
96.5320.99
96. 16.11
89.27=1.78
Connectivity
97.70:0.54
90.73:0.99
97. 59
96.32=0.89
96.9620.83 83
95.81
13.53+1..0
95.4620.98
71.48+2.03
95.19=1.00
94.01=1.09
1.29
93.39:1.48
90.94
E
72.1243.34
74.92:3.33
71.72+3.33
83.66=2.45
87.60t 1.93
91.29:2.34
82.83+2.51
Cycle
71.27+3.45
72.9413.32
70.82+3.43
84.19=2.40
85.26*2.40
90.03+2.39
85.48+2.35
69.9113.34
19.19.
68.1743.35
84.81=2.42
81.23+2.64
87.34+2.64
86.94+2.62
73.80s4.05
72.10+4.41
77.50
68.80a4.60
69.47
74.3343.76
72.300+3.99
Diameter
57.7544.11
50.53+3.49
60.91
47.8243.74
46.15=4.07
57.2843.559
66.2443.66
41.04+3.81
29.1743.62
42.25a3.30
28.3242.59
25.52+2.79
40.03+3370
39.1543.24
96.21=1.37
74.70+3.14
95.78
91.61=1.89
89.61=2.27
84, 84,10&5 04
36.86+1.44 1.4
Shortest
93.72+1.32
59.50+3.52
89.45+2.00 34
88.76=1.72
82.18+2.96
1.81
78. 78.6772..45
89.85+2.06 040
41.52+3.27
89.97=1.92
80.77=2.87
76.84+3.50
83. 13.1542.67
76.04
48.8344.00
41.24+4.50
49.80=4.18
47.3624.18 18
43.95s4.27
47.6554.115
57.53+4.59
Triangle
27.47+2.30
20.06#1.00
27.69+2.33 33
29.13+2.55
24.48=2.45
27.10=2.61
27.28+2.98
13.5620.96
8.2420.96
15.38=0.86
15.28=0.90 98
11.97=0.88
14.92+1.15
15.11
Table 22: Benchmark Results of Graph Type Across Tasks of Closed-source Models (Mean+95% CI Margin). Bold orange Underlined blue highlights indicate best/second-be performance. "L" indicates the graph type is not applicable for that task.
Task
Difficulty
BAF
BAG
BERM
BERP
ERM
ERP
SF
83.6241.34
83.4121440
84.9821.23
85.73=1117
17.72+1.50
83.4541.29
1..16:t1.29
BES
75.54=1.65
63.93+1.91
55.38&1.81
67.58=1667
45.67=1.68
48.9341.68
60.08
61.08=1.79
18.3621.33
16.07=1.06
28.99=1
10.62+0.91
20.46+11.2
19.04#1441
92.9010.74
95.59:0.47
94.17=0.57
96.03a0.40
96.18:00.2
Connectivity
93.2940.66
97.0620.32
96.53=0.39
95.53a0.40
95.9210.45
83.06t1.27
92.3620.87
91.62=0.88
92.49:0.74
95.46220.99
81.22+1.34
81.07+1.20
78.68=1.23
82.18+1.40
83.7741.20
76.63+
Cycle
77.94+1.99
83.95&1.17
77.5=1.27
82.07+1.41
1..8241.14
75.76+1.3
71.11+1.99
83,02+1.19
83.38+1.14
79.35 1.36
80.6741.33
70.99+1.41
71.21+1.76
79.98+1.25
Diameter
53.1 13.15e1..66
48.55+ 1.74
$7.07+1.55
56.48+1 1.47
H
34.07+1 62
34,54+ 1.74
42.31+1.77
29.36=0.94
90.09+1.26
89.28+1.17
85.05+1.24
88.66=1.23
86.36#1.20
90.60:1.099
88.84+1.20
Shortest
M
85.09+1.55
85.51+1.26
80.90=1.17
80.35=1.19
79.25+1.44
83. 33.96+1 1.26
16.53+1.31
H
80.55+2.03 03
80.74=1.66
74.31=1.54
75.11=1.75
71.53+1.69
75.63+1.48
1..2
$1.58+1.78
29.66 29.68+1.53
37.19+
68.04+1.80
Triangle
M
34. 1..1311001
14.28+0.59
18. 18.81.0.755
3..47=1.66
17.97:0.62
8.48+0.33
13.06+0.48
14.45=0.60
43

---

Published as a conference paper at ICLR 2026
Peeformanne Analysis: Claude-3.5
Beal
Swcend Seat
Exary
Mediua
1hard
1:
AMC
0144
4.24
STM
KCM
0:00u
ABC
8:50
0157
0.55
6.20
4.14
Phongp
8:04
CAC
418
420
44T1
CAN
130
674
622
870
1.40
0.48
676
632
Ferress
3X0
150
3.0
C.
Performance Analysis: GPT-40 BPS order
Beal Poffomance
Secend Seat
Tasy
Modiun
Blurd
4:11P
4.11
MCTT
8.72
I:01
4:33
4120
3.2%
3.00
XUM
EPO
0:22
C1R
420
130
1.20
K73
5.00
0.70
0801
9:20
1.00
417
6.45
442
4100
410
5411
8111
4.21
4101
CMML
Seriatation Ferrata
Seral Seralization Farneb
Serial Foronta
Acason Kingo
Asusssy Kange
Accustni Rango
Performance Analysin: GPT mo-m-l on RPS order
Beal Poffrrence
9:29
EYE
CIN
Colgerthen
14:00
K75
13%
1.00
9.20
C111
6-Appritien
0.45
C14
CAM
4271
1.00
3.50
4.11
$.1M
6227
Pountp
CAFI
Ager
438
Apolter
410
CHI
410
4TH
Intral
622
1:31
428
1.00
411
LTS
511
5407
Rariokedaa Ferrets
Ser Meraalodian Foreuk
Astartay Karge
Asussay Kargo
Purformance Aualysis: Gemini BES order
Siscend Beat
Easy
Me-liem
Hard
R.GTT
R.G.T
ETC
X.GHT
0.354
1.407
$.400
$.7%4
S:I
EST
S/01
CTC
5:26
COB
012
3.30
8311
0313
820
SKY
RXC
SPG
CHT
3:20
GTR
1:20
RATI
G0T
630
6400
5:24
8321
0371
8/10
1403
ASV
1122
ANTI
CHI
2210
3:41
A311
GHR
R30
6:11
30:0
3.90
8321
0313
CM.
Accursa
Figure& 8: Performance heatmaps for prompt strategies and serialization formats on the BFS order task (Part 1). Models: Claude-3.5, GPT-4o, GPT-4o-mini, Gemini-2.0.
44

---

Published as a conference paper at ICLR 2026
Performanc lysl: Llame-3
order
Sweend Seat
Exary
Mediun
Hard
AAM
CAA
R.GTT
4277
CAM
Pronupt
sa00 criet base LOOE CANE BAICK
CCM
Performance An Analysic Ilama- 1.1 (RB) on BPS ond
Bast Purfornanco Sacard Sest
Tany
Median
3.91
6.03
122
5.12
edlcc
1.3%
CR7
512
1.0
Frored
648
4.00
0M4
OM4
Seriatatton Formida
Seralizaioon Farmeb
Seriilaat Forrata
6o 012
32 C3.
C20
C%
020
25% CiT)
1.D LXE LOU
Ancoring Rango
Performance Analysis: Mistral (78) on RPS order
Beal Porforennce Sest
1.29
C110
C314
CAkgerthen
CCN
6-Appritiee KCM
#dstroct
CAN
C4M
00.00
Agenter
COI
Agertten
641
Appritie
4235
4131
Foreads
Astastoy Karge
Aotustin Ramga
Parformance Analysis: Phi-4 (14B) on BES order
Medium
6-44prithen
R.G.T
Cdla
R.GHT
RIGHT
Prongh
6x7
3.30
411
5/13
130
0117
3.00
5427
022
3.00
4017
0181
3.80
8.10
048
3.077
6412
048
0411
<c AM
6M0
Figure 9: Performance heatmaps for prompt strategies and serialization formats on the BFS order task (Part 2). Models: Llama-3 (8B), Llama-3.1 (8B), Mistral (7B), Phi-4 (14B).
45

---

Published as a conference paper at ICLR 2026
Performance Analysic Qwnn2.5 (723) order
Beal
Soeend Seat
Exrry
Mediua
Stard
1:74
212
B-007
0161
8:03
CAB
3:21
C211
8400
0140
8:20
422
010
Prongp
C4S
CM
1:00
422
130
640
C211
8:14
010
C12P
CTD
X79
+20
130
Kibb
8:0
0030
1:34
550
33
Performance Analysis: Qwea2.5 5 (7B) BFS order
Bast Poffrrance Sacerd Best
Taay
Modiun
ICXT
CATB
3.00
CRRE
IG0T
CRR
9-001
$.0%
3.00
C4t9
edlcct
CA12
K700
C3H
0013
Fronge
0419
GRA.
Serattadon Ferrata
Sera Seralizaton
Serial Famenn
6C2
6%
Range
Performance Analysis: Owen's (BT) on BPS order
Beal Pofforennce
Beat
1.00
A-Aierttrr
IAlgerttin
1:54
5.30
0.14
4.75
5:30
ESR
9.54
0.58
0.500
Frange&
KCW
Apprttion
330
0.57
642
140
420
1:01
422
822
6.14
A1
40
05
(M6)
08
Sorial ondlern Farrask
Auoussy Rengo
Perform Analysis: od-mini on BFS order
Beat Porformanee
Mediun
Staed
Migertte
R.GTT
S/%
617
4TR
6.CoT
4.755
1447
41N
STM
4.54
4.50
0.44
Phongg
SUR
ERO
C112
0:027
627
4.00
030
1301
438
S14
GRZ
CHF
0:58
610
4244
4310
640
3.0
Accursy
Figure 10: Performance heatmaps for prompt strategies and serialization formats on the BFS order task (Part 3). Models: Qwen-2.5 (72B), Qwen-2.5 (7B), Qwen-3 (8B), 04-mini.
46

---

Published as a conference paper at ICLR 2026
E.2.2 HEATMAPS FOR Connectivity TASK
As shown in Figure 11 (featuring Claude-3.5, GPT-4o. GPT-4o-mini, Gemini-2.0), Figure 12 (featur- ing Llama-3 (8B), Llama-3.1 1 (8B), Mistral (7B), Phi-4 (14B)) Figure 13 (featuring Qwen-2.5 (7B), o4-mini), the heatmaps compare different prompt strategies and graph serialization formats under easy, medium, and hard difficulties for the Connectivit ty task. The color intensity encodes accuracy (darker = higher), and solid/dashed boxes highlight best/second combinations respectively.
Performance Amalysic Claude
Connectivity
Best Perfornazee
Socardl Best
Mediun
1::
LSS
LOS
9-C07
9:01T
1.00
1:411
12W
1:0
LSR
LOS
1:00
0.00
Fronue
Sariaization Forrats
Fareakk
can Range
Car
Asouraay
Per formance Aualysisc GPE-o Comnectivity
Rea Putfereanaa
LOM
COC
Phanye
SUR
EAT
CED
3:0
SCT
1100
SCN
LTV
1W0 CAC
UM
See Lnlion Far eark
Datitin Fareusk
Lastinn Ferrass
Acceracy Range
Asoursy Range
Acousscy Rarge
Perfermance Analysis: GPT- mo- on Connectivity
Parforracce
Mediun
Iladd
0124
SIR
CR1
DvVige
L:7
L1P
CHF
Proppt Porpt&hhees
SAR
LBC
CRF
S12
410
487
CHF
SPS
LTD
CRV
CTN
3UF
411
CH1
120
SPS
SAY
CTD
Laadon
Foreak
aadoon Farrats
Acoursty Pange
Accurna Sange
Accurecy flange
Peeformance Analyisc Gemini-2.0 Connectisity
Sacord Sext
Exsy
Mediun
Ilaed
CYR
64NY
1:2
CIF
CPI
CRT
Prongt
STM
ATM
412
CT1
CHT
2:24
CHI
CAN
130
12%
Figure 11: Performance heatmaps for prompt strategies and serialization formats on the Connectivity task (Part 1). Models: Claude-3.5, GPT-40, GPT-4o-mini, Gemini-2.0.
47

---

Published as a conference paper at ICLR 2026
Peeform Analysiic LJam-33 Connectivity
Secend Boot
Easy
Medlum
EEXT
1:24
1:04
0.CuT
CT3
1:3
XMI
CM
C/4
410
472
L23
Ferrusk
3.3
203
Performance Analysis: Llama Connectivity
Bast Pofforancc
Sacerd Best
Toty
Mediun
241
C11
CTD
2:M
CTR
9-01
180
ASCT
AZD
CRR
SUITT
CTR
00B
G1T1
2:20
STTP
Fronge
CHR
4TR
CED
Serialazdon Ferrata
alizatinn Farmeb
Serial catinn Formu
2%%
LG
012
2.
Ab
012
Accuncy Range
Performance Analysis Mistral (7B) on Connectivity
Beal Porforennce Best
Migertte
CAlperiner
ECM
COF
6-ipurth
1/11
EXI
CITT
Frangn&dheees
2:21
E:5
G4
E15
GCM
9:20
ESLT
A:21
CMAL
AM
Serialackan Ferrets
Scciil
Acnniag Kinge
Ausesay Kenge
Antuston Kargs
Performance Analysise Phi-4 (14E) on Connectisity
Sisced Best
Mediun
Migertie
R.G.T
S.G.T
GAM
KCM
Phongrt
2017
142
STO
4112
0137
6:20
KG08
6:14
1400
Acoussy
Figure 12: Performance heatmaps for prompt strategies and serialization formats on the Connectivity task (Part 2). Models: Llama-3 (8B), Llama-3.1 (8B), Mistral (7B), Phi-4 (14B).
48

---

Published as a conference paper at ICLR 2026
Performence lysis: Qwee2.5 (7285 on Connectivity
Beal
Sweed Seot
Mediun
Tlaed
3F3
RLD
CYF
6407
L:2
CLV
CINE
110
MTM
CMI
2:4
CMA
Frongg
CXC
2:5
CMS
2:20
CRS
CAN
C13
Performance Analysis: 2wea2.5 (7B) Connectivity
Bast Pofforance
Sacard Best
Modiun
1:00
LF7
C:1M
41C
9:00
9.33
6:21
SF3
C22
LWT
3.00
C:20
CEF
CUP
83%
5:21
XTD
CDI
Fronge
<22
417
SR7
410
A:0
AXIT
Kobbal
Serialastio Ferrata
Siral iiatiiaion Farmeb
Serial calon
339
2/
20
532
20
Accuraty faange
Performance Analysis: Qwen' (BB) on Connectiviyy
Seal Porforencce Sest
Migontt
Cokgerther 1:00
EXT
6-Aporttee
9:27
ETTI
SSH
TUN
1/01
KXW
LTV
A/
Se
la Farredk
Suadeon Farealk
Rarge
Performance Analysis: od m-mini ou Connectivity
Siscend Best
MedBum
Hlard
R.GTT
120
SR2
Phonep
414
EUC
SRR
XX0
CHH
120
100
110
10R
1,00
LRD
410
312
Figure 13: Performance heatmaps for prompt strategies and serialization formats on the Connectivity task (Part 3). Models: Qwen-2.5 (72B), Qwen-2.5 (7B), Qwen-3 (8B), 04-mini.
49

---

Published as a conference paper at ICLR 2026
E.2.3 HEATMAPS FOR Cycle detection TASK
As shown in Figure 14 (featuring Claude-3.5, GPT-4o, GPT-4o-mini, Gemini-2.0), Figure 15 (featur- ing Llama-3 (8B), Llama-3.1 (8B), Mistral (7B), Phi-4 (14B)), Figure 16 (featuring Qwen-2.5 (7B), 04-mini), the heatmaps compare different prompt strategies and graph serialization formats under easy, medium, and hard difficulties for the Cycle detection task. The color intensity encodes accuracy (darker = higher), and solid/dashed boxes highlight best/ combinations respectively.
Performance Analysis: Claude-3 Cycle
Buc Perforsazeo
Sicerd Bost
Easty
BCXT
1.32
2.34
CSTL
9-G07
0124
1:301
411
9:021
0.34
01100
3.27
0.1
1
CSTE
3.M4
0113
CST
8:211
1::00
9.27
412
Fronee
Seriktadon Forrats
Sur aaraaization
Serial
XXD
20
6.
230
Pur formancee Aualysioc GPE-o on Cycle
Rasd Putforeanaa
Bost
Exsy
CASE
LDV
CASS
615
1125
Pronept Schorre
CER
CSN
440
6A0
4413
CAU
CAM
CHI
0130
1400
RPC
1:07
Serial Farmeds
Asounny Range
Acursoy Range
Actursy
Purfor munce Amilysis: GP m-i-mini on Cycle
Parfurnance
Mediun
1:41
EXT
6:14
1:61
X30
DST
LTH
STD
L2R
1:00
KKT
476
3:24
4111
112
JUT
ER7
410
LST
ER7
AM
For
Seralation Foruus
Ascursy Kange
Accuriig Range
Accursay
Performancee Analysis: Geminl-2.00 Cycle
Second Best
Mesiun
Hard
RCNT
3:4
CH1
8:Cx7
SFT
STLL
6.10
SF2
CHC
0:30
4:20
SAIT
6.10
C
ENTT
S:24
154
6.00
Kddel
5:0
Accursoy
Figure 14: Performance heatmaps for prompt strategies and serialization formats on the Cycle task (Part 1). Models: Claude-3.5, GPT-40, GPT-4o-mini, Gemini-2.0.
50

---

Published as a conference paper at ICLR 2026
Performance Analysisc LJama33 (83) Cycle
Sween Seot
Exrr
Medium
Hard
R.EXT
CASE
RIGHT
SAM
CAM
A20
CAM
4AM
94M
0:00
Pronupt
CATT
140
RAG
1:30
8331
C31
1-43
Seriikurdee
Serkil Farvalt
1.
16
Peeformance Amalydis: Unma-331 (8B)) on Cyele
Perfermane
Second Seat
Modian
OCLT
1.27
420
1:00
124
Frorgd
LTM
1:00
Seralaston Formuta
Larataionnn Formuds
224
Anarnty
CD
2b.
Acarney Range
Performance Analysis: Mistral (710 on Cycle
Deal Performance
Best
GAlgerther
E-Aporthee
4:44
5:00
STP
CAD
LTD
4211
A0
Beriokeddan Ferrels
Sodiiin Farralk
Sorial
Asianing Kange
Performance Analyiic Phi-4 (14B) on Cycle
Perforeanee
Siccedl
Medion
Migertie
R.G.T
R.GHT
X74
GEM
RIGHT
$220
B4G
Phonggt
4112
1:20
C1D
0121
3:00
San
SABI
1:00
CUF
3.01
Figure 15: Performance heatmaps for prompt strategies and serialization formats on the Cycle task (Part 2). Models: Llama-3 (8B), 1lama-3.1 1.1 (8B), Mistral (7B), Phi-4 (14B).
51

---

Published as a conference paper at ICLR 2026
Performence lysl: Qwee2.3 (7285) on Cycle
Sween Seot
Easy
Mediun
Hard
1:04
130T
C:T
Fronigh Sowerss
L:C4
CTN
CTC
6:01
1:20
14M
K411
441
CM
4758
07
037
10
3%%
Performance Analysis: 2.ea2.5 (7B) Cycle
Bast Pufforance
Sacerd Best
Taay
Modiun
1.30
CTR
6:CAT
6dbc1
A27
BATT
CPT
Fronee
XDF
Seriatiaion Fomrra
Sir Eiralizaiion Farmeb
Serislast Ferrata
L13
Accurcy Range
Accuracy Range
Performance Anslysis: Owen' (BB) on Cycle
Seal Poffrrence
Migortt
910
CYC
CPM
KSW
Golgerther
EAIT
5.%%
E-Aiorthee
K:I1
1:N1
CEC
EMI
9:10
EST
9:0
SCM
Frongn
4350
8X0
92%
CHE
6:4
EXC
KTM
ATM
LTD
A21
4/14
KGT
CAALL
Serial Forruts
la Ferrudk
Loadlon Farealk
Assussyy Range
Performance Analysis: od
on Cycle
Sicerd Best
Medium
R.GTT
EOC
4.00
CR2
510
Phonep
EOC
100
CHS
100
FU0
106
SkM
EOC
100
STW
100
KGB
100
610
4/2
Figure 16: Performance heatmaps for prompt strategies and serialization formats on the Cycle task (Part 3). Models: Qwen-2.5 (72B), Qwen-2.5 (7B), Qwen-3 (8B), o4-mini.
52

---

Published as a conference paper at ICLR 2026
E.2.4 HEATMAPS FOR Diameter calculation TASK
As shown in Figure 17 (featuring Claude-3.5, GPT-4o, GPT-4o-mini, Gemini-2.0), Figure 18 (fea- turing Llama-3 (8B), Llama-3.1 (8B), Mistral (7B), Phi-4 (14B)), Figure 19 (featuring Qwen-2.5 (7B), 04-mini), the heatmaps compare different prompt strategies and graph serialization formats under easy, medium, and hard difficulties for the Diameter calculation task. The color intensity encodes accuracy (darker = higher), and solid/dashed boxes highlight best/second-bes. combinations respectively.
Pezformance Analysia: Cluude-3.5 on Diameter
Baa Puffersanas
Modion
1:43
CAST
CTM
410
Bbiked
1
ASC
LETK
CAT1
611
1:3T
CAM
SZT
8:M
CTU
6.4
4AR
CETS
BAM
CAF
STM
Seraktade Forrais
larallaation Farmeh
Surial Farruta
Acounacy Rarge
Ascuncy Rarge
Accursay Rangs
Performancc Analysis: GP T-4o on Diameter
Dest Purformmeee
Mediun
Staed
2:2
CD2
SU4
1:2
Q2R
KIW
C22
K25
Pronget
S191
410
645
CM1
11
Apoursa Karge
Range
Performunce Anulysic GPT-io-mini Diumeter
Read Purfornazan
Bood
Easy
Medium
KCXT
3.21
X211
C3T1
1.00
BGGT
412
K20
C11
Prone Sokorr
1:311
K:21
K301
41M
671
CT1
840
120
5219
428
KTM
612
3.27
418
1.21
418
Seriatadon Forrats
iaraanaationn
Serial Farruia
3.3
Range
01
Perforaacce Analysis: Gemini-2 2.0 on Diameter eter
Poffereana
Secard Sest
Medium
CDI
E.40
SPR
SUT
CEF
CDM
640
C3R
0.20
0721
Phangg
S:N
SCS
CHF
E30
Apoitie
6012
0.09
2779
1.30
1:2
600
127
640
G10
0.09
0.20
372
133
SKY
UM
4.44
A3
Saria
Serollation Foruus
Seriakadoon Fermuss
Accursy Hange
Accunoey Flange
Asourey Rorge
Figure 17: Performance heatmaps for prompt strategies and serialization formats on the Diameter task (Part 1). Models: Cl5,de-3.5, GPT-4o, 4PT-4ommnn, Gemini-2.0.
53

---

Published as a conference paper at ICLR 2026
Performance Analysis: Llame-3 (AB) Diameter
Sweend Seat
Exsry
Mediun
Hard
X.EXT
2:10
RIG
COT
CAN
KCD
CIM
BACY
4:12
KCB
CAM
KCE
0.50
Pronupt
4221
430
424
422
K22
Ferrutt
37
028
0.5
KM
Performance Analysis: L.mma-311 1 (AB) Diameter
Beal Purforance Smcend Seat
Taay
Mediun
1.09
A201
C111
R301
C320
KTM
C3N
0799
IG0T
C30
9:007
B.C
0100
Frurgg Porrgn//heee
+-bstroct
4310
CM
1:01
0.10
3:10
CAM
64M
600
1451
Serattadon Ferrata
arlaziaton Farmeb
Serial Farmits
Astrriay Ringe
C2 Asassay Kengo
Antursay
Performance ore Analysic Mistral (7B) on Dismeter
Beal Pofforennce
Migentte
KTB
0.544
GAlgerthen 01:2
1.00
C1H
6-Appritien
KCM
113
Pountp w
64P
RAN
CRR
KAN
CON
KAN
COM
CA12
410
4413
5310
41M
C10
LTS
841
(M6)
Ferrols
Astattay
Autussy Kengo
Performance anncomaacce Analysis: 4hi-4 (14B) on Diameter
Second Best
Easy
Mediun
Hard
Mdgertt
C-lieritis
R.G.T
6G
R.GHT
Phonep Prongrtchheee
120
600
ASH
CXT
1:00
1:01
Figure 18: Performance heatmaps for prompt strategies and serialization formats on the Diameter task (Part 2). Models: Llama-3 (8B), Llama-3.1 3.1 (8B), Mistral (7B), Phi-4 (14B).
54

---

Published as a conference paper at ICLR 2026
Perfor Analysisc Qwwe2.5 Diameter
Beal
Sweend Seat
Exsry
Medium
Iard
9400
6.22
CAR
14Q
4.77
6.30
9:24
9.51
4240
Prongp SoapprSoiiess
1:24
ATM
6:7M
1.00
6.21
A72
AMI
6TM
KTM
0.42
1.30
9411
6:77
150
X7D
0.05
447
422
4.50
1:30
of
Foreat
Performance Analysic: Qwen2.5 Diamettr
Bast Puffrrance
Sacerd Best
Taay
Median
ICXT
C319
0.00
AZN
1:21
X32
STW
8/27
K20
C31
9:01
8.10
X-21
4177
edlcct
xzn
KZM
1331
K307
41D
1001
K2M
C30
510
5010
Frored
601
Seriatadoo Ferrats
Ser daraaization Farmeb
Serial onl
C2
EX
5/5
Accurac Range
Performance Analysis: Owen' (RI)) on Diameter
Beal Pofforencce Seat
Migortt
CCR
Golgerther
RTM
6-Apprithee
0:15
02:00
13200
0.311
Frangh&Shhees
C11
0:24
LTD
1:01
CTR
13.05
0441
COR
1:24
SWI
Apprithee
035
6.24
62m
1:51
428
C112
CHI
CAMI
LT7
642
412
A3
05
(M6)
Bariokedaan Ferrels
Sar loaaian
Sorial
Auussay Kange
Pezformance Analsis:: oi-mini Diameter
Beat Performanee
Mediun
Hardd
E-iporth
R.GTT
$/1
8:CoT
$250
4.54
0.00v
1:40
CPC
4200
4.40
Phongg
1.0
ABC
C1D
CNI
4201
3301
0.5
CH2
610
140
100
CTR
ENS
K7U
4200
3337
0287
6329
110
Range
Accursy
Figure 19: Performance heatmaps for prompt strategies and serialization formats on the Diameter task (Part 3). Models: Qwen-2.5 (72B), Qwen-2.5 (7B), Qwen-3 (8B), o4-mini.
55

---

Published as a conference paper at ICLR 2026
E.2.5 HEATMAPS FOR Shortest path TASK
As shown in Figure 20 (featuring Claude-3.5, GPT-4o, GPT-4o-mini, Gemini-2.0), Figure 21 (featur- ing Llama-3 (8B), L (8B), Mistral (7B), Phi-4 (14B)), Figure 22 (featuring Qwen-2.5 (7B), o4-mini). the heatmaps compare different prompt strategies and graph serialization formats under easy, medium, and hard difficulties for the Shortest path task. The color intensity encodes accuracy (darker = higher), and solid/dashed boxes highlight best/second- combinations, respectively.
Purformance Amalysisc Claude-3.5 Shortest
Bus Perforsazio
Sicerd Bost
Easyy
Mo-iion
CSR
CSC
3-C07
4SC
CATE
9:02T
0.142
1.24
CSR
4071
A112
414
Fronee
Serabaado Forruts
Saraalaation
Sorial
Accuracy
Accunsy
Per formonomnc Analysis GPT-cu Shortast
Raa Putforeanaa
CCTE
EXT
CON
1:2
LSN
CAFE
RCM
Pronept Schhrre
C4Tk
1:211
CET
6:M
41M
CPM
CFF
0100
1:01
CED
11:0
SPC
619
Ferrets
Seraliation
Sorial Farrdt
Acournny Range
Acoussy Rango
Accursyy Range
Peeformance Analysis: GPT-4u-mini on Shortest
Bast Pufferaaza
Medion
CTC
4327
1:53
Pronpt
Agerddi
SIX
CUP
1.20
127
132
51R
A:21
1.0
41FF
642
422
1M4
Saris
aaraaaation Fareak
allalaaionn Farrats
Assusay Range
Ascursay Raage
Ascuraay Range
Performancee Analysis: Gemini- 2.0 on Shortest
Sacord Sext
Tasy
Mediun
Ilard
1.423
CETB
CAFE
8:C07
5:44
CAF
CASI
0110
SII
5:30
CTN
627
5:79
CYR
6110
+10
5:0
CYP4
1150
61:
010
32
40
Figure 20: Performance heatmaps for prompt strategies and serialization formats on the Shortest task (Part 1). Models: Claude-3.5, GPT-4o, GPT-4o-mini, Gemini-2.0.
56

---

Published as a conference paper at ICLR 2026
Performence Anslysicc Llame-3 on Shortest
Sweend Seat
Exor
Medium
Strd
R.EXT
CIM
R.CTT
3.M1
6.IK
SUCK
8.GUT
4.10
CALE
140
1344
K3M
0:00
1:21
435
Pronupt
Kibbi
010
Performance Analysic Llama-3.1 (RB) Shorteet
Bast Pofforance
Sacend Best
Tasy
Median
1.29
133
I-01
EX0T
CD4
STM
K7Z
1.07
KG01
420
4:0P
61R
edlact
42R
1336
427
K307
3.01
K25
Frured
4007
616
3.00
CM-
GMM
Serittatio Formdd
Seria Saralization Farmeb
Serial cnlen Forrata
Rang
Auunsay Keage
Assustay Rango
Performance Analysis: Mistral (7B) on Shortwct
Beal Poffrrence
1.311
AXT1
C119
9:0M
1.20
GAkgerthen
1327
2:M0
C231
6-Apprttie
6.14
$1.5
9.505
#dstroct
6744
C1N
5.80
C195
0057
5.01
KXM
C4N
5.00
CAM
60M
05.0
Agentem
CAR
Agertten
6138
1.83
k37
428
3000
Appritie
$20
3.20
4122
CBR
3.80
4126
300
1.001
010
CTH
5278
418
C1M
LT0
611
5475
410
1407
ORR
Ghd
A3
05
(96)
08
Reriokedaan Ferrals
Seri/ladian Foread
Sorial
Asarioy
Assassy Kango
Antusaay Nangs
Performance Analysis: Fhi-4 (14B) om Shortest
Siscend Best
Mediun
6-4pprtteen
R.C.T
R.G.T
R.GHT
$4.0
ProngrtScheee
CC1
C4F1
0,79
SF2
C1R
3.99
0.64
0.00
01:0
0.00
670
0711
1:01
440
676
K/08
647
010
340
010
Figure 21: Performance heatmaps for prompt strategies and serialization formats on the Shortest task (Part 2). Models: Llama-3 (8B), Llama-3.1 1 (8B), Mistral (7B), Phi-4 (14B).
57

---

Published as a conference paper at ICLR 2026
Performancs Anslysis: Quan2.5 (728) Shortest
Sweend Seat
Zxrry
Mediha
Stard
R.C.Y
1:2
BAM
CAM
L:M
SKV
Pronupt
442
CM
9:51
5:M
47X
617
1:01
012
2:21
410
Fareatt
Performance Andlysic 5 (2B) on Shortest
Bast Poffrrance Smcend Sext
Taay
Mediun
CATB
CEN
I:07
2:81
KBS
2.00
CRR
9:001
DCTV
XF3
SCW
Zan
4112
CTH
41N
0:08
4.25
412
Frored
6311
C220
4114
1.80
412
6:20
KMM
410
411
3:34
600
Serittadon Ferrata
Seralization Farmeb
Serial Farto
Accuray Range
24
C2
Performance nacemmance.aaco Ana lydis: Qwenk (RR) on Shortast
Beal Poffrrence
Seat
9:21
LTEE
CEFT
CAlgerther
CCT
KINT
6-Agoritee
11/5
61M
1.29
5:M4
CTN
CAN
5:00
COM
6.10
01:00
5
CAF
RAD
4ET
6414
5:377
CHN
K2M
CAM
Appritin
645
SUDT
CM
CA12
CAR
CHI
1:21
AM
Seciokedaan Ferrals
Sar Aodiian
Sorial
Asarooy Karge
Auussay Kango
Aotusaay Kanga
Performance Analysis: o6-mini on Shortest
Porforence
Siccedl
Mediun
4012
SUN
Phonger
SU0
CSP
212
RXD
012
L11
1:01
KG08
610
3:80
ADC
012
Figure 22: Performance heatmaps for prompt strategies and serialization formats on the Shortest task (Part 3). Models: Qwen-2.5 (72B), Qwen-2.5 (7B), Qwen-3 (8B), 04-mini.
58

---

Published as a conference paper at ICLR 2026
E.2.6 HEATMAPS FOR Triangle counting TASK
As shown in Figure 23 (featuring Claude-3.5, GPT-4o. GPT-4o-mini, Gemini-2.0). Figure 24 (featuring Llama-3 (8B), Llama-3.1 (8B), Mistral (7B), Phi-4 (14B)), Figure 25 (featuring Qwen-2.5 (7B), 04-mini), the heatmaps compare different prompt strategies and graph serialization formats under easy, medium, and hard difficulties for the Triangle counting task. The color intensity encodes accuracy (darker = higher), and solid/dashed boxes highlight best/second- combinations respectively.
Performance Analysis: Claude-3, 3.5 on Triangle
Buc Puffersanaa
Modicn
1.8perta
C411
1.20
C21
0.41
$250
C214
8.11
511
4124
Bbiked
A321
6477
0.44
320
C2T
6377
4277
0.21
0138
9.20
14M
Ape
416
130
C40
1338
1311
510
1334
12H
Seriktadan Forrals
Farmul
Farruta
Acouracy
Ascurca Range
Performance Analysis: GPT- on Triangle
Mediun
Haad
CII1
Cokpertten
120
KDM
0.01
5-Apprithe
6CCT
C4T4
330
C3D
1338
K327
1371
K20
Pronget
K:31
GoT
C313
C3R
0.51
C10
Kibel
0111
0121
6112
Peeformance Analysic GPT--o-mini Trisngle
Best
East
Modiun
R.CTT
4307
B-GHT
6:011
611
6:01
Proner
K2M
648
4118
CAU
3001
840
065
6/4
21
C.3
3/5
Purformance Analysis Gumini-2 Triangle
Bai
ddAgeeiteon
1200
$234
6.Aggriteen
4/11
Prante Poans
0440
CRM
641
413
3.24
820
130
6112
3/36
0.13
3.00
3317
8351
034
A77
0.51
5701
0964
Seralladan Farnuts
Sar Dattalattionn Fareulk
Sertal castinn Farrssi
Apaunaa Range
Actuspy Range
Acceracy Rango
Figure 23: Performance heatmaps for prompt strategies and serialization formats on the Triangle task (Part 1). Models: Claude-3.5, GPT-40, GPT-4o-mini, Gemini-2.0.
59

---

Published as a conference paper at ICLR 2026
Peeformance Analysic LJama33 Trisngle
Sweend Seat
Exsry
Medium
Hard
R.XXT
6:53
CAM
RIGHT
CAR
Pronupt
110
6:50
2001
Kibbal
2401
Performance Ani lydds: Lnma--311 (RI) on Triangle
Beal Porforance
Secend Seat
Taay
Median
ICXT
3773
CTR
412
9:001
5.CH
edlict
8730
K30
KOW
Hi
2.80
COO
3421
Frured
CAM
3401
C41
6424
3:01
608
140
Seratadoo Ferrata
Serial Farmata
S13
C22
5CC
Performance Analysis: Mistral (710) on Trinngin
Performance
KTV.
edsttrct
RP1
K.04
C4D
CA12
0,00
pprttrr
Apeetr
CATT
COT
130
6011
5AM
41N
KGT
0.0M
K00
1TM SAM
LTM K011
6011
5407
044
ORL
086
dide Pernuls
Performance o Analysise Fhiid (14B) Triangle
Porforeace
Siacenal
Mediun
Hard
Adigerttee
R.GHT
C.1A
R.GHT
RIGHT
0.16
411%
1/10
64M
8415
Phongg
CAT
3790
4013
C10
0428
3:90
5700
8413
50:0
3.80
022
3330
0112
92H
540
K7G
4111
840
347
KT1
512
VX
CM'S 1M0 4.0 ONE
3/M 4.5%
AM
014 CM
Figure 24: Performance heatmaps for prompt strategies and serialization formats on the Triangle task (Part 2). Models: Llama-3 (8B), Llama-3.1 1 (8B), Mistral (7B), Phi-4 (14B).
60

---

Published as a conference paper at ICLR 2026
Performence Amalysicc Qwea2.5 (728) on Triangle
Sweend Seat
Easy
Mediun
Stard
110
4.22
GOXT
1.24
4211
6407
616
420
Killed
640
1:20
6:18
64M
630
Performance Analysic 0wen2.5 (78) Trinngle
Bast Pofforance Secend Sext
Taay
Median
ICXT
KTD
C33
1.00
CTD
9:00
edlcct
K30
630
Odeal
3.73
0550
Frured
C1M
4.00
640
3427
Serattadon Ferrata
allaaaiion Farmeb
Serial onlinn Farmata
330 CT1
200 025
212
022
Accusay Range
Performance Analysin Owen' (8T) on Trinngln
Beal Poffrrence
1330
411
#dstrect
CAM
616
4.55
0.99
6.9
K110
4.5
C115
Agenttr
K:01
CAR
CAT1
Agrithe
4.59
1.90
6.54
K2M
6:14
5.54
43H
KGI
6.71
4.18
1.04
4.31
130
1.24
UM
4112
4233
5.24
C1M
616
A.
ON6
Sar Ferrush
Range
Astaaaay Range
Rargee
Pezformance Analysis: o6-mini on Triangle
Siscend Best
Eues
Me-lin
Itaed
Migertte
C-Aipritins
R.G.T
R.CxT
4.10
CAM
3.077
4.31
CBC
510
647
612
C22
0.00
6.50
0321
0.00
SWI
CxT
5430
6:20
CR7
610
OC7
0.03
6330
5.21
140
120
310
6100
610
Kdba
6001
0.00
435
AM
68
68
Accursa
Figure 25: Performance heatmaps for prompt strategies and serialization formats on the Triangle task (Part 3). Models: Qwen-2.5 (72B), Qwen-2.5 (7B), Qwen-3 (8B), 04-mini.
61

---

Published as a conference paper at ICLR 2026
E.3 GRAPH TYPE SENSITIVITY ANALYSIS
While our main heatmaps analyze interactions between serialization formats and prompt schemes, the role of graph types cross-factor analysis requires different approach. Creating individual heatmaps for each graph type x task x difficulty combination would yield over 100+ visualizations that would be comprehensive but impractical to interpret. Instead, we introduce a sensitivity- framework that quantifies how graph types respond to factor variations while maintaining both interpretability and extensibility.
Methodology. For each graph type in given task fask-difficulty setting, we compute two metrics by averaging across all models:
Prompt Sensitivity (Sp): For each serialization format, we calculate the standard deviation of accuracy across different prompt schemes, then average over all formats. This measures how much performance fluctuates when changing prompts.
Format Sensitivity (S:): Symmetrically, for each prompt scheme, we calculate the standard deviation across serialization formats, then average over all prompts.
We visualize each lask-difficulty combination as a scatter plot in (Sp; Sr) space, where each bubble represents graph type, and color encodes mean performance. Using median splits, partition the space into four interpretable quadrants: Robust (low Sp, low Sj). Prompt-Criticaal Critical (high Sp, low St), Carmat-Criticaa (low Sp. high St). and Both Critical (high Sp, high St).
Key Findings: Figures 26-31 present plots covering all task-difficulty combinations. Based on the analysis of these data, arrive the following insights.
1. Open-source models are much more prompt-sensitive than closed-source ones. Across different tasks, the prompt sensitivity range of open-source models is consistently larger than that of closed-source models. For example, in the BFS order Medium setting, the prompt sensitivity typically falls between 0.12 and 0,16, whereas that of open-source models ranges only from 0.02 to 0.05. This indicates that closed-source models rely more heavily on using an appropriate serialization format to achieve strong performance.
2. Closed-source models are more sensitive to serialization format than open-source models. Across tasks, the format sensitivity range of closed-source models is generally higher. For instance, in the Di Diameter calculation aicuuaation - Easy setting, format sensitivity falls between 0.03 and 0.06, whereas open-source models range from 0.15 0.19. This suggests that open-source models depend more on advanced grompt-enginess strategies to improve performance, while closed-source models gain more from suitable serialization formatting.
Notably, the difference in sensitivity between open-source and closed models can be explained by how LLMs typically process graph reasoning tasks, which can be viewed as involving two stages: (i) understanding the task itself, and (ii) interpreting the graph-structured input. Closed- source models, due to their stronger reasoning capabilities, encounter fewer difficulties in task understanding: as a result, they are more sensitive to the information contained in the graph data-i.e.. the serialization format. In contrast, task understanding plays more significant role in open-source models, and prompts exert more direct influence on this stage than serialization formatting. leading to greater prompt sensitivity. This interpretation is also consistent with our earlier finding-Finding 3: Open-source models benefit from multi-shot exemplars, whereas closed-source models do not. Closed-source models do not require additional exemplars to grasp the task, whereas open-source models rely more on examples to enhance task comprehension.
Extensibility. This framework directly supports GraphOmni's extensible design. When adding new graph families (e.g., real-world networks), researchers can apply the same analytical pipeline to assess sensitivity profiles before conducting full evaluations. Complete implementation details and visualization scripts are available in our code repository.
62

---

Published as a conference paper at ICLR 2026
order Easy (Closed-Source Models)
Easy (Opan-Source Models)
Graph Type Sensitivity Analysts
Graph Type Sensitivity Analysis
Format-Critical
Critical
Format-Critical
Both
Rdd
Format Sensitivity
0.13
Format Sensitvity
0.11
0.30
0.09
6.70
0.05
0.08
Robust
Prompt-Critical
Robust
Prompt-Critical
0.04
0.05
0.06
0.07
0.08
0.10
0.12
0.14
0.16
Sensitivity
Prompt Sensitivity
(a) Easy Closed-Source Models
(b) Fasy Open-Source Models
order Medium Mode (Closed-Source Models) Graph Type Sensitivity analysis
order Medium Mode (Open-Source Models) Graph Typa Sansitivity Analysis
Format-Critics
Critical
Cormat-Critical
0.36
6/22
0.15
Format Sansitivity
333
Meam Parform mancc
0.34
0.04
Format
0.33
0.32
0.02
Robust
Prompt-Critical
Robust
Prompt-Critical
8.55
0.055
0.060 0.065 0.070
0.055 0.de0 0,085
0.03
0.04
0.05
0.06
0,07
Prompt Sensitivity
Prompt Sensitivity
(c) Medium Closed-Souree Models
(d) Medium Open-Source Models
BFS order Hard Mode (Closed-Source Models) Graph Type Sensitivity Analysis
BFS order Hard Mode (Opan-Source Models) Graph Type Sensitivity Analysis
Format-Critical
Format-Critical
Format Sansitvity
0.34
Performance
HEPP
Format
0.08
6.65
Prompt-Critical a.b7
Robust
Prompt-Criticall
004
abs
0.00
0.61
0.02
0.03
0.54
Prompt Sensitivity
Prompt Prompt Sensitivity
(e) Hard Closed-Source Models
(I) Hard Open-Source Models
Figure 26: Graph type sensitivity analysis for BFS order task, comparing open-source and closed- source models. This comparison reveals whether sensitivity patterns are consistent across model categories.
63

---

Published as a conference paper at ICLR 2026
Connectivity - Easy Mode (Closed-Soure, Models)
Connectivity Easy Mode (Open-Seurce Models)
Graph Type Senstivvity Analysts
Graph Type Sensitivity Analysis
6.065
Format-Critical
Critical
Format-Critical
Both Critical
Sencitivity
Format Sansitsuity
0.070
0.005
0.050
Parforr
0.065
0.055
372
0.060
0.050
Robust
Prompt-Critical 0.045 0.dso 0.055
1.00
0.040
Robust
Prompt-Critical
6.015 6.020 0.025 0.d35
0.055 0.080 0.085 0.090 0.065 0.100 0.105
Prompt Sensitivity
Prompt Sensitivity
Easy- Closed-Source Source Models
(b) Easy
Open-Source Source Models
Connectivity
(Closed- Models)
Connectivity Medium Mode (Open-Soure) Models) Graph Type Sensitivity Analysis
Type Sensitivity Analysts
For Format-Critical
Both Critical
Format-Critical
Both Critical
0.05
Format Sensitivity
Format Sensitivity
375
0.065
0.04
0.060
0.03-
0,30
0.055
0.050
3002
Robust
Prompt-Critical
0.045
Robust
Prompt-Critical
0.010
0.015
0.020 0.025
0,030
0.035
0,040 0.045
0.b8
0.09
Prompt Senaltivity
Sensitivity
Medium Closed-Soucch Sourcee Models
(d) Medium Open-Source Models
Connectivity - Hard Mode (Closed-Source Models)
Connectivity - Hard Mode (Open-Source Models) Graph Type Sansitivity Analysis
Graph Type Sensitivity Analysis
Cermat-Critical
0.09
Format-Critical
Both
0.32
Format Sensltivity
0.10-
Format Sensitivity
Mean Porforrnn Pefoormanc
0.08
0.06
0.05
0.04
0.05
0.02
Robust
Prompt-Critical
Robust
Prompt-Critical
335
0.020
0.025
0,035
0.040
0.045
0.050
0.06
0.10
012
0.14
Prompt Sensitivity
Prompt Sensitivity
Hard Closed-Source Models
(I) Hard Open-Source Models
Figure 27: Graph type sensitivity analysis for Connectivity task, comparing open-source and closed- source models. This comparison reveals whether sensitivity patterns are consistent across model categories.
64

---

Published as a conference paper at ICLR 2026
Cycle Easy Mode (Closed-Source Models)
Cycle Easy Mode (Open-Source Models)
Graph Type Sensitivity Analysis
Graph Type Sensitivity Analysis
677
Format- Critatt-rrticlll
6.7
Format Sensitivity
Format Sansitivity
Moan Parfemmi Termance
0.33
0.32
0.314
0.05-
0.30
6.72
0.09
Robust
CromittCritical
Robust
Crompt-Critical
0.640
0.045 0.650 0.055 0.60 0.065 0.670 0.075
0.09
6.10
0.11
0.12
6.13
0.14
6.15
Prompt Sensitivity
Prompt Sensitivityy
Closed-Source Models Easy
(b) Easy Open- Open-Source Models
Cycle Medium Mode (Closed-Source Models)
Cycle Madium Mode (Open-Sourca Models) Graph Type Sensitivity Analysis
Graph Type Sensitivity Analysis
Format-Critical
Both Critical
Format Sunsitiviky
0.055
0.12-
354
Moan Performand
0.050
0.045
Format
0.30
0.09
131
6.70
0.68
Robust
Prompt-Critien Critical
0.030
Robust
Prompt-Critical
0.44
0.05
0.06
0.07
0.08
0.66
0.07
0.88
0.00
0.10
0.11
0.12
Prompt Sensitivity
Sensitivity
Medium Closed-Sousee Source Models
(d) Medium Open-Source Models
Cycle Hard Mode (Closed-Source Models) Graph Type Sensitivity Analysis
Cycle - Hard Mode (Open-Source Models) Graph Type Sensitivity Analysis
0.14
Format-Critical
Both Critical
355
0.33
354
0.12
Format Sansitivity
0.060
0.11
0.055
0.30
0.050
0.09
0.040
144
0.07
Robust
Prompt-Critical
0.035
Robust
Prompt-Critical
0.3
0.64
0.d5
0.b5
0.57
0.06
0.08
0.10
0.12
Prompt Sensitivity
Sensitivity
(e) Hard Closed-Source Models
(l) Hard Open-Source Models
Figure 28: Graph type sensitivity analysis for Cycle task, comparing open-source and closed-source models. This comparison reveals whether sensitivity patterns are consistent across model categories.
65

---

Published as a conference paper at ICLR 2026
Diameter- Easy Mode (Closed-Source Models)
Diameter- Easy Mode (Open-Source Models)
Graph Type Sensitivity Analysis
Graph Type Sensitivity Analysis
Format Critical
Both Critical
Format-Criticn
Both Critical
0.08
0.07
0.07
0.70
Format Sensitivity
Meann Performance
Forma Format
0.00
0.06-
0.05
0.04-
0.04
Robust
Prompt-Critical
Robust
Prompt-Critics
0.050
0.035
0.045
0.d50
0.655
0.060
0.15
0.16
0.17
0.18
0.19
Prompt Sensitivity
Prompt Sensitivity
Easy- Souuree Models
(b) Easy Open-Source Source Models
Diameter- Medium Mode (losed-Source Models) Graph Type Sensitivity Ayalysis
Diametar Medium Mode (Opan-Seurce Models) Graph Type Sensitivity Analysis
0.31
Format Coltialr
Both Critical
Format-Critical
Both Critical
827
0.050
0.30
0.00
A.
Format Sansitvvity
0.045
325
Moan Pororannna
2.8
0.08
Format
BBC
0.07
122
0.06
6.40
Robust
Prompt-Critical
Robust
Prompt-Critical
0.53
0bs
0.07
0.11
6.13
0.4
Prompt Sensitivity
Prompt Sensitivity
Medium
Closed-Souss Sousee-Souree Models
(d) Medium Open-Source Source Models
Diameter-
Hard Mode (Closed-Soure) Source Models)
Diameter Hard Mode (Open-Source Models) Graph Type Sansitivity Analysis
Graph Type Sensitivity Analysis
0.32
Format- Critical
Both Critical
0.085
Format-Critical
Both Critical
0.11
0.050
0.30
Format Sunsitivity
0.055
Moan Purformance maraoeee
0.050
0.09
Format
0.065
0.08
0.040
0.67
0.06
Robust
Prompt-Critical
Robust
Prompt-Critical 0.695
122
0.045
adso
nos5 oden 0.665
0.070 a.755 0.80
0.655
0.070
0.075
0.des
Prompt Sensitivity
Prompt Sensitivity
Hard
Closed-Souree Models
Hard Open-Source Models
Figure 29: Graph type sensitivity analysis for Diameter task, comparing open-source and closed- source models. This comparison reveals whether sensitivity patterns are consistent across model categories.
66

---

Published as a conference paper at ICLR 2026
Shortest peth- - Eesy Mode (Closed-Source Models)
Shortest path - Easy Mode (Open-Source Models)
Graph Type Sensitivity Analyse lysis
Graph Type Sensitivity Analysis
Format Critical
Both Critical
Format-Critica
Both Critical
14
0.32
0.20
Format Sensitvvity
Pude
0.18
3.4
0.30-
6.00
0.16
Meann
0.09
0.14
0.08
Robust
Prompt-Critican
Robust
Prooptt-ritichh
0.05
0,77
0.5e
0.09
0.18
0.00
0.22
0.24
0.26
0.28
0.50
Prompt Sensitivity
Prompt Sensitivity
Easy- Closed-Source Source Models
(b) Easy -pen--pnnoss Source Models
Shortest path Medium Mode (Closed-Source Models) Graph Type Sensitivity Analysis
Shortest path Medium Mode (Open-Source Models) Graph Type Sensitivity Analysis
Cormat-Critical
Both Critical
Format-Critical
Both Critical
0.20
0.20
134
0.18
Format Sensitvvity
M eain Rufformance
0.36
Format
0.34
0.14
122
0.32
9.28
0.30
Robust
Prompt-Critical
0.10
Robust
PromptCrrtical
0.035
0.640
0.04s
0.050
o.ds
0.065
0.12
0.14
0.20
0.22
0.24
Prompt Sensitivity
Prompt Sensitivity
(c) Medium Closed-Source Models
(d) Medium Open- Models
Shortest path Hard Mode (Closed-S Slosed-Source Source Models) Graph Type Sensitivity Analysis
Shortest path Hard Mode (Open-Source Models)
Graph Type Sensitivity Analysis
Format- Critical
Both Critical
Format-Critical
Critical
0.30
0.250
6.70
0.225
0.25
Format Sansitivity
0.70
0.175
0.20
Format
0.150
0.15
0.225
0.30
8,075
0375
Robust
Prompt-Critien
Robust
Prompt-Critical
0.06
a.b0
o. olo 6.12 0.14 ale c.ss aãe 0l2 0.b8
Prompt Sensitivity
Prompt Sensitivity
Hard Closed-Source Models
Hard Open-Source Models
Figure 30: Graph type sensitivity analysis for Shortest path task, comparing open-source and closed- source models. This comparison reveals whether sensitivity patterns are consistent across model categories.
67

---

Published as a conference paper at ICLR 2026
Triangie Easy Mode (Clesed-Source Models)
Triangle (Open-Source Models)
Graph Type Sensitivity Analysis
Graph Sensitivity Analysis
Format-Critical
Format-Critical
Both Critical
0.005
0.50
Format Sansttuulty
0.060
0.03-
0.055
Format
0.050
0.02-
0.040
1.10
Robust
Prompt-Critical
Robust
Prompt-Criteen Critical
6.75
0.52
0.63
004
0.05
0.06
0.07
0.02 0.03 0.04 0.55
00e
0à7 obo 0.09
Prompt Sensitivity
Prompt Sensitivity
(a) Easy Closed-Source Models
(b) Easy Open-Source Models
Triangle Medium Mode (Closed-Source Models) Graph Type Sensitivity Analysis
Triangle - Medium Mode (Open-Source Modals) Graph Type Sensitivity Analysis
Format-Critical
Critical
0.05
Format Sensitivity
Mean Formancc anncccee
0.22
0.020
Format
0.015
0.18
0.03
0.010
0.005
Robust
Prompt-Critical
612
Robust
Prompt-Critical
0.005
0.010
0.015 0.020 0.025
0.030
0.635 0.040
0.010
0.015
0.020
0.025
0.030
0.635
0.040
Sensitivity
Prompt Sensitivity
(c) Medium Closed-Source Models
(d) Medium Open-Source Models
Triangle
Graph Type Sensitivity Ana lysis
Hard Mode (Closed-Source Models)
Triangle
Graph Type Sensitivity Analysis
(Open-Soure) Source
Critical
0.025
Format-Crftical
Both Critical
0644
0.05
0.020
Format Sansitivity
0.04
8.015
Format
0.03
Mean
0.01c
0.62
0.005
Robust
Prompt-Critical
Robust
Prompt-Critical
0.615
0.020
0.025
0.630
0.035
0.9b/%.c10m 012...c1ee 118.00 www..oow.....oom o'sae.s
Prompt Sensitivity
Prompt Sensitivity
(c) Hard Closed-Source Models
(f) Hard Open-Source Models
Figure 31: Graph type sensitivity analysis for Triangle task, comparing open-souree and closed-source models. This comparison reveals whether sensitivity patterns are consistent across model categories.
68

---

Published as a conference paper at ICLR 2026
E.4 ERROR ANS
This subsection presents in-depth analysis of common enror patterns observed in model responses. We categorize representative error cases and provide concrete examples illustrate the specific challenges LLMs face in graph reasoning tasks. This detailed error analysis extends the brief discussion of error categories presented Result Section 4.1.
E.4.1
Diameter calculation CASE
Diameter
QUERY: Given graph, your task is to determine the diameter of this graph. Graph: Adjacency Matrix
[[O
1]
1100110
11101111
1010111
01111O11
011 Q: What is diameter of this graph?
A:
TASK INFO: diameter,easy. 0-shot, Adjacency Matrix
GROUND TRUTH: 2.0
MODEL: Mistral
WRONG ANSWER: the diameter of the given graph is
explanation: to find the diameter of graph, we first find the shortest path between every pair of vertices using a shortest path algorithm like dijkstra's 's or floyd-warshall, however, since the graph is unweighted, we find the diameter by simply considering the longest path between any vertices.
in this case, the longest path we can find is: vertex -> vertex >> vertex 2 >> vertex 3 > vertex 4 > vertex 5 >> vertex
vertex vertex 8
the number of edges in this path is 8 7, the diameter of the graph is 7.
ERROR CAUSE A ANALYSIS: This answer does not accurately define the concept of diameter. In the reasoning process, it mentions "first find the shortest path between every pair of vertices." which is correct. However, the final description refers "the longest path," which can be misleading. diameter defined the longest of all shortest paths rather than the longest path between any two vertices. Therefore, this explanation does not fully align with the formal definition of diameter.
fact,
69

---

Published as a conference paper at ICLR 2026
E.4.2 Diameter calculation CASE 2
Diameter
QUERY: Given graph. your
to determine the diameter
graph. Graph: Adjacency
[0: [1, 2, 5, 10, 11, 13, 15], [0. 2, 3,4, 13, 15], 2: [0. 4, 5, 6, 7, 12, 14], 3: 1,2,
4, 7, 13, 14], 4: [0, 1, 3, 6, 8. 17], 5: [0, 1, 3, 9].
6: [2, 4. 91. 7: [2, 4, 3, 10, 14,
15. 16]. 8: 43,4, 0. 12. 16, 17]. 9: (6. 2. 10, 11. 12), 10: [0. 9, 7J. 11: [3. OJ. 12: [2, 9, 8J, 13: [3.
1J. 14: (2, 7J. 15: 17. 1, O, 16). 16: [15, 8, 17J, 17: [16, 8,4]]
Q: What is the diameter of this graph?
A: Let's break down this problem:
TASK INFO: diameter, medium, LTM, Adjacency List
GROUND TRUTH: 3.0
MODEL: Qwen-3
WRONG ANSWER:
To determine the **diameter** of graph, must identify the #*longest shortest path between any nodes in the graph. This involves computing the "*shortest path between all pairs of nodes and then finding the **maxinum** of these values.
### Step I: Understanding the Graph The graph is represented an **adjacency list** with 18 nodes through 17). The graph appears be **<onnected**, which is essential for the diameter be well-defined. We will use ** *Brcadth-Firms Search (BFS)** from each node to compute the shortest paths to all other nodes.
Step 2: Strategy For each node, perform BFS find shortest distances to all other nodes. Keep track of the **maximum shortest *h found across all node pairs. The mmaximmm**** of these distances is the **dimetter***
### Step 3: Key Observations and Calculations Through careful inspection and BFS traversal, the **Inggsst shortest path** was found to be between the following pairs of nodes:
+=Nod 12 to Node 15**; Path: 12 + 9
+ 15 (4 steps) Alternatively: 12 - 4 4
15 (4 steps)
**Nood 9 to Node 16+*; Path: 9 +
16 (4 steps)
#*Node 12 to Node 15+* and ##Node 9 to Node 16 are two such pairs where the **shortest path is exactly 4 steps** **
Other pairs were found to have shorter distances (e.g., 2 or steps). No pair was found to have a shortest path longer than 4 steps.
### Final Result Since the **longest shortest path** in the graph is +44 steps** the **dimeter*** of the graph is:
S$ (4] SS
ERROR CAUSE ANALYSIS: This result fails to find a shorter path because it ignores the two critical
odges 7-15 and 7-16 in the graph. 12 To 15 there is actually a 3-step shorter path: c.g., 12 - 2 + 7 - 16. Once these two edges taken into account, it be seen that no points in the graph have shortest path of than steps. these two edges into account, it follows that no two points in graph have shorter paths than 3 steps. Therefore, diameter of the graph is indeed 3, not
15. For 9 to 16 there is 3-step path: 9 - 10
70

---

Published as a conference paper at ICLR 2026
E.4.3 Connectivity CASE
Connectivity
QUERY: Determine if there is path between two nodes in the graph. Graph: Edge List is
3
42
54
65
75
Q: Is there a path between node and node 5?
A:
Let's break down this problem:
TASK INFO: connectivity, easy, LTM, Edge List
MODEL: Mistral
GROUND TRUTH: False
WRONG ANSWER:
1. we have graph with nodes (1 7) and 6 edges.
2. to find path between two nodes, we can use depth-first search (dfs) or breadth-fing search (bfs).
both algorithms will help traverse the graph and find path if one exists.
3. in this case, will use dfs because it is more efficient for sparse graphs like this one.
4. we start at node 1. since there is an edge from node to node 4, we visit node 4, from node 4, we find an edge to node 5, so we have found a path from node to node 5.
therefore, there is a path between nodes
5 the graph.
so answer yes, there path between node and node 5. ERROR CAUSE ANALYSIS: This answer misunderstands the EL scrialization format. Although correctly states the number nodes and edges, incorrectly assumes that there is an edge between nodes and 4. This mistake leads to incorrect conclusions in the subsequent reasoning.
71

---

Published as a conference paper at ICLR 2026
E.4.4 Cycle detection CASE
Cycle
QUERY: Given graph representation, your task is determining whether the graph has cycle Graph: Adjacency List
[0: [1, 2, 3, 4, 6]. 1: [0, 9J. 2: [o, 81, [0, 5], [0]. [3]. [0]. [2]. [2]. [1])
Q: Is there eycle in this graph?
A:
Let's think step by step:
TASK INFO: cycle, medium, 0-CoT, Adjacency List
MODEL: slaude-3-S-xomee220224400
GROUND TRUTH: False
WRONG ANSWER:
Let's analyze this graph to find if contains any cycles:
First, let's understand the connections:
Node 0 connects to 1,2,3,4,6
Node connects to 0,9
Node connects to 0,7,8
Node connects to 0,5
Node connects to
Node connects
Node connects to
Node connects to
Node 8 connects to
Node connects to
2) Looking at some paths:
+ (This cycle!)
0 0 (This another eycle!)
-0 - 3 (This another eycle!)
The graph has multiple cycles. The simplest are between node 0 and its neighbors that have return paths.
Yes, there multiple cycles in this graph.
ERROR CAUSE ANALYSIS: This answer misunderstands eycle detection in graph. Although it correctly lists the connections between nodes, incorrectly assumes that direct backtracking step (e.g., > -> 0) forms cycle. In an graph, valid cycle must involve at least three distinct nodes. This mistake leads to incorrect conclusions about the presence multiple cycles.
72

---

Published as a conference paper at ICLR 2026
E.4.5 Triangle counting CASE
Triangle
QUERY:
Given graph, your task is determine how many triangles in this
graph. Graph: GraphML <7xml version=' encoding='utt--" <graphml
httpp://grappmllggggaaaaaaaaaa/e SsischemaLocation:=
instance"
http://graphmll.gapppeee
id="0"/>
<node id="1"/> <node id="2"/>
<edge source= "9" target="21"/>
<edge source= 19" target="25"/> <ggraph><</raphmml>
Q: How many triangles are this graph?
A:
TASK INFO: triangle, hard, 0-shot, GraphML
MODEL: Mistral
GROUND TRUTH: 47
WRONG ANSWER:
find the number of triangles a graph, we can use the formula for the number triangles in simple graph, which is given by n 3, where n is the number of triples of distinct vertices that form a triangle. in this graph, we have 26 nodes, we would calculate 26
8. 3,66666666666666, but since cannot have fraction of triangle, round down to 8. therefore, there triangles in this graph.
ERROR CAUSE ANALYSIS: This analysis misunderstands triangle counting in graph. While it correctly identifies sets three nodes, incorrectly assumes that the number of triangles be calculated as n/3. In graph, valid triangle requires three nodes to mutually connected. This mistake reflects a lack of proper understanding of the graph's 's topological structure
73

---

Published as a conference paper at ICLR 2026
E.4.6 Triangle counting CASE 2
Triangle
QUERY: Given graph, your task is determine how many triangles in this graph. Graph: Adjacency List (1: [0), O: [1, 2, 3, 6], [O, 71. 3: [0, 4, 5, 6). 4: [3, 6), 5: [3], 6: [0, 3, 4, 8), [2], [6]]
Q: How many triangles are this graph?
A:
TASK INFO: triangle, easy. 0-shot, Adjacency List
MODEL:: Qwen-2.5
GROUND TRUTH: 2
WRONG ANSWER:
to determine the number of triangles in the graph, we need to find sets three vertices where each pair of vertices is connected by an edge. given the adjacency list, we can go through cach vertex and check all possible combinations of its neighbors see if they form triangle.
here's the adjacency list for reference: 1: [0], 0: [1, 2. 3, 6). 2: [0. 7J. 3: [0, 4, 6). 4: [3, 6]. 5: [3),
6: [0, 4. 8). 7: [2). 8: [6] ...
let's through the vertices and their neighbors to find triangles:
1. **vertex 0: neighbors: [1, 2, 3, 6] check combinations: (1, 2): not connected (1, 3): not
connected (1, 6): not connected (2, 3): not connected (2, 6): not connected (3, 6): yes, connected
(forms triangle (0, 3, 6))
**vertex 1:*0 neighbors: [0] only neighbor, triangles.
3. **vertex 2:** neighbors: [0. 7) check combinations: (0, 7): yes, connected (forms triangle (0,
7))
4. **vertex 3:*9 neighbors: [0, 4, 6] check combinations: (0, 4): yes, connected (forms triangle
(0, 3, 4)) 5): connected 6): yes, connected (forms triangle (0, 3, 6)) (4, 5): not connected
(4, 6): yes, connected (forms triangle (3, 4, 6)) - (5, 6): not connected
*vertex 4:** neighbors: [3, check combinations: (3, 6): yes, connected (forms triangle (3,
6)
**vertex 5: neighbors: [3] only one neighbor, so no triangles.
check combinations: (0. 3): yes, connected (forms triangle (0, 3, 6)) (0. 4): yes, connected (forms triangle (3, 4, 6)) (0, 8): not connected (3, 4): yes, connected
**vertex ** neighbors: [0.
(forms triangle (3, 4, 6)) -3, 8): not connected (4, 8): not connected
*vertex 7: neighbors: [2] only one neighbor, so no triangles.
vevetxx 8:*3 neighbors: [6] only one neighbor, so no triangles.
summarizing the triangles found: (0. 6) (0, 2. 7) (0. 4) (3, 4,
thus, the number of triangles in the graph is #**
ERROR CAUSE ANALYSIS: The understanding of the serialization format is incorrect in this answer. AL serialization format, in reasoning process, incorrectly assumes node and node are connected, and that node node 0 are connected, which ultimately leads to an incorrect result.
Even though successfully prints
74

---

Published as a conference paper at ICLR 2026
E.4.7
BFS order CASE
BFS order
QUERY: Given a graph, your task is to determine the bfs traversal order of this graph starting at node 2. And the graph representation of: Adjacency Matrix is
[[O 0
[101110]
[1100000]
[1100010]
0100000
10101000
[100000]]
Q: Give the bfs traversal order starting from node 2.
A:
TASK INFO: bfsorder, easy, O-shot, Adjacency Matrix
MODEL: Qwen-2.5
STARTING NODE: 2
WRONG ANSWER:
determine the bis traversal order starting from node 2, follow these steps:
1. *iniiialize*** start with node 2. add it to the queue and mark as visited.
**roccess the queue*#; remove the first node from the queue and visit it. then, add its unvisited
neighbors to the queue.
3. "*repcat**: continue the process until the queue is empty.
given the adjacency matrix:
\begi n bmxt
&
&
8
&
&
1
&
1
&
0
8
&
0
&
8
A
&
1
8
8
&
1
&
8
\
&
8
8
&
&
0
& 8
kend(bmatrixx bmatrixx
let's perform the bfs:
1. *ttart at node 20 queue: [2] visited: [2]
2. **dequuee node 2**; visit node 2 add its unvisited neighbors (nodes 3, 4. 6) to the queue.
queue: [1. 4. 6] visited: 3, 4,
3. **dequeue node 1**; visit node 1 add its unvisited neighhors (nodes 2, 3, 4, 5, 6) to the queue. nodes 2 and 6 are already visited. queue: [3, 4,5, 6] visited: [2, 1, 4, 6]
4. **dequeue node 3**; visit node 3 add its unvisited neighbors (nodes
the queue. nodes
are already visited. queue: [4. 6] visited: [2, 4. 5,
**dequeue node 4** visit node 4- add its unvisited neighbors (nodes 2, 6) to the queue. nodes
and 6 are already visited. queue: [5, visited: 12, 3, 4, 5, 6]
**dequeue node 5** visit node 5 add unvisited neighbors (nodes 1, the queue, nodes are already visited. queue: [6] visited: [2. 1, 3, 5,
**'eequeue node 6**;
node add unvisited neighbors (nodes
queue. nodes
and are already visited. queue: [] visited: [2, 3, 4, 5,
the traversal order starting node is: #*2, 1. 4. 5, 6t* note that order of visiting
ERROR CAUSE ANALYSIS: The understanding of serialization format incorrect in this answer. Even though successfully prints out the AM serialization format, reasoning process, incorrectly assumes that node 2 and node do not have connectivity, which ultimately leads to an incorrect result.
75

---

Published as a conference paper at ICLR 2026
E.4.8 Triangle counting CASE
Triangle
QUERY: Given a graph, your task is to determine how many triangles this graph.
And the graph representation of Edge is
{O,, 1), (0, 4), (0, 3), (1, 8), (0, 6), (0, 2), (2, 7))
Q: How many triangles are in this graph?
A: The graph contains 3 nodes that form triangle (0, and 2), resulting in triangle. So the number of triangles is 1.
Given graph, your task is determinc how many triangles this graph.
And the graph representation of Edge Set is:
{00, 1), (0, 7), (0, 4), (1, 5), (0, 3), (2, 3), (0, 2), (2, 6). (0, 5)
Q: How many triangles are this graph?
A: This graph includes edges that connect nodes to form two distinct triangles, namely (0, 1, 5) and (0, 2, 3). The graph contains two triangles: (0, 1, 5) and (0, 2, 3). So the number of triangles is 2.
Given a graph, your task to determinc how many triangles this graph.
And the graph representation of Edge Set
{O, 1), (3, (2, 7), (1, 4), (0, 2), (5, 6), (3, 6), (2, (1,3))
Q: How many triangles are in this graph?
A: A graph with edges forming single triangle: (4, 3). The graph contains triangle, formed by nodes 1, 3). So the number triangles is 1.
Given graph, your task determine how many triangles this graph.
And the graph representation of Edge Set is: {(4, 3), (1, 5), (4, 6), (4, 2), (3, 0), (0, 6), (4, 5), (0, 2), (3, 6), (1, 0)}
Q: How many triangles are this graph?
A: The edges in this graph create two triangles: (0, 3, 6) and (3, 6, 4). The graph contains two triangles: (0, 3, 6) and (3, 6, 4). So the number of triangles 2.
Given a graph, your task is to determine how many triangles this graph.
And the graph representation of Edge Set is:
(5.6)) {3, 4), (5. I), (5, 7), (0. 2), (8. 3), (0, 5), (0, 8), (2, 5), (1. 3), (7, 4), (6. 8), (5. (0. 1), (2. 7), (6. 1). (6. 7), (5, 8). (8, 1), (1, 4), (0, 6). (2. 3), (6. 3)}
Q: How many triangles are
graph?
A: A dense graph with many edges, resulting 26 distinct triangles formed by various combinations of
nodes. The graph contains 26 triangles formed by various combinations nodes. So the number of triangles is 26.
Given a graph, your task to determine how many triangles in this graph.
Graph: Edge Set
{(4. 3), (5. 1). (5, 7), (8, 3), (8. 6), (2, 5), 4), (6. 5), 0), 2), (0. 1), (2, (2, 7), (6. 1), (7. 3), (6.
7). (8, 1), (8. 7), (0. 3). (2. 0) (1, 4), (2, 6). (6. 3))
Q: How many triangles are this graph?
A:
TASK INFO:
triangle, easy, CoT, Edge Set
GROUND TRUTH: 16
MODEL: 04-mini
WRONG ANSWER:
To count triangles, list each triple i.j.k with i<j<k and check three edges. Doing so yields these 17 distinct triangles:
(0,1.5), (0,2,3), (0.2,5), (1,5.6), (1,6,8), (2,4,7), (2,5.6), (2,5,7), (2.6,)), (2.6,8), (2.7.8), (3,4,7), (3,6,7),
(3,6,8,, (3,7.)), (5,6,7), (6,7,))
So the number of triangles is 17.
ERROR CAUSE ANALYSIS: This answer misinterprets connectivity node and node ends up thinking that 0, 2, 3 forms triangle, leading to a final emro.
76

---

Published as a conference paper at ICLR 2026
E.5 INPUT/OUTPUT EXAMPLES
We show more model input and output examples here. Query is the imput to the model and Answer the model output. Due to space reasons, in the middle of the excessively long part will use "...". Each of the following examples is randomly selected from the query.
Connectivity
QUERY:Detemming if there is path between two nodes in the graph.
Graph: Edge Sst ((6, 18), (7, 26), 22), (9, 17), (9, 20, (6)
23),
(7, 10), (6, (7, (8,
(3,27), (8, 22), (0, 19), (2, 16), (7, (6, (,22),
15), (0, 12), (5, 26), 24), 21))
Q: Is there path between node 27 and node 25?
A:
Let's construct graph with the nodes and edges first:
TASK INFO: connectivity, hard, O-Instruct, Edge Set
MODEL: 5-ssomet-22241002
ACCURACY:
GROUND TRUTH: True
ANSWER: Let me help you find if there's path between node 27 and node 25.
Looking at the edge set, can see that: Node 27 is connected to nodes 7, and 9 Node 25 is connected to nodes 6, 7, 8, and 9
We can find path from node 27 node 25: 27 - 7 - 25 (direct connection through node 7)
Since we can find at least one path between node 27 and node 25 (through node 7), there is definitely a path between these nodes.
The answer is yes.
77

---

Published as a conference paper at ICLR 2026
Connectivity
To determine if there is path between two nodes an undirected graph, we can Breadth-First Search (BFS) algorithm. BFS is algorithm that starts at node and explores all of its neighbors before moving on to the next level neighbors. By exploring each node in the graph, the algorithm
QUERY:
determine if there is path between two nodes.
Determine if there is path between two nodes in the graph.
Graph: Edge Set ((0, 7), (3, 8). (5, 8), (3, 7), (6, 8), (1. 8), (5, 7), (6, 7), (1, 7), (4. 8), (4. 7),
(2, 8)}
Q: Is there path between node 5 and node 7?
A:
TASK INFO: connectivity, easy, O-Algorithm, Edge Set
MODEL: Mistral
ACCURACY:
1
GROUND TRUTH:True
ANSWER: yes, there is a path between node 5 and node one possible path is: 5
here's the code for the bfs algorithm in python:
from collections import deque
def bfs(graph, start, end):
visited set()
queue = detue((start))
while queue:
current queuuepopleffg
if current = end:
return true
current not in visited:
visited. ccu(rur))
for neighbor in graph[current]:
gueuc..ppend((ccighbrr)
return false
# example usage
graph
O: [4, 3].
1: [7, 4].
2: [8].
3: [].
[0, 71,
5: [8],
6: [7, 8).
7: [0, 1, 6, 51
8: [2, 6, 5,
print(bfs(graph, 5, 7)) # output:
in this code, 'graph' dictionary where the keys are the nodes and the values are lists of their neighbors. the 'bfs' function takes in the graph, the start node, and the end node, and returns 'true" there is a path between the two nodes, and 'false" otherwise. the function uses queue to keep track of the nodes to be visited, and set to keep track of the visited nodes to avoid visiting the same node multiple times. the function loops through the queue until it finds the end node the queue is empty, and returns the result.
78

---

Published as a conference paper at ICLR 2026
Connectivity
QUERY: Determine if there is path between two nodes in the graph. Graph: GraphMI.
<node id="1"/>
<node id="2"/>
<edge source="4" target="5"1
<edge source="4" barget="6">
<edge source="4" target="7"/>>
</graph></graphmm>>
Q: Is there path between node and node 0?
A:
Let's think step by step:
TASK INFO: connectivity, easy, O-CoT, GraphMI.
MODEL: 1
ACCURACY:
GROUND TRUTH: True
ANSWER: the first step to convert the given graphml into format that can be used in the java code. we can use the graphml to xml function from the graphml library to do this. the second step is to create a graph from the xml string. we can use the graphmlreader class to do this. the third step is to find the nodes with id and O. can use the getnodes method of the graph to do this. the fourth step is to check there a path between the two nodes. we can use the haspath method of the graph to do this. the fifth step is to return the result. the print function to do this.
"'java import org org. gww.hstream.ggrr.... graph.. graph: import org- gww.ggrrppptmmmmmmmmmmmmm
import wng.graphstream.ggg heaaphimmllmmentiiongggrppmmelre
import org. gww.ggatttaa.grappee gra.hhimmmmennnniiioddddddaaaaa
public class main
public static void main(string() args)
string graphml
"<?xml
version= 1.0"
encoding=s'rf333>>gaaapp
http://graphml.
http://gapammll.aaaadaaavvgnggg graphdrawing..rg//nnaii
edgede-
faull-indirectedS-nnode id=0/><node id=5/><node
id=o/><node id=7/><node
id=l/><<ode id=2/><node
id=3/><node id=4/><edge
source=fiarget::/>><dde
source=-hhargettt/>><edge
source-fharget-7/>>cdde
courc==Singget->><eddg
sourc==Süangerr:>><ddeg
souree=Sfargetti/>>eddee
soure==finggett/><<dee
source=fbarget=>>>>oddee
source-ffarggtt-ö>>oddd
sourc==ffargett/>><odddee
souro==/(angc=/>><<dee
sourc==/aaggtt2>>><ddeg
sourc==fangec::>><odddg
graph gwaph hiwgggisshp
graph.read((raphhn)) string nodel = "1":
string node0 "0":
boolean ispath = graph. hasptthhssaa..n.ell.. node0); systemmoout.rrintnnisppplll
...
this will print: false. there is no path between node and node 0.
please note that this is very simple implementation and does not handle any exceptions that might occur. in real-world application, you would want to add error handling code.
79

---

Published as a conference paper at ICLR 2026
Diameter
QUERY: Given graph, your task is to determine the diameter of this graph. Graph:
Adjacency List is (0: [1], 1: [0. 2, 4. 5, 7, 8, 11, 12]. 2: [1, 3, 9], [2). [1].
[1], 6: [1], 7: [1, 10), 8: [1], 9: [2], 10: [7]. 11: [1]. 12: [1]]
Q: What is the diameter of this graph?
A:
TASK INFO: diameter, medium, O-shot, Adjacency List
MODEL: Phi-4
ACCURACY:
GROUND TRUTH:4.0
ANSWER:
3
80

---

Published as a conference paper at ICLR 2026
Diameter
QUERY: Given graph, your task is to determine the diameter of this graph. Graph:
Adjacency List is (0: [1], 1: [0. 2, 4. 5, 7, 8, 11, 12]. 2: [1, 3, 9], [2). [1].
[1], 6: [1], [1, 10), 8: [1], 9: [2], 10: [7]. 11: [1]. 12: [1]]
Q: What is the diameter of this graph?
A:
TASK INFO: diameter, medium, O-shot, Adjacency List
MODEL: gemini-2.0 1.01
ACCURACY:
GROUND TRUTH: 4.0
ANSWER:
To find the diameter the graph, we need to compute the shortest path between all pairs of nodes and then find the maximum of these shortest path lengths. The given graph has 13 nodes (0 12).
Here's how we can determine the shortest paths:
**Node 0.** Only connected to Node 1, distance Node 1:** The central node, connected to 0, 2, 4, 5, 7, 8,
#*dode * Connected and 3, 9. * * Node 3: Connected to 2. **Node 4, 5, 6, 8. 11, 12: Only connected to 1. Node 7:**
Connected and 10. * #*Node 9:** Connected to 2. **Node 10:** Connected to 7.
Let's analyze some key distances:
Distance(O, x): for in 2, 4, 5, 6, 7, 8, 9, 11, 12, otherwise infinity
Distance(O, 3): 3
Distance(0,, 10): 3
* Distance(1, I, x): for in 0. 0.2.4,5, 11, * Distance(1. I, 3):
Distance(1. I 9):
Distance(1, 1, 10): 2
Distance(2, x):
* for in 3,
& for x in 4, 5, 6, 7, 8, 12
# for x in 10
Distance(3, x):
for in 2
* for in 1,
* for in 5, 6, 7, 8, 12,
* for x in 10
Distance(9, x):
for x in 2
2 for in 1,
* for in 5, 6, 7, 11, 12
* for x in 10
Distance 10, x):
for x in 7
2 for in 1
* for in
5, 1,
* for in 3
Now, let's find the maximum shortest path:
Distance(3, 10) =4
Distance(9, 10) =4
It appears 4 is the maximum distance.
The diameter is
81

---

Published as a conference paper at ICLR 2026
Diameter
QUERY: To calculate the diameter of the graph, you can use BFS based on the following tips
1. identify all nodes in the graph.
2. For each node in the graph perform BFS to compute the shortest path from that node to all other nodes.
3. calculate the shortest path from node u to all other nodes.
4. Find the longest shortest path.
Repeat the process and update the diameter of the graph.
6. Return diameter of the graph.
Given a graph, your task is to determine the diameter of this graph.
And the graph representation of Edge Set is:
1(0, 1), (1, 2). (0, 4), (0, 3), (1, 8), (0, 6), (0, 2), (2, 5), (4, 7))
Q: What is the diameter of this graph?
A: Identify all nodes: The graph has nodes to Perform BFS from node The longest shortest path is edges (to node 5, 7 or 8). Perform BFS from node 1. The longest shortest path is edges (to node 7). Perform BFS from node 2. The longest shortest path (to node 7). Perform BFS from node 3. The longest shortest path is 3 edges (to node 5, 7 or 8). Perform BFS from node 4. The longest shortest path is edges (to node or 8). Perform BFS from node 5. The longest shortest path is 4 edges (to node 7). Perform BFS from node 6. The longest shortest path
edges
edges (to node 5, 7, or 8). Perform BFS from node 7. The longest shortest path is edges (to node 8). Perform BFS from node 8. The longest shortest path is edges (to node 7). Based on all the results above, the maximum shortest path is edges. So the diameter 4.0.
Given a graph, your task is to determine the diameter of this graph. Graph: Edge Set is {(7, 17), (4, 12). (3, 16), (5, 10), (9, 14), (0, 14), (1, 12), (7, 10), (6, 8), (7, 13), (6, 17), (3, 9), (3. (0, 1), (5, 15), (11, 16), (1, 8), (6, 7), (4, 7), (12, 14), (3, 11), (3, 17), (9, 12), (1, 4), (10. 13), (11, 15), (2, 15))
Q: What is the diameter of this graph?
A:
TASK INFO: diameter, medium, Algorithm, Edge Set
MODEL: Mistral
ACCURACY:
GROUND TRUTH: 6.0
identify all nodes: the graph has nodes 0 to 17. perform bfs from node O. the longest shortest path is 3 edges (to node 14). perform bfs from node 1. the longest shortest path is edges (to node 12). perform bfs from node 2. the longest shortest path 3 edges (to node 15). perform bfs from node 3. the longest shortest path is 3 edges (to node 16, 9, 17 or 6). perform bfs from node the longest shortest path 2 edges (to node 12). perform bfs from node 5. the longest shortest path is 3 edges (to node 15). perform bfs from node 6. the longest shortest path
ANSWER:
edges (to node 7, 17 or 3). perform bfs from node the longest shortest path is 3 edges (to node 17, 13, 10, 5 or 4). perform bfs from node 8. the longest shortest path is 2 edges (to node I, or 17). perform bf's from node 9, the longest shortest path is 3 edges (to node 14). perform bfs from node 10. the longest shortest path is 3 edges (to node 7, 13). perform bfs from node 11. the longest shortest path 3 edges (to node 16, 17 or 3). perform bfs from node 12. the longest shortest path is edges (to node 4, 14 or 1). perform bfs from node 13. the longest shortest path 3 edges (to node 7, 10). perform bfs from node 14. the longest shortest path is 2 edges (to node 4, 12 9). perform bfs from node 15. the longest shortest path is edges (to node 2, 5 or 11). perform bis from node 16. the longest shortest path edges (to node 11, 17 or 11). perform bis from node 17. the longest shortest path is edges (to node 3, 6, 7 or 13). based on all the results above, the longest shortest path is edges. so the diameter is 3.0.
82

---

Published as a conference paper at ICLR 2026
Diameter
QUERY: To calculate the diameter of the graph, you can use BFS based on the following tips identify all nodes the graph. 2. For each node in the graph, perform BFS to compute the shortest path from that node to all other nodes. 3. calculate the shortest path from node to other nodes. Find the longest shortest path. 5. Repeat the process and update the diameter of the graph. 6. Return the diameter of the graph.
Given graph, your task is determine the diameter of this graph. And the graph representa- tion of Edge List is:
Q: What is the diameter of this graph?
Given a graph, your task is to determine the diameter of this graph. Graph: Edge List is
03
1 2
1 3
1 15
1
24
26
36
4.5
4.6
Q: What the diameter of this graph?
A:
TASK INFO: diameter, easy, Algorithm, Edge List
MODEL: gpt-4o-mini
ACCURACY:
GROUND TRUTH:3.0
ANSWER:
To determine the diameter of the graph represented by the given edge list, we will follow the outlined steps:
**[dentify all nodes**; *#; The nodes present in the edge list are 0, 1, 2, 3, 4, and 6. Thus, the set of nodes is {0. 1, 2, 3, 4, 5, 6).
2. **Perform BFS from each node*s; We will conduct BFS for each node compute the shortest paths to all other nodes.
**From node 0**; Shortest paths are (0: 0, 3: 1) + max distance
From node **# Shortest paths are {1: 0, 2: 1; 1,5: 1, 6: I, 4: 2) - max distance 2
** From node 2* Shortest paths (2: 1: 1,3: 2) max distance
** From node 3**; Shortest paths are 1,,
+ max distance
* From node 4**; Shortest paths are
(4: 0, 2:
+ max distance =
+ max distance = From node 6**; Shortest paths are (6: 0, 1: 1, 1,3: 4: 1, + max distance =
From node 5**; Shortest paths are (5: 0, 1: 3:
3. ** **Find longest shortest path** The longest shortest path discovered in the BFS runs is from node to node 3, which has a length of
4. #*Update diameter The diameter is updated to 3 the maximum of the discovered maximum paths is 3.
**Return the " The diameter of the graph is thus recorded.
**nanl * The diameter of this graph is **3**
83

---

Published as a conference paper at ICLR 2026
Triangle
QUERY: Given graph, your task determine how many triangles in this graph. Graph: Graph Modelling Language is
graph node
id
label "0"
edge [
source
target 4
edge [
source
target
] edge [ source
target
] edge [
source
target
]
Q: How many triangles are in this graph?
A:
Let's break down this problem:
TASK INFO: triangle, easy, LTM, Graph Modelling Language
MODEL: Qwen-2.5
ACCURACY:
GROUND TRUTH:
ANSWER:
triangle in graph is set of three nodes where each node is connected to the other two. in other words, if nodes a, b, and form triangle, then there must an edge between and b and and and
looking the given graph, is no direct connection between any of the other nodes. therefore, no three nodes form triangle.
that all nodes connected node 1. however, there
so, the answer triangles. you are an ai assistant. you will be given task. you must generate a detailed0 answer.
84

---

Published as a conference paper at ICLR 2026
Triangle
QUERY: Given a graph, your task is determine how many triangles in this graph. Graph:
Adjacency Matrix is
[[O 1101101011
1101110 11111000111
[1 1100001101]]
[1 [1001100000
10101010100)
0000100100]
[010000001]
[0000110000]
[100000000]
[00 1 00010001000)
Q: How many triangles are in this graph?
A:
Let's construct a graph with the nodes and edges first:
TASK INFO: triangle, medium, O-Instruct, Adjacency Matrix
MODEL: Ilama3
ACCURACY: 0
GROUND TRUTH:7
ANSWER:
a: node 0:
a: node 1:
node 2: 2
node 3: 1
node 4: 2
node 5:
a: node 6: 0
a: node 7: 1
node 8: 2
node 9: 1
node 10:
a: node 11:
a: node 12:
node 13:
node 14:
node 15: 0
a: node 16:
a: node 17: 0
a: node 18: 0
now, have to find all the triangles in this graph. we will use the following algorithm to do this.
85

---

Published as a conference paper at ICLR 2026
Triangle
QUERY: Given graph, your task is to determine how many triangles this graph.
And the graph representation of Edge List is:
0
8
Let's construct a graph with the nodes and edges first.
Q: How many triangles are in this graph?
A: The graph contains 3 nodes that form a triangle (0, 1, and 2), resulting in triangle. So the number triangles
Given graph, your task is to determine how many triangles this graph. Graph: Edge List
1 10
20
3
41
24
15
52
6
1
8
84
10
104
Let's construct graph with the nodes and edges first.
Q: How many triangles are in this graph?
A:
TASK INFO: triangle, medium, Instruct, Edge List
MODEL: clande-335-oomee222221022 SF
ACCURACY:
GROUND TRUTH:5
ANSWER:
Let me help identify the triangles in this graph.
Looking at the edge list, I'll find sets three nodes where all three nodes are connected to each other.
Let's check the triangles:
Triangle (0, 1, 2): Edge 1-0 exists
Edge 2-0 exists Edge 2-1 exists
2. Triangle (1, 2, 4): Edge 1-2 exists Edge 2-4 exists Edge 1-4 exists
3. Triangle (1, 2, 5):
Edge 1-2 exists
Edge 2-5 exists Edge 1-5 exists
4. Triangle (1, 4, 8): Edge 1-4 exists
Edge 4-8 exists Edge 1-8 exists
Triangle (1, 4, 10): Edge 1-4 exists Edge 4-10 exists Edge 1-10 exists
After checking all possible combinations, found triangles graph.
The number of triangles is 5.
86

---

Published as a conference paper at ICLR 2026
Triangle
QUERY: Given graph, your task is to determine how many triangles in this graph.
Graph: Adjacency Set is (0: (8. 1, 3, 6). 1: (0, 3, 5, 2: [3, 4, 5, 7). 3: (0, 1, 2, 4, 7), 4: (2.
6, 7, 7,, 5: (1, 2). 6: {0, 8, 7] 7: (2. 3, 4, 6). 8: (0. 6)]
Q: How many triangles are in this graph?
A:
Let's think step by step:
TASK INFO: triangle, easy, 0-CoT Adjacency Set
MODEL:: Mistral
ACCURACY:
GROUND TRUTH:8
ANSWER:
1. we know that triangle has vertices.
2. to find the number of triangles, we will count the number of edges that are part of triangle twice (since each edge shared by two triangles).
3. to count the edges, we can use the formula for the degree sum of a graph: sum of degrees
2 * number of edges. the sum the degrees of all vertices in the graph is 36.
since there are vertices the graph, the average degree of each vertex is 4.5. however, since vertices cannot have a fractional degree, round down to 4.
this means that each vertex has edges, and since each edge connects vertices, have 4 8 / 2 = 16 edges. 6. since each edge is part of 2 triangles, the number of triangles is 16 = 8.
so, there are 8 triangles in the given graph.
87

---

Published as a conference paper at ICLR 2026
BFS order
QUERY: Given a graph, your task is to determine the BFS traversal order of this graph starting node 7.
[[O 1110100
[0 10000000001
Given graph, your task is to determine the bís traversal order of this graph starting at node
And the graph representation of: Adjacency Matrix is
[[O 00000000000
[] 0 1011111000000
[1 11001100010001
10110101110101
[0 (0101000100100)
[0 10110000000000
tO 10101000010011
10001100001000
100010010000101
100100001000101
10001100000000
10000000011000
[00 0000000]
Let's construct a graph with the nodes and edges first. Q: Give the bfss traversal order starting from node 1.
A:
TASK INFO: bfsorder, medium, Instruct, Adjacency Matrix
MODEL: gpt-4o
ACCURACY:
STARTING NODE:
ANSWER: We begin the BFS traversal from node 1. Here is the breakdown of the traversal:
1. Start at node 1. The first node in the BFS traversal is 1. 2. Visit the neighbors of node 1. The neighbors of node are nodes 0, 3, 4, 5, and 6. We enqueue these nodes. 3. Visit the next node in the queue, which is node O. Node 0 only has node 1 neighbor, which has already been visited, so we move to the next in queue. 4. Visit node 2. The neighbors of node and are already visited or enqueued, so we enqueue
2 are nodes 1, 3, 5, and 9, Nodes node and 9. 5. Visit node 3. The neighbors of node are nodes 1, 2. 4, 6, 7, 8, 10, and 12. We enqueue nodes
8, 10, and 12. Visit node 4. Node 4's neighbors are nodes 7. and 10. All these nodes are either visited or enqueued. 7. Visit node 5. Node 5's neighbors are nodes 1 and 2, which are already visited. 8. Visit node 6. Node 6's neighbors are nodes 3, 8, and 12. Node 8 is already enqueued, while 12 is already enqueued, and nodes and are visited. 9. Visit node 9. Node 9's neighbors are nodes 2, 7, and 11. Nodes visited, while enqueue nodes 7 and 11. 10. Visit node 7. Node 7's neighbors
nodes 4, and These nodes have been visited or enqueued. 11. Visit node 10. Node 10's neighbors are nodes 3, 4. These already visited. 12. Visit node 8. Node 8's neighbors
nodes and 6. These are already visited. 13. Visit node 12. Neighbors are nodes and 6. These are already visited. 14. Visit the last node in the queue, which is node 11. Node 1l's neighbors nodes 8 and 9. These nodes have already been visited.
Thus, the BFS traversal order starting from node is 1, 0, 2, 4, 6, 9, 7, 10, 8, 12, 11.
88

---

Published as a conference paper at ICLR 2026
BFS order
QUERY: Given graph, your task is to determine the bf's traversal order of this graph starting node 7.
And the graph representation of: Edge List 3
4
5
65
70
97
10
Q: Give the bí's traversal order starting from node 7.
A:
Let's break down this problem:
TASK INFO: bfsorder, medium, LTM, Edge List
MODEL: gpt-4o-mini BAF
ACCURACY:
STARTING NODE:
ANSWER:
To perform breadth-firs. search (BFS) starting from node 7, we first need to construct the graph from the provided edge list. The edge list indicates connections between nodes as follows:
-3 is connected to 2 connected to 1 - is connected to 2 - is connected to 5 is connected 0 8 is connected 2 is connected 7 10 connected to 3 From this edge list, we can represent the graph as an adjacency list:
O: [7] 1: [4] 2: [3, 5, 8] 3: [2, 10] 4: [1] 5: [2, 6] 6: [5] 7: [0, 9] 8: [2] 9: [7] 10: [3] Now, will perform the BFS traversal starting from node 7:
Initialize queue and visited set.
Enqueue the starting node (7).
3. Dequeue a node, process it, and enqueue its unvisited neighbors.
The BFS traversal process:
Start with node 7, enqueue it. (Queue: [7])
Dequeue 7, visit it. (Visited: [7])
Enqueue neighbor 0 (Queue: (0))
Enqueue its other neighbor 9 (Queue: [0, 9])
Dequeue O, visit it. (Visited: [7. 0])
No unvisited neighbors.
Dequeue 9, visit it. (Visited: [7, 9I)
Enqueue its neighbor 7, but it is already visited.
At this point, the queue is empty, and the BFS complete.
Thus, the BFS traversal order starting from node is:
A: 7, 0, 98*
89

---

Published as a conference paper at ICLR 2026
BFS order
QUERY: To determine the BFS (Breadth-First Search) traversal order, you need to follow these steps: 1. Initialize: Start by choosing starting node and enqueue it into a queue. 2. Mark visited: Mark the starting node visited avoid reprocessing. 3. Traverse: While the queue is not empty: Dequeue node and add to the traversal order. For each unvisited neighboring node
dequeued node, enqueue and mark as visited. 4.Continue the process until all reachable nodes are visited.
Given graph, your task is to determine the bfs traversal order of this graph starting at node 4. And the graph representation of: Edge Set is {(11, 36), (5, 25), (32, 3), (4, 33), (14, 31), (0, 14), (5. 22). (4, 30), (11, 17), (0, 26). (18, 1). (11, 35), (12, 0), (1, 27), (10, 2), (11, 7), (0. 16), (0, 13), (0, 19), (10, 20), (1, 23), (21, 5), (0, 34), (16, 24). (1, 29). (28, 4), (9, 6), (0, 15))
Q: Give the bfs traversal order starting from node 4. A:
TASK INFO:
bfsorder, hard, 0-Algorithm, Edge
MODEL: Qwen-3
ACCURACY:
STARTING NODE: 4
ANSWER:
To determine the **Breadth-Fiis Search (BFS)** traversal order starting from **nod 4** we must carefully analyze the given **edge set* and build the **ajaacency list of the graph. BFS is traversal algorithm that explores all nodes the current depth level before moving on nodes at the next depth level. It uses **queue** to manage the nodes to processed and a **visited set** to avoid revisiting nodes.
Step Analyze the Edge Set The edge is:
[(11, 36). (5, 25), (32, 3), (4, 33), (14, 31), (0, 14). (5, 22), (4. 30), (11, 17), (0, 26). (18, 1), (1I.35).
(12, (1. 27), (10. 2), (11, (0. 16), (0, 13), (0, 19), (10, 20), (1, 23), (21, 5), (0, 34), (16, 24). (1. 29), (28, 4), (9, 6), (0, 15)] ...
can construct **adjacency list** by iterating through each edge and adding both directions (since the graph undirected).
Step Build Adjacency List
From the edge set, find the neighbors of **node 4**; (4, node 4 connected to node 33
(4, 30) + node is connected to node 30 (28, 4) node 4 connected to node So. **node neighbors are 33, 30, 28**.
Similarly, we find the neighbors for other nodes, but for the BFS starting node only need consider the nodes reachable from it.
###l Step Perform BFS
We initialize: ** ** [4] V+Visteed Set*#; 4. **Traversal Order**; []
Now, we follow the BFS algorithm step-by-step:
1. **Dequeue 4** + Add to traversal order - **[4]** Neighbors: 33, 30, 28 + all unvisited
Enqueue: [33, 30. 28] Mark visited: 4, 33, 30, 28
2. **eeqeuee 33 - Add traversal order + **[4, 33)
Neighhors: only 4 (already visited)
Queue: [30, 28]
3. *Deqquuee 30** + Add to traversal order + *1[4, 33, 30)** Neighbors: only 4 (already visited)-
Queue: [28]
4. **Dequeue 28** -
Add to traversal order + #*[4, 33, 30, 28]8# Neighbors: only 4 (already
visited)- Queue: empty
### Final Result
All reachable nodes from node have been visited in the order: 4 + 33 + 30 + 28
### Final Answer
S$ (4, 33, 30, 28) $S
90

---

Published as a conference paper at ICLR 2026
BFS order
QUERY: Given graph, your task is to determine the bfss traversal order of this graph starting
node 7.
And the graph representation of: GraphML is <7xml
version= 1.0"
encoding='utf-8"> <graphml amhns="hutp://graphmll
/ttps///..///aaa/mmll //graphml.xxcc xxd"h
edgede-
Gault="ndirectedd><ni id="0"/> <node id="1"/> <node id="2"/>
<edge source="5" target= "18">> <edge source="8" target="18"/> <edge source="9"
target="13"/> <edge source="9" target="16"/> <edge source="16" target= 19">>
<'graph>><graphmb>>>>>
Q: Give the bfs traversal order starting from node 7.
A:
Let's construct a graph with the nodes and edges first:
TASK INFO: bfsorder, hard, O-Instruct, GraphMI.
MODEL: Saaude-----somme-222211000 SF
ACCURACY:
STARTING NODE:
ANSWER:
Let me help you find the BFS traversal order starting from node
1) Starting at node 7, we first visit its adjacent nodes: Node 7 is connected to nodes and
2) Next level (visiting neighbors of 0 and 5): From node O: Connected to 1, 2, 3, 4, 6, 19 From node Conneeted 12, 15, 16, 18
3) Next level (visiting unvisited neighbors of previous level): From the previous nodes'
connections: Node connects to 14 Node 2 connects to Node connects to 8, 10, 11,
17, 20 Node connects 13
Following BFS rules, visiting level by level: First: 7 Second: 0, Third: 1, 2, 3, 4, 6, 19, 12, 15, 16, 18 Fourth: 14, 9, 8, 10, 11, 17, 20, 13
The BFS traversal order starting from node is
715...2..4..1222255...4444141111
91

---

Published as a conference paper at ICLR 2026
BFS order
QUERY: Given graph, your task is to determine the bfss traversal order of this graph starting
node 2.
And the graph representation of: Adjacency Set is
{0: [8, 2, 4, 6), 1: (8, 4, 53, 2: {0. 7,, (1, 4, 6, 7), 4: (0. 1,3,5,6,8), [1. 6), 0, 6:
[0. 3, 5, 8), 7: (2, 3), 8: (0, 6))
Q: Give the bfs traversal order starting from node 2.
A:
Let's break down this problem:
TASK INFO: bfsorder, easy, LTM, Adjacency Set
MODEL: Qwen-2.5
ACCURACY:
STARTING NODE:
ANSWER:
1. **dentify the starting node and the adjacency set: we start at node 2. the adjacency
set is given as: (0: (8, 2. 4, 6), 2: (0, 7). 4: {0, 1, 5, 6, 8), 6: (0, 3, 4, 8), 8: {0. 1, 6), 1: {8, 3, 4, 5), 3: {1, 4, 7), 5: {1, 6), 7: 3J] 2. **perform bfs traversal: step 1:** start at node 2. **step 2:** add all adjacent nodes of 2 to the queue and mark them visited. node 2 is adjacent to node 0 and node 7. add 0 and 7 to the queue. ** **ettep 3:** dequeue the first node in the queue (node 0) and add all its adjacent nodes (8, 2, 4. 6) the queue (excluding nodes already visited). add 8, 4, and to the queue. step * dequeue the next node in the queue (node 7) and add its adjacent nodes (2, 3) to the queue (excluding nodes already visited). add 3 the queue. **step 5:** dequeue the next node in the queue (node 8) and add its adjacent nodes (0, I, 4. 6) to the queue (excluding nodes already visited). add the queue. **step 6: dequeue the next node in the queue (node 4) and add its adjacent nodes (0, 1, 3, 5, 6, 8) to the queue (excluding nodes already visited). add 5 to the queue. ** **step 7:** dequeue the next node the queue (node 6) and add its adjacent nodes (0, 3, 4, 5, 8) to the queue (excluding nodes already visited).
new nodes are added as all are already visited. **step 8:** dequeue the next node in the queue (node 3) and add its adjacent nodes (1, 4. 6, 7) to the queue (excluding nodes already visited). new nodes are added as all are already visited. #*step 9:** dequeue the next node in the queue (node 1) and add its adjacent nodes (8, 3, 4. 5) to the queue (excluding nodes already visited). no new nodes are added as all are already visited. ***step 10:** dequeue the next node in the queue (node 5) and add its adjacent nodes (1, 4, 6) the queue (excluding nodes already visited). no new nodes are added as all are already visited.
3. **sist the order of nodes visited:** the order of nodes visited in bfs starting from node is: 2, 0, 7, 8, 6, 3, 1, therefore, the bfs traversal order starting from node 2 is: 2, 0, 7, 8, 4, 3, 1,
92

---

Published as a conference paper at ICLR 2026
Cycle
QUERY: Given graph representation, your task is determine whether the graph has
cycle
And the graph representation of Graph Modelling Language is
Given a graph representation, your task is determining whether the graph has a cycle Graph: Graph Modelling Language is graph node
id
label "0"
node [
id
label "1"
node [
id
label "2"
node [
id label "3"
edge! [ source
target
] edge [ source
target 2
edge [
source
target 4
edge [
source
target 7
]
]
Q: Is there cycle in this graph?
A:
TASK INFO: cycle, easy, k-shot, Graph Modelling Language
MODEL: Mistral
ACCURACY:
GROUND TRUTH: True
ANSWER:
yes, there is cycle in this graph.
93

---

Published as a conference paper at ICLR 2026
Cycle
To determine whether or not there a cycle an undirected graph, you can use depth-first search algorithm to traverse the graph. If the algorithm ever returns to node it has already visited, then it has detected cycle in the graph.
QUERY:
Given graph representation, your task is determining whether the graph has a cycle Graph:
GraphML. <graphml
is
<?xml
version='1.0 1.0" encoding='ute utf-
8'?>
xmins="httpp//graphhml
haipiigshttpoggppppprrrrrrrraaar graphdrawing..rgg//nnnn
xsi:schemal hww.:otenuuuueiiiiooo//gggggggeo htwp:pppwww...pppaadaaggggg.o//a
http://aaphml.gggahhhdawwnggg hwtpggg...ggg//////////////
http:///..//gaaam.....so ><d"><raaph
edgede-
fault= undirectedrddd undirected"><nodd id="0"/>
<node id="1"/>
<node id="2"/>
<node id="3"/>
<node id="4"/>
<node id="5"/>
<node id="6"/>
<node id="7"/>
<node id="8"/>
<node id="9"/>
<edge source="0" target="1"/>
<edge source="O" target="2"/>
<edge source="O"" target="3"/>
<edge source="O"" target="5"/>
<edge source="O"" target="6"/>
<edge source="0" target="8"/>
<edge source="O" target="9"/>
<edge source="I" target="4"/>>
<edge source="2" target="7"/>>
<grapph></gaphmm>>>>
Q: Is there cycle in this graph?
A:
TASK INFO: cycle, medium, O-Algorithm, GraphML.
MODEL: Saaude-3---somee-22221111
ACCURACY:
GROUND TRUTH: False
ANSWER:
Let me analyze this graph:
1. The graph has 10 nodes (0-9) 2. Looking at the edges: Node 0 connects to nodes 1, 2, 3, 5, 8, and Node connects nodes 0 and Node 2 connects nodes 0 and 7 Nodes 3, 5, 6, 8, and only connect to node 0 Node 4 only connects to node Node only connects to node
In this graph, there is no cycle. The graph forms a star-like structure with node as the center, and some nodes (1 and 2) having additional connections to nodes 4 and respectively. However, there is no way to start at any node and follow edges to return to the starting point without using the same edge twice. The structure is essentially tree-like. No, there is no cycle in this graph.
94

---

Published as a conference paper at ICLR 2026
Cycle
QUERY: Given graph representation, your task determine whether the graph has cycle And the graph representation of GraphML.
Q: Is there cycle in this graph? A: No, there is no cycle in this graph
Given a graph representation, your task is determining whether the graph has a cycle Graph: GraphML
<edge source="4" target="6"/>
<edge source= target="6"/>
</graph>></grphmmb>
Q: Is there a cycle in this graph?
A:
TASK INFO: cycle, easy, CoT, GraphML
MODEL: Qwen-2.5
ACCURACY:
GROUND TRUTH: False
ANSWER:
the edges (6.5), (5,6) form a cycle, so there is cycle in this graph. given the graph representation in graphml format:
"'xml <7xml version='1.0 encoding='utfaa uuf-8">> <graphml wuthass/ttgggggggppmmngaaaaaaaar
knllsxxsi"httpp/wwwwww...pp
http://mww..oooooeome/////mmmnnn http://wwpp..gggooomaaaaa/ggagnp <graphh
xmlschema-inntancee
<graph "
<node id="0"/>
<node id="6"/>
<node id="5"/>
<node id="1"/>
<node id="4"/>
<node id="3"/>
<node id="2"/>
<edge source="("" target="6'/>
<edge source="6" "6" target="5"/>
<edge source= "6" target="1" 1"/>
<edge source= "6" target="4"/>>
<odge source= '6" target="3"/>>
<edge source= target="2"/>
</graph>
</graphm>>
we need to determine if there is cycle in the graph.
#*# analysis: node 6 is connected to nodes 0, 5, 1,4, 3, and 2. - the connections from node to nodes 5, 4, 3, and 2 do not form cycle by themselves. however, the connection from node 6 to node 5 and back node forms cycle.
#### conclusion: the edges (6, 5) and (5, 6) form cycle.
#*
is there cycle in this graph? a: yes, there is cycle in this graph. the cycle is formed by the edges (6, 5) and (5, 6).
95

---

Published as a conference paper at ICLR 2026
Cycle
QUERY: Given graph representation, your task determining whether the graph has cycle
Graph: Edge Set is [(4, 6), (3, 10), (0, 2). 14), 8), (I, 3), (0, 17), (13, 14), (2, 17), (3, 9), (3, 6), (3, 15), (0, 1). (0, 7), (0, 4), (0, 16), (0. 13), (3. 5), 11), 17), (0. 3), (0. 12). (2. 3), (0, 15), (7.
Q: Is there eycle in this graph?
A:
Let's think step by step:
TASK INFO: cycle, medium, O-CoT, Edge Set
MODEL: gpt-4o-mini
ACCURACY:
GROUND TRUTH: True
ANSWER:
To determine whether the graph represented by the given edge set has cycle, employ Depth First Search (DFS) or a Union-Fine Find algorithm.
1. *iitiializttion*** Start from any vertex and initiate search (DFS Union-Find),
"*Tracking Visits** Keep record of visited nodes. For DFS, maintain parent reference detect back edges.
**Finding Cycles**; During traversal, if you find visited vertex that is not the parent of the current vertex, cycle exists. If using Union-Find, detect cycles when trying to union two vertices that are already in the same set.
For the provided edge set, we systematically traverse each edge.
#i# Edges:
(4, 6)
(3, 10)
(0, 2)
(9, 14)
(0, 8)
3),
(0.
(13, 14)
17)
(3,9)
(3, 6)
(3, 15)
(0, 1)
(0,7)
(0,
(0, 16)
(0, 13)
(3,
(3, 11)
(3, 17)
(0,3)
(0, 12)
(2, 3)
(0, 15)
(7, 8)
#*# Analysis: From the edges: The edge (3, 6) and (3, 5) suggests multiple connections around vertex
3. The edge (3, 17) connects back to vertex 3, which confirms presence of cycle. ### Conclusion: Yes, the graph has a cycle.
96

---

Published as a conference paper at ICLR 2026
Cycle
determine whether there is cycle in an undirected graph, you can use a depth-first search algorithm to traverse the graph. If the algorithm ever returns to a node has already visited, then it has detected cycle in the graph.
QUERY:
Given a graph representation, your task is to determine whether the graph has a cycle And the graph representation Adjacency Set [3: (4, 5) 4: (3). (2, 3), 1: (0), (1, 2: (0, 5}]
Q: Is there a cycle in this graph?
A: traversal visits nodes {0, 2, 3, 4). no point does the traversal return an already visited
node that is not the parent. Hence, no cycle is detected.
Given graph representation, your task to determine whether the graph has cycle
And the graph representation Adjacency Set is (3: {0, 4. 5). (3), {0. 4), (1.3), 4: (1.
3). 2: {
Q: Is there a cycle this graph?
A: During DFS, the traversal path includes nodes {0, 1, 4, 4, 5, 2). A back edge detected from node
to node 4, where the latter already recursive stack. So, there is cycle this graph.
Given graph representation, your task determine whether graph has cycle
And the graph representation of Adjacency Sct (1. 3, 5), (7), {6. 7), 0: (2), 2: (0, 4, 6), (7), (2), (2, 5)}
Q: Is there cycle this graph?
A: DFS visits nodes (0, 6, 7, 1, 1, The traversal completes without encountering any back edges, there is no cycle in this graph.
Given graph representation, your task determine whether graph has cycle
And the graph representation of Adjacency Set is: (0: (1, 2), 1: (0, 4), [0, 6), 6: (2, 4), (3). {5. 7), {1, 6, 7), 7: (3.4)
Q: Is there cycle in this graph?
A: While traversing nodes {0. 4, 3. 5, 6, 3. back edge from node to node 0 identified, the latter is already in recursion stack DFS, yes, there is cycle in this graph.
Given graph representation, your task is determine whether the graph has cycle
And the graph representation of Adjacency Set is {1: {0, 2, 4, 5, 6). 3: {0, 1, 5, 6), 2: {0, 1, 6), (0, 2, 5, 6). 6: (0, 2. 4, 0: {1, Q: Is there cycle in this graph?
5, 6), (0, 1, 2, 6))
A: This graph contains multiple cycles. During DFS, cycles are detected through paths like (3-4-6-3), (1-2-3-1), and (4-5-6-4). Multiple back edges indicate the presence of several cycles.
Given graph representation, your task is determining whether the graph has cycle
Graph: Adjacency Set is [0: [1. 2. 3, 5, 6, 18, 20, 30), 1: [0, 2. 3, 4, 5, 6. 21), 2: (0, 1. 4, 6.
24. 25
1, 32, 4, 28), 4: [0, 3, 6, 14, 23),
1, 3,
10, 11, 31],
1.
2, 5,
13, 17, 19, 22, 26, 27). (2), (6), (6), 10: (5), 11: (5), 12: (6. 15), 13: (16.
6). 14: (4), 15: (12), 16: (13), 17: (6), 18: (0), 19: (6) 20: (0) (1), (29, 6), 23:
(4).
(2). 25: (2), 26: [6), 27: (6), 28: (3), 29: (22), 30: 31) 32: (3))
Q: Is there a cycle in this graph?
A:
TASK INFO: cycle, hard, Algorithm, Adjacency Set
MODEL: 04-mini-2025-04-16
ACCURACY:
GROUND TRUTH: True
ANSWER:
The subgraph induced by nodes 0 through 6 is complete graph (each of those nodes connects to every other). which immediately contains many 3-node cycles (for example, 0-1-2-0). Therefore, this graph definitely has cycle. Yes, there cycle in this graph
97

---

Published as a conference paper at ICLR 2026
Shortest path
can use Depth-First Search (DFS) algorithm to find the shortest path between two given nodes in undirected graph. The basic idea is to start at one of the nodes and use DFS to explore all of its adjacent nodes. At each node, you can keep track the distance takes to reach that node from the starting node. Once you have explored all the adjacent nodes, you can backtrack and pick the node which has the shortest distance to reach the destination node.
QUERY:
Given graph representation, your task is to compute shortest path between the specified two nodes
And the graph representation of Edge List is 0
066
Q: Give the shortest path from node 5 to node 8.
A: We use Depth-First Search (DFS) starting from node 5. First, explore the neighbors of node 5, which connected to node From node we explore its neighbors, node 1,8,4. And we find that node is connected node 8. Therefore, the shortest path from node to node 8 is 5,0,8.
Given graph representation, your task is to compute shortest path between the specified two nodes.
And the graph representation of: Edge List 0
02
1 2
2
3
30
43
41
40
54
53
64
63
70
76
84
8.
Q: Give the shortest path from node to node
A:
TASK INFO: shortest path, easy, Algorithm, Edge List
MODEL: llama3
ACCURACY:
STARTING AND ENDING NODE: 0, 1
ANSWER:
we use depth-firs first search (dfs) starting from node 0. first, we explore its neighbor node 1. therefore, the shortest path from node node is 0.1.
98

---

Published as a conference paper at ICLR 2026
Shortest path
QUERY: Given graph representation, your task is to compute shortest path between the
specified two nodes
And the graph representation of Adjacency List is: (0: [8. 4. 3, 2, 6]. [0], [0],
[0]. 2: [O]. 5: [0]. I: [0]. 6: [0]]
Q: Give the shortest path from node 5 to node 8.
A: The shortest path from node 5 to node 8 is 5,0,8.
Given graph representation, your task is to compute shortest path between the specified two nodes
And the graph representation of Adjacency List is: (0: [1], 1: [0, 2, 3, 4, 8). 2: [1. 7). 3:
[1]. 4: [1]. 5: [1]. 6: [1], 7: [2], [1]]
Q: Give the shortest path from node 3 to node 7.
A: The shortest path from node 3 to node 7 is 3, 3,1,,77.
Given graph representation, your task is to compute shortest path between the specified two nodes
And the graph representation of Adjacency List is [3: [1, 7, 4, 61, 1: [3. 7, 5], 8: [0. 4.
6, 5, 7]. 0: [8, 2, 7], [0, 5, 7, 1, 3). [3, 0, 6, 81, 5: [7. 8, 1J. 4: [6.
31, [1, 8, 5, 0])
Q: Give the shortest path from node 4 to node 2. A: The shortest path from node to node 2 is 4,8,2.
Given graph representation, your task is to compute shortest path between the specified two nodes
And the graph representation of Adjacency List is : (0: 7], [0], 7: [0. 1, 31, 7], 5:
[1]. 2: [4], 4: [2. 3J, 3: [4, 7]}
Q: Give the shortest path from node 6 to node 5.
A: The shortest path from node 6 to node 5 is 6,0,7,1,5.
Given graph representation, your task is to compute shortest path between the specified two nodes
And the graph representation of Adjacency List is [0: [1, 2, 5J, 1: [0, 2, 3, 6). 2: [0. 1, 3, 5], [1, 4], 4: [3, 0], 5: [0, 2, 6], 6: [1. 5]] Q: Give the shortest path from node to node 0.
A: The shortest path from node node 0 is 6,1,0.
Given graph representation, your task is to compute shortest path between the specified two nodes.
And the graph representation of: Adjacency List is (0: 2], [0. 3, 6, 7], 0, 4, 5], 3: [2. 1, 4, 7J. 4: [2, 3, 6]. 5: [1, 2), 6: [4, 1]. 7: [1. 3]}
Q: Give the shortest path from node 3 to node 2.
A:
TASK INFO: shortest path, easy, k-shot, Adjacency List
MODEL: Qwen-2.5
ACCURACY:
STARTING AND ENDING NODE: 3,
ANSWER:
the shortest path from node to node 2 is 3,1,2.
99

---

Published as a conference paper at ICLR 2026
Shortest path
QUERY: Given graph representation, your task is to compute shortest path between the specified two nodes.
And the graph representation of: GraphML is <mmll version= 1.0"
encoding='utf-8"> <graphml httpa//s//.tgggg.ooooo/ooo////no http:////oaaill//////////gngnnno
http://rrahhll.grapiddavwgg http:/ng...gggp/////.oomm
><graph <<dggg
edgede-
fault= "niiiauuuddrceee">><onnnio id="0"/>
<node id="1"/>
<node id="2"/> <node id="3"/>
<node id="4"/>
<node id="5"/>
<node id="6"/>
<edge source="O" target="1"/>
<edge source="0"" target="2"/>
<edge source="O" "0" target="3"/>
<edge source= "0" target="6"/>>
<edge source="3" target="4"/>
<edge source="3" target="5"/>
<'graph></grapmm>>
Q: Give the shortest path from node 2 to node
A:
Let's construct a graph with the nodes and edges first:
TASK INFO:
shortest path, easy, O-Intrrut,, GraphML.
MODEL: Qwen-2.5
ACCURACY:
STARTING AND ENDING NODE: 2, 1
ANSWER:
nodes: 0, 2,2,3,4,5, 6 edges: (0, 1), (0, 2), (0, 3), (0, 6). 4), (3 now, we want find the shortest path from node 2 node 1.
we can use the breadth-firs' search (bfs) algorithm find shortest path an unweighted graph
bi's traversal of the graph starting from node 2:
start at node 2. visit node (since it's directly connected to node 2) 3. visit nodes 3,
and 6 (all directly connected to node 0)
since node one of the visited nodes, the shortest path from node 2 to node 1 is: -~>0~>
you are an assistant. user will provide you with a task. your goal to complete the task as faithfully you can. you will ask for clarification in cases where the task is not clear. you will not make any assumptions and you will not introduce any information not provided in the input.
100

---

Published as a conference paper at ICLR 2026
Shortest path
QUERY: Given graph representation, your task is to compute shortest path between the
specified two nodes
And the graph representation of Adjacency Set is: (0: (1. 4, 6, 8). 8: (0), 4: (0),
{0), 2: (0), 5: (0), I: (0), 6: {0]]
Let's construct a graph with the nodes and edges first.
Q: Give the shortest path from node 5 to node 8.
node let's explore the graph: Node is directly connected to node 0 (via the edge 5). Node 0 is directly connected to node 8 (via the edge 0
A: To find the shortest path from node
8). Thus, the shortest path from node 5 node 5,0,8.
Given graph representation, your task is to compute shortest path between the specified two nodes.
And the graph representation of: Adjacency Set
[3: (8. 9, 6). 8: 2, 3, 4), 4: (8, 9, 6, 71, 6: [1. 2, 3, 4), 1: (8, 9, 10, 6), 1: 2: (8. 9, 10, 6).
10: (0. 2). 9: (0, 1, 2, 4), 7: (4), 0: (9, 10, 5J. (0))
Let's construct a graph with the nodes and edges first.
Q: Give the shortest path from node to node
A:
TASK INFO: shortest path, medium, Instruct, Adjacency Set
MODEL: Slande-3---oonne-22024111
ACCURACY:
STARTING AND ENDING NODE: 6, 5
ANSWER:
Let me help find the shortest path from node 6 to node 5 in this graph.
Let's analyze the possible paths: 1. Node 6 is connected to nodes 1, 2, 3, and 2. To reach only connected to node 0) 3. To reach go through either node 9 or node 10 4. From node 6: We can reach node or node 2, which both connect to nodes and 10 Through node 9 or 10, we can reach node 0 From node O,
node 5, we need to go through node (as node
node O,
reach node 5
The shortest path would be: +
1 + 0-
or
6 + - 6
(both paths have the same length) The shortest path 6,1,9,0,5
101

---

Published as a conference paper at ICLR 2026
Shortest path
QUERY:
Given graph representation, your task is to compute shortest path between the specified two nodes. And the graph representation of: Edge List is
0
20
00
04
2
61
70
8
90
94
100
114
Q: Give the shortest path from node to node A: Let's think step by step:
TASK INFO: shortest path.mcchimm0--CoTT..Uu List
MODEL: 04-mini-2025-04-16
ACCURACY:
STARTING AND ENDING NODE: 1,2
ANSWER: We can use a breadth- -fisst search starting from node 1:
Start node Neighbors are 6. 2. Visit node (distance 1). Its neighbors are 1, 2, 4, 7, 9, 10. We discover node 2 here. 3. We stop soon we reach node 2.
Reconstructing the path gives
2. The shortest path from node to node 2 is 1,0,2.
E.6 MORE FINDINGS FROM EVALUATION RESULTS
Finding @: Serialization exhibits task-dependent effects, with both open-source and closed- source models excelling at concise and structured representations. From Tables 18 and 21, the formats most suitable for open-source and closed-source models vary across tasks such as Diameter calculation, Triangle counting, Shortest path and Cycle detection. In open-source models, nearly every task exhibits a preference for a distinct format. In contrast, closed-source models tend to perform well with formats such Adjacency List (AL) and Adjacency Set (AS), yet Graph Markup Language (GMaL) also achieves superior performance in certain tasks. Taking the Cycle detection benchmark an example, the Edge Set (ES) format outperforms other alternatives, whereas in closed- source models, formats like Graph Markup Language (GMal.) demonstrate marked advantages. Such task- lask-specific preferences for serialization formats further highlight the importance and significance of GRAPHOMNI.
Finding @: Complex multi-step prompts can negatively impact the performance of closed- source models. In the Triangle counting task, open-source models performed very well with more examples in Instruct and k-shot scenario, while closed-source models excelled using minimal prompting strategies such O-Algorithm, which avoid elaborate reasoning steps intermediate explicit guidance (Tables 17, 20 in Appendix E. E) This pattern suggests that complex or abstract multi-step prompts can confound closed-source models in certain challenging tasks.
102

---

Published as a conference paper at ICLR 2026
E.7 ANALYSIS ON EFFICIENCY VIA NUMBER OF OUTPUT TOKENS
To assess inference efficiency, we measure the total number of output tokens each model pro- duces-tokeniize consistently with the OpenAl GPT-3.5-turbo tokenizer' analyze how token counts vary across four key dimensions: difficulty levels (Table 23), task categories (Table 24), serialization formats (Table 25), and prompt schemes (Table 26). The average token counts under each condition are reported in these tables, together with the main results of accuracy, providing comprehensive view of the trade-offs between output verbosity and model performance across our benchmark.
E.7.1
OVERALL ANALYSIS
Model Response Length vs Accuracy
1400
0.8
Avg. Response Token Count
1200
1000
Avg. Accuracy
0.6
800
0.5
600
0.4
200
0.3
Liama3
Mistral
Phl-á
Llama3, 1
Owan2.5
GPT-4o-mini
Gemini-2.0
GPT440
Claude-3.5 5
od-mini
Model
Average output tokens versus overall accuracy across all graph-theoretic tasks. Models are ordered by the average performance. Models Green are open-source models while others in Red are closed-source ones.
Figure 32:
Figure 32 highlights two distinct patterns. Closed-source models GPT-4o, Claude-3.5. and Gemini- 2.h-aehieve the highest accuracy while keeping total output below roughly 300 tokens, showing tight control over generation length. o4-mini, reasoning- focused model stands out: final answers remain short (about 100 tokens), but it adds lengthy chain-of- ohain-of-thought (up to 1.6 K tokens), yielding strong accuracy with markedly larger overall output. Open-source models display a different trend. Llama-3.1 and Qwen-2.5 match the best accuracies only when they generate much longer responses, whereas Llama-3 and Mistral remain shorter and less accurate. These contrasts persist across difficulty levels, task categories, serialization formats, and prompt schemes, detailed in Tables 23-26.
E.7.2 ANALYSIS CONCERNING DIFFICULTY OF TASK
Table 23: Average output tokens per model at each difficulty level (Easy, Medium, Hard). largess number of
Orange Blue Light purple highlights indicate the output tokens.
Difiialtt Llamm 188)
LJamn-3.1 (MB)
Misrral (710)
Phi-4 (14m-
Qwen (TB)
Clande-35
GPT-4b
GPT-4s mint
Gemini 2.0
ed-nini
Average
Amver
Revomiing
eaty
210.51
1050
375
517.31
881.2
19833
257.17
248.50
440.26
143377
$41.68
459.60
hard
743.44 46
419 96
694.8
873.01
181.21
261.1
411
70.35
1960
27
Averagg
Table 23 presents the token output across different models under varying levels of task difficulty. Overall, most models exhibit small variation in output length as task difficulty increases. However,
Note that the numher of the reasoning of o4-mini is obtained from the metadata of cach API call.
103

---

Published as a conference paper at ICLR 2026
notable exception is the reasoning model, which demonstrates distinct pattern: task difficulty rises, the number of tokens in the final answer tends to decrease, while the length of the reasoning process correspondingly increases.
E.7.3
ANALYSIS CONCERNING TASK TYPE
Table
24:
Average output
tokens per model for each graph-theoretic task. Light purple highlights indicate the lanees//seeot-lgges//tidddddddd number of
Orange Blue/
output tokens.
Tonk ty
Elbray (88)
T. (88)
Mixeral
(14-4
(7
Claide35
GPT-i
GPT-4e-id
Grmisi-3.
Aseroge
Assvect
Roaesing
Buensoor calestation
31327
1830.11
San55
tesr path
Trasngle counting
16890 347.19
24697
2156.73
613.0
Anerage
Further insights can be drawn from Table 24, which reveals a clear correlation between output tokens and task type. Specifically, tasks such as Connectivity and Cycle detection consistently yield significantly shorter outputs compared to other tasks, as they are relatively casier compared to others. Among open-source models, the Shortest path task results in the longest outputs, whereas for closed-source models, the BFS order and Triangle counting task generate the highest average token counts In the of the reasoning model, the token output associated with the reasoning process increases markedly with the complexity and difficulty of the task- when considering task accuracy. For instance, in the Triangle counting task, the reasoning component alone produces an average of over 2000 tokens, highlighting the model's ''' tendency to elaborate more extensively as task complexity increases.
E.7.4 ANALYSIS CONCERNING SERIALIZATION FORMATS
Table 25: Average output tokens per model under different serialization formats. number
Orange Blue Light purple highlights indicate the larges//scoond-laressthhiii of output tokens.
Scrieliution formet
Llama 3 (it
C.ama (8B)
Mistral (7B)
P1i-4:140) Qwee- Clunde-1.5
GPT-4s
GPT-4a-mitt
Genizi-3.
Averige
Ansrre
Reesesing
Adjaceey
959,19
Adjacezy
-02.13
2H1.33
243.48
1315.50
Adjacezy
236.35
392.00
256.15
49157
Edge Lis
2505:
$50.30
383.71
1250.25
444.77
Bltc
Graph Medcling Laggage
GrephNE
212.87
487.70
Table 25 presents the influence of different graph serialization formats on the number of output tokens generated by various models. Overall, more complex formats-such as GMaL and Adjacency Matrix-tend to induce longer outputs, whereas simpler formats-such as Adjacency List and Edge List associated with significantly shorter outputs. Among the evaluated models, open-source models such as Llama-3.1 and Qwen-2.5 consistently produce a higher number tokens across most formats. This effect is particularly pronounced for Llama-3.1 under the GMal. format, where its output length reaches a peak. In contrast, closed-source models generally yield more concise outputs, with Claude-3.5 being especially compact. An exception is observed in o4-mini, whose output length is substantially higher due to the inclusion of intermediate reasoning steps.
E.7.5 ANALYSIS CONCERNING PROMPT SCHEMES
Table 26 further examines the impact of different prompting strategies on model output. Prompts that involve reasoning instruction (e.g., CoT, Instruct, and 0- O-Istruut) significantly increase output length, a trend that is particularly salient in open-source models. For instance, under the O-Instruct prompt, both Llama-3.1 and o4-mini produce extended outputs. In contrast, prompts with no
104

---

Published as a conference paper at ICLR 2026
Table
26:
Average
output
tokens
per
model for each prompt scheme.
Orange Blue
Light purple highlights indicate the lagess/scoon-aagee/hhiii number of
output tokens.
Prompt
0-Algrina
1071.04
053.42
305.95
654.55
123.07
Alpontm
302.5
936
245.75
149.0
OUT
192.8
936.17
375.05
851.31
1042.1
16495
15463
203 33
255.13
92.05
1271.55
501.90
Insract
212322
95722
335.99
91742
1045.12
176.65
109.42
200.20
255.77
199.7
130296
822.33
ITM
WITM
1116 18
405.14
961.33
061M8
205.71
50.87
179 M4
119.04
182.4
1319
421.07
K-Slor
160.9
750.30
440.39
1024.7
150945
170.22
196.37
39:31
270.39
67.35
1277.14
492.7
Dabet
305.2
115318
331.4
145.15
$41
15:.01
200.99
14092
329/1
1201.00
46
Average
253.8
1023.83
399.57
293.24
867.2
193.93
266.77
230.8
447.32
115.56
1253.59
instruction (O-shot) or few-shot examples (K-Shot) tend to yield shorter outputs. Closed-source models exhibit relatively stable output lengths across prompt types, suggesting stronger control over generation behavior.
E.7.6 COST-ACCURACY TRADEOFF ANALYSIS
Table 27: Per-Query Inference Cost Analysis. Costs are calculated based on cur- rent API pricing (as of November 2025) with average input tokens of 933 per query.
Bold orange Underlined blue Light purple lowest cost each category.
highlights indicate lowest/sconddowwst/hhic
Model
Input Cost ($)
Output Cost (S)
Total Cost (S)
Open-Source Models
Llama-3.1 (8B)
0.000019
0.000031
0.000049
Mistral (7B)
0.000187
0,000080
0.000267
Phi-4 (14B)
0.000056
0,000084
0,000140
Qwen-2.5 (7B)
0.000037
0,000089
0,000126
Closed-Source Models
Claude-3.5
0.002799
0.002894
0.005694
GPT-4o
0.002333
0.002666
0.004998
GPT-4o-mini
0.000140
0.000142
0.000282
Gemini-2.0
0.000093
0.000178
0.000271
04-mini
0.001026
0.006168
0,007194
Table 27 presents per-query inference costs based on current API pricing. Cost varies by three orders $0.007194 (o4-mini)
of magnitude across models, ranging from $0.000049 Alama-3/Llama-3../ per query. Open-source models uniformly cost less than $0.0003 per query, while closed-source models span from $0.000271 (Gemini-2.0) to $0.007194 (o4-mini).
Figure 33 visualizes the cost-(mean) accuracy tradeoff on all tasks. 04-mini achieves the highest accuracy (80.96%) but incurs the highest cost. Notably, no model dominates across all metrics. The optimal choice depends application requirements: open-source models for cost-sensitive deployments with relaxed accuracy constraints, Gemini-2.0 or GPT-4o-mini for balanced cost- performance, Claude-3.5 or GPT-4o for high-accuracy applications, and 04-mini when maximizing accuracy justifies premium costs. For full benchmark evaluation (241,726 queries), total costs range from $1 1.85 (Llama-3) to $1,739 (o4-mini), a 147 x difference that has significant implications for large-scale graph reasoning deployments.
F DETAILED RELATED WORKS
As more and more semantic information is needed in graph-related applications (Zhu et al., 2025; Wu et al., 2022; Jian et al., 2025a; Wang & Zhuang, 2025; Zhu al., 2026), adopting L.LMs for
105

---

Published as a conference paper at ICLR 2026
Cost-Accuracy Tradeoff Across Evaluated Models
Open-Source Closed-Source
Claude-3.3
Cost per Query (USD, log scale)
12-3
Mistral
Gemini
GPT-4o-mini
Qwen-2.5
12-
Liama-3 Llama-3.1
Accuracy
Figure 33: Cost-accuracy tradeoff across evaluated models average. Each point represents model's mean accuracy versus per-query inference cost (log scale).
graph-structured data has become increasingly important. Integrating LLMs with graph-structured data merges linguistic reasoning capabilities with structural representation insights. While compre- hensive discussions on LLM- LLM-graph integration can be found in Appendix F.1, recent benchmarks specifically targeting LLM applications for graph reasoning, such as LLM4DyG (Zhang al., 2024b), GraphTMI (Das et al., 2024), GraphInstruct (Luo et al., 2024b), and MAGMA (Taylor al., 2024), have highlighted substantial progress and persistent limitations. These benchmarks reveal issues including narrow graph diversity, scalability constraints, and pronounced sensitivity input formatting. Studies on graph pattern comprehension and multi-hop reasoning (Dai al., 2024; Wang et al., 2023; Jin et al., 2024b) further emphasize brittleness under complex or noisy data conditions. Empirical analyses conducted by GPT4Graph (Guo al., 2024a) and Graph Wiz (Chen et al., 2024a) underscore performance gaps relative to specialized graph models and highlight computational inefficiencies. Additionally, recent contributions through transformer scaling studies (Sanford et al., 2024), comprehensive benchmarks like GraphFM (Xu et al., 2024), GLBennch (Li et al., 2024d), GraphAbstract (Zhao et al., 2025), and Graph Vista (Han et al., 2025), and specialized datasets (Yan al., 2023; Fatemi et al., 2024) have provided valuable yet often limited insights. ProGraph (Li et al., 2024b) offers innovation but introduces extra computational overhead due to external dependencies.A detailed summary of these benchmark-related works is provided below.
F.1 LLM APPLICATIONS ON GRAPH DATA
The intersection of LLMs and graph-structured data has emerged as an active research domain, combining the nuanced contextual reasoning abilities of LLMs with the structural representational power of traditional Graph Neural Networks (GNNs). Initial studies addressed fundamental challenges such reducing sensitivity to prompt formulation (Sclar et al., 2024) and enabling zero-shot cross- dataset transferability (Li et al., 2024c). These foundational efforts have supported the development of generative models that jointly leverage textual and structural graph information, creating unified semantic embeddings for enhanced performance (Wang et al., 2024b; Fang et al., 2025b; Li et al., 2024a; Kong al., 2024), which have much more powerful expressive ability compared traditional self-supervised learning (Jian et al., 2025b).
Subsequent research built upon these foundations by focusing on enhancing the robustness of LLMs when applied graph tasks (Guo et al., 2024b) and advancing techniques for effectively translating complex graph structures into natural language, notably through methods like graph-syntax trees (Zhao et al., 2023). Recent advancements have directly embedded graph reasoning capabilities within LLM architectures, significantly extending their application beyond purely textual domains (Hu al., 2023). In this context, specific methodologies have been developed that embed graph learning
106

---

Published as a conference paper at ICLR 2026
modules and leverage instruction tuning to improve alignment between structural data and LLM input modalities (Chai et al., 2023; Tang et al., 2024).
Parallel efforts have provided extensive overviews of the evolving field through comprehensive surveys (Li et al., 2023; Jin et al., 2024a; Zhao et al., 2026), highlighting foundational concepts such Graph Foundation Models that employ dedicated graph vocabularies for effective cross-domain learning (Mao et al., 2024). Concurrently, advances in parameter-efficien encoding techniques, exemplified by GraphToken (Perozzi et al., 2024), and retrieval-augee eetrevaal-uugmentee frameworks such as G-Retriever (He et al., 2024), have further refined the processing and utilization of graph structures. Moreover, basistant-based frameworks employing instruction-tuie tuntingg strategies, including LLaGA (Chen et al., 2024b) and InstructGraph (Wang et al., 2024a), demonstrated significant potential for enabling L.LMs to produce high-quality graph-structured outputs through preference-aligned interactions.
Complementing these directions, significant innovations have emerged within graph representation learning, exemplified by models like OpenGraph (Xia et al., 2024) and MuseGraph (Tan et al., 2024), which integrate scalable transformers, data augmentation, and graph-specific instruction tuning for robust zero-shot performance and general graph mining applications. Additional methods employing compact node identifiers (Luo et al., 2024a) and attributed random walks for fine-tuning (Tan al., 2023) have notably improved inference efficiency, collectively illustrating coherent evolution towards integrated frameworks that effectively harness the combined strengths of LLMs and graph-centric approaches.
F.2 BENCHMARKS ON LLM APPLICATION TO GRAPH DATA
Recent benchmarks assessing LLM capabilities graph reasoning tasks have significantly advanced understanding yet still present important limitations. Benchmarks such as LLM4DyG (Zhang et al., 2024b), which emphasizes spatial-temporal dynamics, typically neglect the complexity inherent to static graph structures. Similarly, GraphT'I (Das et al., 2024), exploring various graph input modali- ties (text, motif, image), has exposed inherent trade-offs between token efficiency and representational expressiveness, potentially impacting scalability.
Other benchmarks, including GraphInstruct (Luo al., 2024b) and MAGMA (Taylor al., 2024), incorporate traditional graph reasoning tasks with explanatory strategies but remain limited by small-scale graph sizes and lack comprehensive coverage across diverse graph structures. Studies specifically targeting graph pattern recognition and natural-language-bsseee graph problem-solving (Dai et al., 2024; Wang et al., 2023) have further revealed pronounced sensitivity to input formats, resulting in brittleness under complex or noisy conditions. Additionally, frameworks designed to mitigate multi-hop reasoning inaceuracies through graph-centric reasoning chains (Jin et al., 2024b) and examinations of generalization beyond memorized patterns (Zhang et al., 2024a) continue to illustrate significant unresolved challenges.
Empirical assessments conducted by initiatives such GPT4Graph (Guo et al., 2024a) and instruction- tuned benchmarks like Graph (Chen et al., 2024a) highlight persistent performance gaps com- pared to specialized graph neural architectures, accompanied by elevated computational demands. More recent contributions, including scaling analyses of transformer models (Sanford et al., 2024), comprehensive benchmarks like GraphFM (Xu et al., 2024) and GLBench (Li et al., 2024d), and specialized datasets such as CS-TAG (Yan et al., 2023) and encoding studies (Fatemi et al., 2024), have substantially enriched the literature but remain constrained by challenges related to homogeneity, training inefficiencies, and limited scalability, While innovative, solutions such as ProGraph (Li et al., 2024b), employing programming- based integration and external API retrieval, introduce additional computational overhead and dependencies.
G LIMITATIONS AND FUTURE DIRECTIONS OF GRAPHOMNI
While GRAPHOMNI significantly advances the evaluation of large language models (LLMs) on graph-theoretic tasks, several considerations highlight opportunities for future enhancement:
Diversity of Tasks: The benchmark presently includes key canonical tasks, which may not fully represent the diversity of graph-related problems encountered in practice. Expand-
107

---

Published as a conference paper at ICLR 2026
ing the task set include dynamii, temporal, heterogeneous graph challenges could offer deeper insights into model performance. Future work should focus on defining and integrating tasks that capture evolving network structures and multi-relat nulti-relational data.
Generalizability of Findings: GRAPHOMNI evaluates LLMs under controlled experimental conditions, which might not entirely reflect performance in less structured, roal-world environments. Future work could include testing the generalizability of models across various practical conditions, such as noisy data, incomplete graphs, or domain-specific variations, better understand the robustness and applicability of LLMss.
Addressing these aspects will further enhance the robustness, applicability, and inclusivity of GRAPHOMNI, fostering wider adoption and deeper insights into LLM performance.
H ADDITIONAL ABLATION STUDIES
H.1 PERFORMANCE VS. TIME COMPLEXITY OF TASKS
H.1.1 TIME COMPLEXITY ANALYSIS
The time complexities are determined based on well-established algorithms in graph theory (we are aware that more efficient algorithms are available, especially for Diameter calculation and Triangle counting, but we use the most naive implementations since they typically reflect how I.IMs approach these tasks):
Connectivity: O(V + E) Determined via single breadth- first search (BFS) or depth- first search (DFS) traversal starting from node to check reachability another node.
Cycle detection: O(V E) Implemented using DFS with back-edge detection; each node and edge visited at most once.
BFS order: O(V + E) - Standard breadth-first traversal visits each node once and examines each edge once.
Shortest path: O(V + E) for unweighted graphs using BFS, or O(E + V Voog V) for weighted graphs using Dijkstra's algorithm. Since our benchmark uses unweighted graphs, we report O(V + E).
Diameter calculation: O(V(V + E)) - Requires computing all-pairs shortest paths, typically achieved by running BFS from each node, resulting OVV) BFS operations each costing O(V E).
Triangle counting: O(V3) naively by checking all triplets of nodes, or O(V d with neighbor-based enumeration where davg is the average degree. For dense graphs or without optimizations, this remains the most computationally intensive task.
H.1.2 ALIGNMENT ANALYSIS
Tables 28, 29, and 30 demonstrate partial alignment between computational complexity and LLM difficulty. At the extremes, correspondence clear: Triangle counting (O(V3) achieves only 15.45% accuracy (closed-souree Hard) and 6.77% (open-source, Hard), while Connectivity ierectiiittyy ty (O(V- E)) reaches 91.90% and 75.97% respectively. Similarly, Diameter calcutation (O(V(V E))) yields 40.09% (closed-source) and 21.33% (open-source), ranking as the algorithmically and empirically.
task both
However, among tasks with identical O(V + E)) complexity, performance diverges substantially. Connectivity maintains 91, 91.90% accuracy on hard instances, while BFS order collapses to 27. 15%, 64.75 percentage point gap despite equivalent asymptotic complexity. This divergence indicates that computational complexity alone does not determine LLM difficulty.
H.1.3 FACTORS BEYOND COMPUTATIONAL COMPLEXITY
Three task characteristics account for this divergence. First, output structure critically impacts performance: binary decisions (Connectivity, Cycle detection) achieve 91.99 91.90% and 79 99.24%.
108

---

Published as a conference paper at ICLR 2026
while sequence generation (BFS order) and numerical enumeration (Triangle counting, Diameter calculation) fall to 27.15% 15.45% and 40.09% respectively. Second, error propagation varies by task aope-sequence tasks suffer cascading failures where single emrrs invalidate entire outputs, as evidenced by BFS order's severe 62.52% performance drop. Third, reasoning scope distinguishes task difficulty: local reasoning tasks (Connectivity Cycle detection) degrade minimally (4.31% 1,3,%, 2.74 2.74%), while global reasoning tasks requiring complete graph traversal (Diameter calculation. BFS order) drop sharply (41.34%, 62.52%)
Table 30 quantifies these effects: open-source models degrade 16.93% on average from Easy to Hard, while closed-source models drop 26. 26.26%. Crucially, this degradation correlates more strongly with reasoning scope and output structure than with algorithmic complexity- BFS order (O(V E)) degrades more severely than Diameter calculation (O(V(V + E))). demonstrating that maintain- ing sequential dependencies in textual representations poses greater challenges than computational intensity per se.
H.1.4 CONCLUSION
Our analysis reveals that computational complexity establishes a baseline for LLM difficulty, as evidenced by Triangle counting and Diameter calculation ranking as both algorithmically expensive and empirically challenging. However, output structure and reasoning scope play equally critical roles. The 64.75 percentage point gap between Connectivity and BFS order-both O(V + E`) lasks-demonstren sequential dependencies, performing combinatorial enumeration, and generating outputs under strict ordering constraints. These limitations manifest independently of algorithmic complexity and persist across all evaluated models (Tables 28-30), indicating fundamental constraints in how current LLM architectures encode and manipulate graph-structured information through natural language representations.
that LLMs struggle disproportionately with maintaining long-range
Table 28: Open-Source LLM Performance Across Tasks Ranked by Computational Complexity (Mean Accuracy %). Bold orange Underlined blue Light purple highlights indicate best/second- best/third-bet best performance in each difficulty level.
Taok
Time Complecty
Limin-JJ1 J.1
Xlistril
Pla-
Fass (5-10mmdes nodle)
Qwco-25-7:38
QWcU-LL5-7
QwanJ
Liunn-3.
Mistral
Ph-44
Hand (20--30 modec)
QwennL15 15-72
Qwcc-J5577
Qwcn-
Triugge
(19)
12:55
18.55
41.36
4.95
2.55
4.3
19:54
OXV(V-E1)
D(V
65.87
Shonre park
O(V+48)
36.75
31.18
72.89
23.85
12:21
12.53
56.38
Oyte
(0V-48)
$5.49
55.44
45.25
14.00
62.19
90.30
$2.40
51.86
40.64
88
$5.85
86.81
Donsectvviy
O(V+8)
70.33
79.90
96.29
30.74
80:10
97.17
74.15
76.77
68.99
$0.00
31.19
02.89
Table 29; Closed-Source LLM Performance Across Tasks Ranked by Computational Complexity
(Mean Accuracy %). Bold orange Underlined blue Light purple highlights indicate best/second-
best/third-bee -bett performance in each difficulty level.
Easy (5-10 nodes)
Hard (20-30 nodes)
Task
Time Complexity
Claude-3.5
GPTT-4o
Gemnnn--200
o4-mmini
Claude-3.5
GP'T-4o
Gemini-220
o4-mini
Triangle
O(V")
43.41
36.32
50.33
84.54
15.92
12.81
15.55
17.53
Diameter
83.71
63.99
79.14
98.88
56.70
45.60
23.45
34.61
BFS order
O(V+F)
91.42
81.48
90.31
95.46
26,80
21.58
27:17
32.45
Shortest path
O(V+E)
94.35
92.17
81.75
95.08
87.88
74.98
78.16
88.63
Cycle
O(V+E)
82.56
85.08
62.30
97.97
80.10
82.96
58.30
95.61
Connectivity
O(V+E)
98.38
95.63
92.61
98.23
96.99
90.59
87.99
92.02
H.2 SCALING BEYOND 50 NODES
To address scale concerns, we extend evaluation to 50-100 node graphs on representative models (Qwen-2.5-72B and o4-mini). Table 31 compares performance against the 20-30 node Hard split.
Performance degrades uniformly as graph size increases, but the fundamental patterns remain un- changed. Task difficulty ranking stays identical: Triangle counting and BFS order remain hardest, while Connectivity and Cycle detection remain most stable. Relative model perfor- mance gaps persist similar magnitudes across scales. Critically, failure modes emerge, i.e.,
109

---

Published as a conference paper at ICLR 2026
Table 30: Aggregate Performance Comparison by Model Category and Task Complexity with
Performance Degradation. Accuracy (%) with color intensity indicating performance level. A shows Easy- performance drop.
Open-Source
Closed-Source
Task
Time Complexity
Easy
Hard
A
Easy
Hard
A
Triangle
O(Va)
22.70
6.77
15.993
53.65
13.45
-38.20
Diameter
O(V(V+E))
52.30
21.33
-30.97
81.43
40.09
-41.34
BFS order
O(V+B)
37.37
9.43
27.94
89.67
27.15
62.52
Shortest path
O(V+E)
54.62
37.83
-16.79
90.84
82.41
8.43
Cycle
O(V+B)
63.78
59.76
-4.02
81.98
79.24
2.74
Connectivity
O(V+E)
81.87
75.97
5.90
96.21
91.90
4.31
Mean
52.11
35.18
16.99
82.30
56.04
266226
the same challenges identified in smaller graphs (combinatorial enumeration, sequential dependencies, serialization sensitivity) simply intensify.
These results confirm that our 5-30 node design captures the essential reasoning challenges. Larger graphs amplify these challenges quantitatively but reveal no new qualitative phenomena, validating our focus controlled-scall evaluation where reasoning capability, rather than resource constraints, determines performance.
Table 31: Results on 50-100 node graphs (EEH = Extremely Extra Hard). Results on the 20-30 node Hard split are shown in parentheses for comparison. Bold orange Underlined blue highlights indicate best/second-best performance.
Open-source Model
Closed-source Model
Task
Difficulty
Qwen-2.5 (72B)
04-mini
BFS order
EEH
8.1942.03 (22.03)
10.23+2.07 (32.45)
Connectivity
EEH
62.00+4.90 (84.09)
81.8618.24 (92.02)
Cycle
EEH
37.78+4.11 (68.40)
74.81+4.90 (95.61)
Diameter
EEH
8.8942.39 (29.59)
40.44+3.76 (34.61)
Shortest path
EEH
33.28+6.09 (72.53)
68.51+11.04 (88.63)
Triangle
EEH
2.36+0.67 (4.73)
2.85+0.71 (17.53)
H.3 ROBUSTNESS CHECK UNDER PROMPT NOISE (PERTURBATION)
To address concerns about robustness natural language variation, conduct supplementary evaluation examining model sensitivity to paraphrased prompts. In our main evaluation, we deliber- ately use deterministic phrasing within each prompt scheme to isolate the effects of our three core dimensions, i.e., graph types, serialization formats, and prompt schemes, without confounding factors from linguistic variation. This controlled design allows us systematically attribute performance differences to structural representation choices (serialization formats) and reasoning guidance strate- gies (prompt schemes) rather than incidental phrasing variations. However, real-world applications inevitably encounter diverse linguistic expressions of the same semantic content, and robustness to such variation is practical concern. We therefore design a controlled perturbation framework to assess whether our conclusions remain stable under realistic linguistic variation.
H.3.1 DESIGN OF THE STUDY
Task and Sample Selection. We choose to conduct this robustness analysis BFS order. This choice is motivated by three considerations: (1) is among the most challenging tasks our benchmark, exhibiting substantial performance gaps across models and difficulty levels; (2) it
110

---

Published as a conference paper at ICLR 2026
requires complex structured output (a full node ordering), making it potentially more sensitive to prompt variations that might affect the model's 's understanding of output format requirements; and (3) single representative hard task allows
given limited time and budget constraints, concentrating for deeper analysis. From the full BFS order dataset, we subsample 4,000 instances to balance coverage across graph types, serialization formats, prompt schemes, and difficulty levels.
Perturbation Design. Our perturbation framework defines noisy prompts as semantically equivalent (so it still problem with the same answer) but syntactically diverse variants of the original prompts. And they are generated through systematic paraphrasing of natural-language components while maintaining the absolute structural preservation of graph data. The design adheres to three core principles:
1. Semantic Equivalence: All perturbations preserve the semantic content and task requirements through lexical substitution, syntactic restructuring, and stylistic variation. So is designed to test linguistic invariance.
2. Structural Preservation:
identical across all perturbations. This ensures that performance variation reflects model sensitivity to linguistic expression rather than changes in the underlying graph structure. In this way, the nature of the problem does not change much, and the ground truth results will still be the same.
Graph representations remain character-for-chant
3. Comprehensive Coverage: Perturbations span all nine prompt types in our framework (Algorithm, COT, k-shot, Instruct, LTM, and their variants) and all seven serialization formats (Adjacency Matrix, Adjacency List, Adjacency Set, Edge List, Edge Set, GMoL, GMaL).
Perturbation Methodology. We construct task-specific variation pools for each perturbable com- ponent. For prompts containing algorithmic explanations, we develop multiple paraphrases that express the same procedural steps using different vocabulary, sentence structures, and explanatory styles. Figure 34 illustrates a representative example: the original formal description uses numbered list structure with technical terminology ("Initialize"" "enqueue", "ddeqeue"" "Mark visited"), while the perturbed version adopts a conversational flow with colloquial alternatives ("First", "pick your", "put it in, "take out", "Mark it as visited we don't 't check it again"). The transfor- mation achieve 47.9% word-level change while maintaining algorithmic correctness and semantic equivalence.
For few-shot answer components, we generate variations that maintain identical logical reasoning and final answers while modifying transitional phrases and technical terminology. Figure 35 demon- strates this approach: the original example uses formal procedural terms ("Dequeue". "neighbors", "enqueue", "visited")) that are systematically replaced with more natural alternatives ("Extract from queue", "neighboring nodes", "insert into queue", "seen"). This achieves 17.5% word-level change through 29 replacements with 82.5% similarity, preserving the reasoning structure while varying linguistic expression. For instructional components, we create alternatives for opening statements, reasoning indicators, and procedural connectives. For task descriptions in minimal prompts, we paraphrase the task specification itself.
The perturbation process employs delimiter- based component extraction to precisely identify natural language elements while avoiding graph data. Specifically, identify boundaries between natural language answers and graph representations (e.g., "And the graph representation of [format] is") ensure that variations are applied exclusively linguistic content. For each prompt, we ran- domly sample variations from component-specific pools matched to the prompt's (prompt scheme, serialization format) combination, apply targeted string replacement using bounded pattern matching, and verify post- post-perturbation that all graph representations remain unchanged through sprmat- validation procedures.
Quality Assurance. To guarantee structural preservation, we implement multi-level verification: format-specific validation for each of the seven serialization types (e.g., character-leve comparison of matrix blocks, structural validation for GML/GraphMLL exact content matching for list and set formats), automated testing on representative samples spanning all prompt-scrializate combinations, and per-instance validation confirming preservation before evaluation. Our implementation achieves
111

---

Published as a conference paper at ICLR 2026
Figure 34: Example of algorithm explanation perturbation. The original formal, numbered description (left) is transformed into conversational phrasing (right) while preserving algorithmic correctness. Highlighted changes show systematic replacement of technical terms with colloquial alternatives. Word-level changes: 47.9%.
92.%
17.5%
29
Word
Orginll Start Doanles
intplerrs 110 pestiag
trevens onaer mearting the
Pertabed Cumple
Scrrect free seal
Healforaig mite
Inspaber rolel
mel litts Strett
Enteart mueel
(enpeder midey
streciy leeey lnto musel
Enteort eveel
(enipter midey
becis SHi Street
Sotrict Woe sesoents
Insproonn metel of 11, soce
Feart wike vole
lntrict the cumee
Beleask from
inspleciing
Mace (emy moles Betine
knighering
glan,
Figure 35: Example of few-shot answer perturbation. The original formal reasoning (left) is para- phrased with natural language variation (right) while maintaining identical logical structure and final answer Color-coded highlights show systematic terminology replacement (e.g.. "Dequeue" + "Extract from queue", "visited" "seen"). Word-level changes: 17.5%.
100% graph structure preservation across all perturbations while successfully modifying 87.9% of prompts of the samples (with the remaining 12.1 representing cases where random sampling selects the original phrasing).
Summary. This framework enables systematic evaluation of whether model performance and our main conclusions remain stable under realistic linguistic variation, providing evidence for the robustness of our findings beyond the specific phrasings used in the primary benchmark.
H.3.2 EXPERIMENTAL RESULTS AND ANALYSIS
Experimental Setting. We evaluate two representative models from our main benchmark: 04-mini (top-performing closed-source reasoning model) and Qwen-2.5-72B (strongest open-source model). These models provide coverage of both closed-source and open-source categories and exhibited the highest performance in main evaluation. We report results aggregated across prompt schemes and serialization formats separately, as well as fine- grained breakdowns per model, to assess whether main conclusions about representation sensitivity remain stable under linguistic perturbation.
112

---

Published as a conference paper at ICLR 2026
Overall Results. emerge:
Tables 32 and 33 present results averaged across both models. Several key patterns
Preservation of relative performance patterns. The relative rankings of prompt schemes and serializa- tion formats remain largely stable between original and perturbed conditions. For prompt schemes (Table 32), Algorithm, CoT, and Instruct consistently rank among the top three performers in Easy mode under both conditions, while 0-Shot maintains strong performance in Medium and Hard modes. For serialization formats (Table 33), AL and AS consistently dominate across all difficulty levels in both original and perturbed settings, with AL achieving 92.26% to 93.41% % (Easy), 83.44% to 83.56% (Medium), and 48.27% to 50.33% (Hard). The persistence of these rankings confirms that our main finding holds under linguistic variation, as no single configuration works universally, but certain formats consistently outperform others.
Evidence of real perturbation effects. While relative patterns are preserved, absolute performance values shift measurably between conditions. For example, CoT improves from 85 26% 90.98% in Easy mode, while K-Shot shows variation from 80.48% to 78. 33.66. These changes confirm that our perturbations introduce meaningful variation rather than being trivial paraphrases. We note that performance differences may be partially attributable to the subsampling from the full dataset to 4,000 instances, though the consistency of relative patterns suggests this effect is limited.
Tusk
Diliculty
+-Agoritthm
0-CoI
0-Instruct
0-Sbot
Algorithm
Col
Instruct
K-Sbot
LIM
84.71s8.99
76.36110.06
82.32:9.07
83:22:9.08
85.227.12
25.2617.45
96.6166.3
80.0848.52
Original
M
65.82412.78
64.60e12.1
64.97e11.99 199
66.90133.99
66 19
61.7211.01
02 32.31
18.1144
65 48a11.94
32.22:10.81
31.878964
29-4249.90
31.69210.66
26.9018.37
22.651
20.9016.63
20.2648.55
18410.15
E
84.98+1400 09
75.50=14.00 14.08
85.5449.64 64
79.13e15.83
85 45.8249.49
90.98=7.83
78.1116.30
78 16.6660077
S7. 59.1199959
70.7%&16.77
70.58=11.78
60024:18.49
76.93:13.23
6622214.667
56.44:00442
79.9
14.05416.17
68.73414.16
H
14.26419.22
28.38=12.29
27 46
41.06e15.69
2693312225
22. 91439
18. 75+7.55
21 21,27612,25
$2.97412.00
Table 32: Performance of Prompt Schemes with perturbed prompt 9Mean+95% CI Margin of All
Models). Averaged over 04mini and Qwen-2.5-72B. Bold orange Underlined blue Light purple highlights indicate best/sccond-bess/hhrrd--ee performance in each difficulty level.
Task
Difficulty
AL
AM
AS
EL
ES
GMal.
GMoL.
92.2613.71
15.24
91. 88:43.776
82.2847.11
82.1726.40
85.6824.64
74.5128.09
Original
M
83.44+5.82
46.03+13.79
79.4466.3 32
59.84+7 94
53.39+7.86
67 04.666666
56.01=10.77
48.27=6.14
7,46+2,72
48.23+6.57 57
20.74+3.70
15,56+2,76
26.95+3.29
23 23.46+5.37
E
93.4125.30
74.08
87.40&7.19
80.64+15.75
81.91+8.49
85.6048. 14
77.67+12.33
Perturbed
M
1..56
12.54214.98
83.93+7.28
61.40=11.35
57.9921 1.80
62.85=10.07
60.51=14.09
H
50.33:10.84
6.9726.66
50.02+13.24
22.52+8.83
24.93+11.77
19.2426.17
22.38+8.42
Table 33:
Performance of Serialization Formats with perturbed prompt (Meant95%
CI Margin
of All Models). Averaged over 04-mini and Qwen-2.5-72B.
Bold orange Underlined blue Light purple highlights indicate beaeecoodd--es/ttirr-bbbbbe
performance in each difficulty level.
Fine-Grained Results. Tables 34-37 break down results per model, revealing differential robustness characteristics:
04-mini exhibits high robustness. The closed-soure reasoning model shows remarkable stability across perturbations (Tables 34 and 35). For serialization formats, AL maintains 98.54% 97.89% (Fasy), 91.75% to 92.20% (Medium), and 54.24% to 56.08% (Hard), with minimal changes in ranking. For prompt schemes, the relative ordering remains nearly identical between conditions, with only minor absolute shifts (c.g., 0-Algorithm improves from 95.84% to 97.64% in Easy mode). This stability suggests that o4-mini's reasoning capabilities are relatively invariant to surface-level linguistic variation, consistent with its design for robust multi-step reasoning.
Dwen-2.5-728 shows greater sensitivity. The open-source model exhibits larger absolute performance shifts and wider confidence intervals under perturbation (Tables 36 and 37). For example, in serialization formats, performance on AS varies from 86.68% to 77.64% (Easy) and 47.74% to 52.59% (Hard), with substantially increased variance (e.g., Hard mode: 10.30 to 27.65). Similarly,
113

---

Published as a conference paper at ICLR 2026
prompt scheme performance shows notable fluctuation (e.g., CoT: 75.10% to 85.22% in Easy, 47.82% to 46.85% in Medium). However, crucially, the relative rankings remain consistent: AL and AS continue outperform other serializations, and Algorithn/CoT/Instuue remain competitive prompt schemes. This indicates that while open-source models may be more sensitive to phrasing variations, comparative conclusions about which representations work better robust.
Task
Difficulty
0-Algucithm
0-CuT
O-Instruct
0-Shot
Algurithm
CuT
Instruct
K-Shut
LTM
E
95 34
96.6633.4
96.5341.95
94.6612.
97.02:2.06
95/42:1.55
96.3222.15
93.90.2.26
91,80u2.51
Original
83.06e6300
80.8555
80.3427.04
84.0815944
28.558555
76.1226.47
72.7426.35
79.7366
78.1226.32
H
17.88e12..55 33
18077111:51
35. 13.334
39.31=14.47
26.98110881
26.26e7.95
21,48+7,00
31.85e11555
34.45e12.51
E
95.54
07.3941.95
95.20:4.17
95.92:2.5
94.71a3.32
93.37u2.53
95.4112.2
Penureed
M
89.42s4.41
8144124.99
75,2728.119
86, 86.2647.86
74.24a8.63
68.37a11.29
71.3329.88
78.26.699 9.9
78.7629.5 5.
11:16:19:67
50.55s15.98
27.23a13.65
34.28:10.70
35.71812.200
34.86:17.31
Table 34: Performance of Prompt Schemes with perturbed prompt (Mean:95% CI Margin of All
Models) o4-mini. Bold orange Underlined blue Light purple highlights indicate best/second- best/third-ber performance in each difficulty level.
Task
Difficulty
AL
AM
AS
ES
GMaL.
GMoL
E
98.5420.63
96.06#1.47
97.09=1.40
95.74+1.14
95,25=1.00
94.71=1.51
90.83+2.72
Original
91.75=1.49
74.71
87.20s4.0 62
74.92+2.89
69.10:2.66
79.6343.68
18.31
H
54.24=5.95
12.83#1.73
48.7248.25 78
27.4942.78
19.01=3.80
31.11=4.22
33.74+4.02
E
97.89=1.76
96, 15,99
97. 97.17+2.64 64
95.29+2.12
95. 05.20=2.90
94,20+2.88
93.39+2.76
Pertubedd
92.20=2.48
73.15+5.68
87.2247.44
76.96#6.46
67.16=7.90
72.84=6.6 68
77.57+5.61
H
56.08=10.23
13.51
47.74=7.90
31.36=12.84
28.6819.55
24.25
33.20=11..
Table 35: Performance Serialization Formats with perturbed prompt (Mean+95% Margin of
Models) on 04-mini. Bold orange Underlined blue Light purple highlights indicate best/second- best/third-bee performance in each difficulty level.
Dificulty
0-Algorithmn
Algorithm
E
73.5kee13.6
62.07e90.0
68 1011009).70 70
71.78e13.48 48
74.226741
25 1.e10.29
77.39e6.59
67. 064866888888
72.95e1 2,66
Original
95
48.03+17.71
50w16-46
40.73+19.69
15.31116.14
47.82+15.08
3.18
28 64+7.77
84+1843
H
17.77
25.6741487
23 07.133188
24.77444.43
2688e13.67
19.08e10.17
30.32e11.66
8. 68683.39
23 5.6995599
E
72.33a25 53
$2.11a15.51
71.71a14.31
61.91126.8
72.70116.55
18.22616.18
11:29.20
60.84
73.39121.22
Pertrred
M
52, 14+27 47
67.94+11.56 58
42.71+35.28
65.01225511
56,20+27.85
70.40:23.799
27. 18+14,46
58 70+25.48
II
28. 28.39133240 49
23.71a18.80
20.12226.21
31.56:26.38
26.22.22775
1616117
11.43610.14
1.0512.06
87.2220
Table 36: Performance of Prompt Schemes with perturbed prompt (Mean*95% CI Margin of All Models) on Qwen-2.5-72B, Bold orange Underlined blue Light purple highlights indicate sereeeoond--bss/hhrret performance in each difficulty level.
Summary. Our robustness analysis demonstrates that the main conclusions of GRAPHIOM remain stable under realistic linguistic perturbation. While absolute performance values shift measurably, confirming that perturbations introduce real variation rather than trivial paraphrases, the relative performance patterns across prompt schemes and serialization formats are preserved. Specifically, the finding that no single configuration works universally. but that certain serialization prompt combinations consistently outperform others, holds across both original and perturbed conditions. The differential sensitivity between models (o4-mini showing higher robustness than Qwen-2.5-72BB provides an additional dimension for understanding model capabilities. These results validate the reliability of our benchmark findings while highlighting that prompt perturbation represents valid and interesting dimension for future investigation. Importantly, our extensible framework design readily accommodates such extensions: future work could systematically incorporate perturbation as an additional evaluation axis alongside graph types, serialization formats, and prompt schemes, enabling deeper exploration of linguistic robustness in graph reasoning tasks.
114

---

Published as a conference paper at ICLR 2026
Task
Difficulty
AL
AM
AS
EL
ES
GMal.
GMoL
85,98=4.51
54,4248.
86.68+5.68
68.8246.28
69.09+2.97
76.65=3.28
58.20+3.92
Original
M
75.13+8.68
17.3543.83
71.6949 52
44.7626.41
37.67+4.27
54.4424.82
33.7043.43
42.30+9.52
2,08#0.76
17.74=10.30
13.9942.60
12,10+2.53
22.80=2.67
13.1742.23
87.65110.82
51.17421.9
77.64&11.01
47 47.66134.42
68.62211.31
75.94114.67
89,99L20.12
Perturbed
M
74.92+21.44 142
29.36#22.16
80.64=12.64
45 15.83466.54
47, 47.68+22.06 05
50.00= 17.76
38.57+23.07
43.85
1.25+2.45
52.59+27.65
13.67+9.54
21.18+21.98
13.59=8.54
11:57=671
Table 37: Performance of Serialization Formats with perturbed prompt (Mean+95% CI Margin of All Models) on Qwen-2.5-72B, Bold orange Underlined blue Light purple highlights indicate best/second-besstthir---ooos performance in each difficulty level.
THE USE OF LARGE LANGUAGE MODELS
We declare that we only use LLM to aid polish writing in this paper. Of course, we use LLMs to do inference our experiment since we need to evaluate them on GRAPHOMNL.
115


Reviews:
[{"review_id": "sHxPN2b1k9", "reviewer": "Reviewer_xfYv", "summary": "The paper introduces a large-scale benchmark (241,726 queries) for evaluating LLMs’ reasoning ability on graph-theoretic problems expressed in natural language. The framework systematically varies three dimensions—graph types, serialization formats, and prompt schemes—and assesses model performance across multiple difficulty levels and tasks (e.g., connectivity, BFS, cycle detection, triangle counting). The authors analyze open- and closed-source models, report detailed results, and propose a reinforcement learning–based optimization method (RL-Opt, RL-Scale) to reduce evaluation cost by 75% while maintaining accuracy.", "questions": "1. In Sec. 4.4; Table 4, you define Cost = e/E and Rate = m/M but provide no equation for the reward function or RL objective. Could you explicitly describe: the state/action/reward formulation of your RL process.\n\n2. The symbol m is reused for “edge count” in Sec. 2 (Graph Generators) and again for “accuracy” in Sec. 4.4 (Rate = m/M). please clarify notation consistency.\n\n3. Sec. 4.4 presents “RL-Scale” to assess scalability. What feedback signal or evaluation metric was used to balance computational cost and accuracy when adding new serialization factors?\n\n4. The paper systematically varies prompt schemes but all examples appear deterministic (Sec. 2; Appendix A.5). Did you test robustness under paraphrased or noisy prompts to evaluate linguistic invariance?\n\n5. Table 1 omits recent benchmarks in graph reasoning tasks (e.g., GraphWild in GCoder, 2025).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark jointly varies graph type, serialization, and prompt scheme—more complete than prior work. This design provides strong coverage for structured reasoning.\n\n2. Includes 241,726 queries across 7 graph generators and 9 prompt types; this ensures statistical reliability and generalizability.\n\n3. Both open-source (Llama‑3, Qwen‑3) and closed-source (GPT‑4o, Claude‑3.5) models are evaluated to ensure representativeness.\n\n4. Error analysis (Sec. 4.1, Result 3; Appendix E.3): Provides qualitative insights into misinterpretations (e.g., misunderstanding diameter definition), strengthening interpretability.", "weaknesses": "1. The benchmark provides empirical comparison but lacks analytical justification for observed behaviors—e.g., why serialization types yield specific effects. No direct evidence found in the manuscript.\n\n2. The reuse of m, M, e, E in RL metrics (Sec. 4.4) is not clearly linked back to earlier sections or consistent definitions; may confuse readers.\n\n3. No robustness check under prompt noise: All prompts assume fixed phrasing; real-world variance in natural language not examined. \n\n4. While model output tokens analyzed (Sec. 4.3), inference or hardware cost details (e.g., GPU hours) are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjZ0jURjFF", "reviewer": "Reviewer_nDLz", "summary": "The paper introduces GraphOmni, a comprehensive and extensible benchmark for evaluating large language models on graph-theoretic reasoning tasks expressed in natural language. It systematically varies three critical dimensions—graph type, serialization format, and prompt scheme—across six canonical tasksand three difficulty levels. Experiments reveal substantial variability: no single model or prompt-serialization combination consistently dominates. The authors further introduce a reinforcement learning–based optimizer that adaptively selects optimal factor combinations.", "questions": "Q1. How do the authors ensure that the comparison between LLM-generated outputs and the ground truth is accurate? Given that the answers are evaluated from natural-language responses, what mechanisms or verification procedures are used to avoid misparsing or misjudging correctness during evaluation?\n\nQ2. Could the authors provide a cross-factor analysis showing how different graph-theoretic tasks align with combinations of graph type, serialization format, and prompt scheme?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The benchmark framework is comprehensive, jointly considering three critical dimensions — graph types, serialization formats, and prompting schemes — to provide a multidimensional evaluation of LLMs’ graph reasoning abilities.\n\nS2. The paper explores the use of reinforcement learning to search for optimal combination strategies, achieving significant cost reduction while maintaining high performance.", "weaknesses": "W1. The benchmark mainly focuses on classical graph-theoretic problems, which limits its applicability to real-world large-scale or labeled graphs that involve complex attributes, heterogeneous structures, or task-specific supervision.\n\nW2. While GraphOmni reports accuracy across various tasks and factor combinations, the paper lacks a systematic analysis of how different task types interact with specific factor combinations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7iZKcDOO3V", "reviewer": "Reviewer_61MW", "summary": "The paper proposes GraphOmni, a comprehensive benchmark for LLMs on Graph-theoretic tasks.\nThe benchmark makes breakthroughs in the number of instances, the number of investigated schemes, and the number of serializations.\nKey findings from using GRAPHOMNI include: No single serialization format or prompt works best for all tasks; performance varies widely, and even state-of-the-art LLMs show significant room for improvement on graph reasoning tasks.\nMotivated by these findings, the authors also propose an RL-inspired selector to dynamically choose the best settings for a given task, thereby improving performance.", "questions": "See Weaknesses 1 & 2", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark is comprehensive in both scale and involved methods, providing a solid exploration in graph reasoning.\n2. The analysis is very solid and presents interesting viewpoints. The findings in Section 4.2 provide meaningful insights.\n3. The paper is very clearly written overall. The presentation is excellent and the logic is complete.", "weaknesses": "1. The significance of comparing LLMs on graph reasoning is questionable. The challenges LLMs face in graph reasoning seem largely constrained by context length, rather than being focused on core reasoning abilities like mathematical reasoning. There is a lack of discussion on what specific deep capabilities of LLMs the study of graph reasoning is meant to reflect.\n2. Expanding on point #1, a key characteristic of graph reasoning is that even humans often rely on external tools (e.g., writing code or drawing diagrams) to solve problems like finding shortest paths or Hamiltonian cycles. The inability of a human to solve these problems entirely \"in their head\" is not typically taken as a sign of deficient reasoning ability. Therefore, the practical significance of evaluating a single LLM's isolated reasoning capability on these tasks is limited.\n3. The study only considers graphs with up to 50 nodes, which is still too small in scale. \n4. The description of the RL-based method is too brief. While I understand space constraints, the current description fails to adequately explain in the main text: 1) the core limitation it addresses, 2) the experimental setting, and 3) it lacks a comprehensive comparison with baselines. These elements should be fully described in the main body to ensure the paper is self-contained and coherent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xf7rfe3xgN", "reviewer": "Reviewer_CVe5", "summary": "The paper introduces a comprehensive benchmark for evaluating LLMs' ability on graph-theoretic tasks. The evaluation is conducted on six graph-theoretic problems, which are encoded with different serialization formats and prompting schemes. The main results show that current LLMs have substantial room for improvement on graph-theoretic problems, and there is no consistently good serialization and prompting strategies among different tasks and models. The paper further introduces a reinforcement learning-based optimizer that adaptively selects factor combinations.", "questions": "1. Which main results are first found in this paper or are different from previous benchmarking papers?\n\n2. What's the real reason for the LLM's unsatisfying performance on graph-theoretic problems? In Result 3, it says representative categories of errors are commonly the misinterpretation of serialization formats and incorrect reasoning about graph-theoretic concepts. However, I think this can not be claimed as \"fundamental\" and can be solved by prompt engineering.\n\n3. Why do open-source models benefit from multi-shot exemplars, whereas they do not help closed-source models much?\n\n4. What is the complexity of the six tasks (e.g., time complexity related to node and edge numbers), and does the ranking align with the performance of LLMs?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The benchmark is comprehensive with multiple aspects of encoding strategies of graph-theoretic problems, covering both open-sourced and closed-sourced LLMs. The workload is heavy, and the experimental results are detailed.", "weaknesses": "1. Each part of the key components (Benchmark Tasks, Graph Types, Prompt Schemes, and Serialization Formats) was proposed and researched in previous works. Although this paper considers all four components, the experiments and analysis are still conducted individually, and more importantly, do not bring substantially **new** insights or findings.\n\n2. For most experiments, results are reported without **in-depth interpretation**, and therefore, the inspiration provided is limited.\n\n3. The **conclusions are limited to six graph-theoretic tasks** (Connectivity, Cycle detection, Diameter, BFS, Triangle, Shortest path). What are the differences among the six tasks, and do they measure different aspects of LLMs' graph reasoning ability? It's unknown whether the current conclusion changes when evaluating on other graph-theoretic tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)