SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Heng Wang\*1, Shangbin Feng\*2, Tianxing He\*2, Zhaoxuan Tan\*3, Xiaochuang Han\*2, Yulia Tsvetkov2
$^{1}$ Xi'an Jiaotong University  $^{2}$ University of Washington  $^{3}$ University of Notre Dame
wh2213210554@stu.xjtu.edu.cn, shangbin@cs.washington.edu
Large language models (LLMs) are increasingly adopted for a variety of tasks with implicit graphical structures, such as planning in robotics, multi-hop question answering or knowledge probing, structured commonsense reasoning, and more. While LLMs have advanced the state-of-the-art on these tasks with structure implications, whether LLMs could explicitly process textual descriptions of graphs and structures, map them to grounded conceptual spaces, and perform structured operations remains underexplored. To this end, we propose NLGraph (Natural Language Graph), a comprehensive benchmark of graph-based problem solving designed in natural language. NLGraph contains 29,370 problems, covering eight graph reasoning tasks with varying complexity from simple tasks such as connectivity and shortest path up to complex problems such as maximum flow and simulating graph neural networks. We evaluate LLMs (GPT-3/4) with various prompting approaches on the NLGraph benchmark and find that 1) language models do demonstrate preliminary graph reasoning abilities, 2) the benefit of advanced prompting and in-context learning diminishes on more complex graph problems, while 3) LLMs are also (un)surprisingly brittle in the face of spurious correlations in graph and problem settings. We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems. Build-a-Graph and Algorithmic prompting improve the performance of LLMs on NLGraph by  $3.07\%$  to  $16.85\%$  across multiple tasks and settings, while how to solve the most complicated graph reasoning tasks in our setup with language models remains an open research question. The NLGraph benchmark and evaluation code are available at https://github.com/Arthur-Heng/NLGraph.
Originally designed for textual data, large language models (LLMs) are increasingly leveraged for tasks beyond language processing. In robotics and planning, LLMs are adopted to guide agents through structured environments [Huang et al., 2022, Andreas, 2022]. In theory-of-mind reasoning, LLMs are required to maintain and update local and global graphs that reflect the beliefs of different characters [Adhikari et al., 2020, Ammanabrolu and Riedl, 2021]. In structured commonsense reasoning, LLMs are expected to generate graph-based action plans to achieve objectives with diversified prerequisites [Tandon et al., 2019, Madaan et al., 2022]. In multi-hop question answering, LLMs implicitly find connections and paths among a vast network of entities and concepts [Creswell et al., 2023]. Together these works demonstrate that LLMs are widely adopted for tasks with implicit graphical structures while achieving preliminary success. However, one underlying yet crucial question remains underexplored: Can LLMs reason with graphs? More concretely, are LLMs capable of mapping textual descriptions of graphs and structures to grounded conceptual spaces and solving
graph algorithm problems explicitly with natural language? The answer to this question has profound implications for large language model applications with implicit graphs and structures, the reasoning ability of LLMs in advanced and graph-based settings, and more.
To this end, we propose the Natural Language Graph (NLGraph) benchmark, a comprehensive testbed of graph and structured reasoning designed for language models and in natural language. NLGraph contains a total of 29,370 problems, covering eight graph reasoning tasks with varying complexity from intuitively simple tasks such as connectivity, cycle, and shortest path to more complex problems such as topological sort, maximum flow, bipartite graph matching, Hamilton path, and simulating graph neural networks. We control for problem difficulty through generated graph size, network sparsity, numeric range, and more, presenting easy, medium, and hard subsets in each distinct graph reasoning task to enable fine-grained analysis. In addition to using exact match accuracy as a standard metric, we also design several partial credit solutions for various graph reasoning tasks.
With the NLGraph benchmark, we evaluate whether various large language models [Brown et al., 2020, Ouyang et al., 2022, Bubeck et al., 2023] could perform graph-based reasoning and whether different prompting techniques [Brown et al., 2020, Wei et al., 2022, Kojima et al., 2022, Zhou et al., 2023, Wang et al., 2023] improve the graph reasoning abilities of large language models. Extensive experiments on the NLGraph benchmark demonstrate that:
1. LLMs do possess preliminary graph reasoning abilities. Specifically, large language models demonstrate an impressive level of performance that is  $37.33\%$  to  $57.82\%$  above the random baseline on simple graph reasoning tasks such as connectivity, cycle, and shortest path. With chain-of-thought prompting, LLMs could generate intermediate steps that are sound and accurate while further improving task performance.
2. The benefit of advanced prompting methods diminishes with complex problems. On one hand, chain-of-thought [Wei et al., 2022], least-to-most [Zhou et al., 2023], and self-consistency [Wang et al., 2023] successfully enhance the graph reasoning abilities of LLMs on simple tasks such as cycle and shortest path. On the other hand, these approaches are mostly ineffective, even counterproductive in certain settings, on more complex graph reasoning problems such as topological sort and Hamilton path.
3. Learning from examples did not happen on complex graph reasoning problems. While in-context learning is widely credited for teaching LLMs to learn from examples [Brown et al., 2020], its benefit on more advanced graph reasoning tasks is unclear: Few-shot in-context learning fails to improve over zero-shot prompting across multiple tasks, while increasing the number of exemplars may even be counterproductive for tasks such as Hamilton path.
4. LLMs are (un)surprisingly brittle to spurious correlations in problem settings. We find that in two special cases in the connectivity task (chain and clique), LLMs perform much worse than the general dataset with a performance drop of more than  $40\%$  across various settings. This indicates that large language models are implicitly relying on certain spurious correlations (for example, use node mention frequency to determine connectivity), falling short of performing robust structured reasoning in graph-based contexts.
To improve large language models as better graph reasoners, we propose two instruction-based prompting approaches to better elicit the graph reasoning abilities of large language models. Builda-Graph prompting encourages LLMs to map the textual descriptions of graphs and structures to grounded conceptual spaces [Patel and Pavlick, 2022] before tackling the specific problem through a one-sentence instruction, while Algorithmic prompting instructs LLMs to revisit the algorithmic steps for a given task before learning from in-context exemplars. Experiments demonstrate that builda-graph and algorithmic prompting successfully empower LLMs to better tackle graph reasoning problems, resulting in  $3.07\%$  to  $16.85\%$  performance gains across multiple tasks, while the most complicated graph reasoning problems remain an open research question.
2 The NLGraph Benchmark
To examine whether language models are capable of reasoning with graphs and structures, we curate and propose the Natural Language Graph (NLGraph) benchmark. Specifically, we first employ a random graph generator to generate graphs and structures while controlling for the network size, graph sparsity, and more. We then adopt the generated graphs as bases to synthetically generate problems for eight graph-based reasoning tasks with varying algorithmic difficulties. We control for
1. Connectivity
Determine if there is a path between two nodes in the graph. Note that (i,j) means that node i and node j are connected with an undirected edge. Graph: (0,1) (1,2) (3,4) (4,5)
Q: Is there a path between node 1 and node 4?
2. Cycle
In an undirected graph, (i,j) means that node i and node j are connected with an undirected edge.
The nodes are numbered from 0 to 5, and the edges are: (3,4) (3,5) (1,0) (2,5) (2,0).
0: Is there a cycle in this graph?
3. Topological Sort
In a directed graph with 5 nodes numbered from 0 to 4: node 0 should be visited before node
node 0 should be visited before node 4.
Q: Can all the nodes be visited? Give the solution.
4. Shortest Path
In an undirected graph, the nodes are numbered from 0 to 4, and the edges are: an edge between node 0 and node 1 with weight 2 ...
Q: Give the shortest path from node 0 to node 4.
6. Maximum Flow
In a directed graph, the nodes are numbered from 0 to 3, and the edges are:
an edge from node 1 to node 0 with capacity 10,
an edge from node 0 to node 2 with capacity 6,
an edge from node 2 to node 3 with capacity 4.
Q: What is the maximum flow from node 1 to node 3?
6. Bipartite Graph Matching
There are 4 job applicants numbered from 0 to 3, and 5 jobs numbered from 0 to 4. Each applicant is interested in some of the jobs. Each job can only accept one applicant and a job applicant can be appointed for only one job.
Applicant 0 is interested in job 4, ...
Q: Find an assignment of jobs to applicants in such that the maximum number of applicants find the job they are interested in.
7. Hamilton Path
Figure 1: Overview of the NLGraph Benchmark, featuring eight tasks with varying complexity. We show an intuitive figure representing each task along with the example natural language prompts being passed to the LLMs.
In an undirected graph,  $(i,j)$  means that node i and node j are connected with an undirected edge.
The nodes are numbered from 0 to 4, and the edges are: (4,2) (0,4) (4,3) (0,1) (0,2) (4,1) (2,3)
Q: Is there a path in this graph that visits every node exactly once? If yes, give the path. Note that in a path, adjacent nodes must be connected with edges.
8. GNN
In an undirected graph, the nodes are numbered from 0 to 4, and every node has an embedding. (i,j) means that node i and node j are connected with an undirected edge. Embeddings: node 0: [1,1], ...
The edges are: (0,1)
In a simple graph convolution layer, each node's embedding is updated by the sum of its neighbors' embeddings.
Q: What's the embedding of each node after one layer of simple graph convolution layer?
problem difficulty within each of the eight tasks, resulting in easy, medium, and hard subsets in each graph reasoning task to enable difficulty scaling and fine-grained analysis.
2.1 Random Graph Generator
We first employ a general-purpose random graph generator to generate base graphs while using the number of nodes and graph density to control for complexity. Formally, to generate a graph  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$  where  $\mathcal{V}$  and  $\mathcal{E}$  denote the set of nodes and edges, we specify the number of nodes  $n$  thus  $\mathcal{V} = \{v_{1},v_{2},\ldots ,v_{n}\}$ , and  $|\mathcal{V}| = n$ . We then specify the edge probability  $p$  such that all the edges are generated according to  $P(e_{ij}\in \mathcal{E}) = p$ , where  $e_{ij}$  is an edge from  $v_{i}$  to  $v_{j}$ . The edges could be directed or undirected depending on the task. We use varying  $n$  and  $p$  values to control for the complexity of the random graph structure. Building on top of these generated base graphs, we also adopt graph edits and other difficulty control factors tailored for each task.
2.2 Tasks
Armed with the general-purpose random graph generator, we adapt the synthetically generated graphs to eight graph reasoning tasks with varying complexity and describe the problem setups in natural language. Specifically, we first generate easy, medium, and hard subsets of base graphs for each task by controlling for node amount and graph sparsity. We then adapt or edit the base graphs for each task and design queries accordingly. We present an overview of the NLGraph benchmark with specific natural language instructions in Figure 1.
- Task 1: Connectivity In an undirected graph  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$ , two nodes  $u$  and  $v$  are connected if there exists a sequence of edges from node  $u$  to node  $v$  in  $\mathcal{E}$ . We randomly select two nodes in the base graphs  $u,v\in \mathcal{V}$  to ask whether node  $u$  and node  $v$  are connected with a true/false question. We retain a balanced set of questions where half of the node pairs are connected and the other half are not connected by discarding additional questions.
- Task 2: Cycle In an undirected graph  $\mathcal{G} = \{\mathcal{V}, \mathcal{E}\}$ , a cycle is a non-empty trail  $(e_1, e_2, \ldots, e_n)$  with a node sequence  $(v_1, v_2, \ldots, v_n, v_1)$ . We present an undirected graph  $\mathcal{G} = \{\mathcal{V}, \mathcal{E}\}$  to ask whether there exists a cycle through true/false questions. We retain base graphs without cycles as the False subset, while we randomly add edges to these base graphs to generate graphs with cycles as the True subset. We retain a balanced set of cyclic and noncyclic graphs in the dataset.
- Task 3: Topological Sort A topological sort of a directed graph is a linear ordering of its nodes such that for every directed edge  $(u, v)$  from node  $u$  to node  $v$ ,  $u$  comes before  $v$  in the ordering.
Subset	Connect.	Cycle	Topo. Sort	Shortest Path	Max. Flow	Bipartite Graph	Hamilton Path	GNNs
# EASY	352 / 730	150 / 300	180 / 360	180 / 360	150 / 300	300 / 600	150 / 300	100 / 200
SPEC.	n: 5-10	n: 5-10	n: 5-10	n: 5-10	n: 5-10	n: 6-20	n: 5-10	n: 5-8
# MEDIUM	1,200 / 8,580	600 / 1,800	150 / 1,350	/	/	/	/	/
SPEC.	n: 11-25	n: 11-25	n: 11-25	/	/	/	/	/
# HARD	680 / 7,090	400 / 2,000	200 / 1,200	200 / 1,200	200 / 1,200	210 / 1,260	200 / 600	140 / 840
SPEC.	n: 26-35	n: 26-35	n: 26-35	n: 11-20	n: 11-20	n: 17-33	n: 11-20	n: 9-15
Table 1: Statistics of the NLGraph benchmark. We use A / B to indicate that there are A and B problems in the standard and extended set of NLGraph. SPEC. denotes difficulty specifications.
The task is to find a valid topological sort given a directed graph and there could be multiple valid solutions. We ask LLMs to generate a valid topological sort for the given directed graph and employ an external program to examine its correctness.
2.3 Benchmark Statistics
Using the above methodology, we generate the NLGraph benchmark with 5,902 problems in a standard version and 29,370 problems in an extended version. Intuitively easier tasks, including connectivity, cycle, and topological sort problems, are further divided into easy, medium, and hard subsets based on graph size, sparsity, among other difficulty control factors. More algorithmically advanced tasks, including shortest path, maximum flow, bipartite graph matching, Hamilton path,
Method	Connectivity	Cycle	Shortest Path
Easy	Medium	Hard	Avg.	Easy	Medium	Hard	Avg.	Easy	Hard	Easy (PC)	Hard (PC)	Avg.
RANDOM	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00	6.07	6.69	14.73	13.81	17.81
ZERO-SHOT	83.81	72.75	63.38	71.31	50.00	50.00	50.00	50.00	29.40	21.00	46.00	26.76	30.79
FEW-SHOT	93.75	83.83	76.61	84.73	80.00	70.00	61.00	70.33	31.11	26.00	49.19	35.73	35.51
CoT	94.32	82.17	77.21	84.57	84.67	63.33	53.25	66.75	63.89	29.50	76.84	35.79	51.51
0-CoT	79.55	65.83	68.53	71.30	55.33	57.67	49.00	54.00	8.89	7.50	62.39	43.95	32.03
CoT+SC	93.18	84.50	82.79	86.82	82.00	63.67	53.50	66.39	68.89	29.00	80.25	38.47	54.15
Table 2: Model performance on the connectivity, cycle, and shortest path tasks. PC denotes partial credit. Large language models with CoT or CoT+SC prompting greatly outperforms the random baseline by  $37.33\%$  to  $57.82\%$ , indicating that LLMs have preliminary graph reasoning abilities.
and graph neural networks problems, are divided into easy and hard subsets. We present benchmark statistics in Table 1. Accuracy (whether the true/false answer is correct, whether the proposed solution is valid) is the default evaluation metric, while tasks 4, task 5, and task 8 have additional partial-credit metrics aforementioned. We envision NLGraph as a comprehensive testbed for graph and structured reasoning for large language models.
3 Experimental Settings
Based on the NLGraph benchmark, we aim to investigate whether language models can solve graph algorithm problems in natural language by evaluating large language models and different prompting approaches.
Baselines We adopt a wide range of prompting approaches as baselines. Specifically, zero-shot prompting (ZERO-SHOT), few-shot in-context learning (FEW-SHOT) [Brown et al., 2020], chain-of-thought prompting (CoT) [Wei et al., 2022], zero-shot chain-of-thought (0-CoT) [Kojima et al., 2022], least-to-most (LTM) [Zhou et al., 2023], and self-consistency (SC) [Wang et al., 2023] are leveraged to tackle various graph reasoning tasks in the NLGraph benchmark.
We also adopt a RANDOM baseline: for true/false questions such as connectivity and cycle, we use RANDOM to denote a baseline that randomly selects an answer from true and false with an expected accuracy of  $50\%$ ; For the shortest path task, RANDOM denotes a baseline that randomly selects a valid path between the query node pair. For the maximum flow task, RANDOM denotes a baseline that randomly selects a value between 0 and the sum of all the edges' capacities. The performance comparison between different prompting techniques and the RANDOM baseline could indicate whether LLMs are capable of performing graph reasoning instead of giving randomly generated answers.
Models and Settings We use TEXT-DAVINCI-003 as the default large language model, while we also evaluate three additional LLMs (GPT-3.5-TURBO, CODE-DAVINCI-002, and GPT-4), presenting results in Appendix E. For all baselines except self-consistency, we set temperature  $\tau = 0$ ; For self-consistency prompting, we sample five chain-of-thought responses with temperature  $\tau = 0.7$ . For few-shot prompting techniques (i.e., FEW-SHOT, COT, LTM, and COT+SC), the input prompt includes  $k$  exemplars selected from the extended version before the problem of interest. For the connectivity task and cycle task, we set  $k$  to 4, for the GNN task, we set  $k$  to 1 due to the context size limit, while for other tasks  $k$  is 5. We evaluate LLMs and various prompting techniques on the standard set of NLGraph due to monetary costs, while we encourage future research to leverage the extended version for enhanced evaluation.
4.1 LLMs Have (Preliminary) Graph Reasoning Abilities
We first find that on intuitively simple graph reasoning tasks, large language models achieve impressive performance and demonstrate preliminary graph reasoning abilities. As demonstrated in Table 2, LLM performance on the connectivity, cycle, and shortest path tasks is significantly better than the RANDOM baseline, indicating that LLMs are not giving random answers and they do have preliminary graph reasoning abilities. In the first two tasks with true/false questions, LLM performance is  $37.33\%$  to  $57.82\%$  higher than random with CoT or CoT+SC prompting. The performance is also impressive
Figure 2: (left) Model performance on the topological sort task.  $\mathrm{COT}$ , LTM, and self-consistency are mostly ineffective on this problem. (right) Model performance on the maximum flow task. FEW-SHOT prompting outperforms  $\mathrm{COT} + \mathrm{SC}$  prompting on both easy and hard subsets, suggesting that LLMs fall short of generating valid intermediate steps to solve the more complex graph reasoning problem. Together these results demonstrate that advanced prompting is ineffective for advanced graph reasoning.
in the ZERO-SHOT setting, as the accuracy is  $33.81\%$  and  $23.33\%$  higher on the connectivity and shortest path task than random even without any exemplars and chain-of-thought reasoning. Though the shortest path is intuitively harder than the first two true/false tasks since it requires generating the specific shortest path in the response, LLM combined with CoT and  $\mathrm{CoT + SC}$  achieves an accuracy  $22.81\%$  to  $62.83\%$  higher than RANDOM. Together these results demonstrate that large language models do possess preliminary abilities to process graphs and structures in input texts.
4.2 Mixed Results with Advanced Prompting
Table 2 shows that advanced prompting techniques such as chain-of-thought [Wei et al., 2022] and self-consistency [Wang et al., 2023] successfully improve performance on simple graph reasoning tasks. For the task of simulating graph neural networks (Table 3), chain-of-thought also significantly improves model performance. However, these approaches can be ineffective, even counterproductive on more complex graph reasoning tasks. From the results on the topological sort and maximum flow task (Figure 2), we observe that  $\mathrm{COT}$ ,  $\mathrm{COT + SC}$ , and LTM prompting generally underperform FEWSHOT prompting. We believe this may be be
Method	PC (↑)	Acc (↑)	RE (↓)
ZERO-SHOT	13.61	0.00	20.04
FEW-SHOT	20.04	0.00	37.83
CoT	64.55	31.00	14.34
0-CoT	13.85	0.00	44.55
CoT+SC	63.92	28.00	13.28
Table 3: Model performance on the task of simulating graph neural networks. PC and RE are two partial credit metrics introduced in §2.2. Chain-of-thought prompting significantly improves the model performance across all metrics.
cause LLMs fail to learn the correct way to generate intermediate steps in the face of complex graph reasoning tasks. This casts doubt on the generalizability of COT, LTM, and self-consistency to more advanced graph reasoning problems.
4.3 In-Context Learning Can be Counterproductive
Although in-context learning is widely attributed to teaching LLMs to learn from in-context exemplars [Brown et al., 2020], we observe that few-shot in-context learning fails to improve LLM performance over complex graph reasoning problems. For tasks such as Hamilton path and bipartite graph matching (Figure 3), zero-shot prompting generally outperforms few-shot learning with in-context exemplars. The results suggest that LLMs fail to learn from the in-context exemplars when the problem involves hundreds of graph-based reasoning steps. In-context exemplars might even distract large language models, evident in that few-shot learning underperforms zero-shot learning by  $1.00\%$  to  $10.48\%$  on Hamilton path and bipartite graph matching. We further study the correlation between the number of exemplars and model performance on the Hamilton path task. As illustrated in Figure 4, when the number of exemplars increases, model performance is not trending up, and the performance is consistently lower than zero-shot prompting. These results further suggest that in-context learning could be counterproductive in complex structured reasoning.
Figure 3: (left) Model performance on the Hamilton path task. ZERO-SHOT prompting consistently outperforms all other prompting techniques. (right) Model performance on the bipartite graph matching task. The effect of in-context learning and advanced prompting is also mostly marginal in this complex graph reasoning problem. Together these results demonstrate that in-context learning can be counterproductive in advanced graph reasoning problems.
Dataset	ZERO-SHOT	FEW-SHOT	COT	0-CoT	COT+SC	Avg.
GENERAL	74.67	83.33	85.33	66.00	82.67	78.40
CHAIN	51.67 (-23.00)	45.00 (-35.33)	40.83 (-44.50)	92.50 (+26.50)	44.17 (-38.50)	54.83 (-23.57)
CLIQUE	60.83 (-13.84)	73.33 (-10.00)	85.00 (-0.33)	52.50 (-13.50)	83.33 (+0.66)	71.00 (-7.40)
Table 4: Model performance on the chain and clique subset of the connectivity task. Large language models indeed rely on spurious correlations in problem settings, evident in the reduced performance on the two special cases compared to the general connectivity task.
4.4 LLMs are (Un)surprisingly Brittle