SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although large language models achieved performance that is significantly better than random on simple graph reasoning tasks (§4.1), we hypothesize that LLMs may be able to reach the correct answer by leveraging certain spurious correlations. For example, on the connectivity task, since nodes with higher degrees are more frequently mentioned and they are more likely to be connected, LLMs might just be counting node mentions instead of actually finding paths. To this end, we design two special cases (chain and clique) for the connectivity task that are the exact opposite of the spurious correlation.
Figure 4: Model performance on the Hamilton path task with an increasing amount of exemplars. When more in-context exemplars are introduced, model performance is not trending up and is consistently lower than zero-shot prompting in both difficulty settings.
Chain We firstly generate a graph with  $k$  components, where each component is a chain. Formally,  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$  can be divided into subgraphs  $\mathcal{G}_1,\mathcal{G}_2,\ldots ,\mathcal{G}_k$ , where subgraph  $\mathcal{G}_i$  consists of a chain of nodes and edges  $v_{i1},e_{i1},v_{i2},e_{i2},\ldots ,e_{it_i},v_{it_i}$ . We then randomly select query node pairs that are at the two ends of a chain, i.e.  $v_{i1}$  and  $v_{it_i}$ . These nodes only have a degree of one but they are actually connected through the chain structure. We curate a chain dataset with 120 examples.
Clique We first generate a graph with  $k$  densely connected subgraphs. Formally,  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$  can be divided into subgraphs  $\mathcal{G}_1,\mathcal{G}_2,\ldots ,\mathcal{G}_k$ , where subgraph  $\mathcal{G}_i$  is generated by the random graph generator (§2.1) with a high edge probability  $p\in \{0.7,1.0\}$ . We then randomly select query node pairs with each pair in two different densely connected subgraphs,  $\mathcal{G}_i$  and  $\mathcal{G}_j$  ( $i\neq j$ ). These nodes are associated with a large number of edges and thus are frequently mentioned in the natural language prompt, but they belong to two different subgraphs and are not connected. We curate a clique dataset with 120 examples.
We evaluate LLMs with different prompting approaches on the two special cases and compare performances with the general connectivity task in Table 4. LLM performs much worse than ordinary
Standard Prompting
Figure 5: Overview of Build-a-Graph prompting and Algorithmic prompting, aiming to make LLMs better at reasoning with graphs by encouraging conceptual grounding and algorithmic reflection. Red underline indicates our proposed instructions while green background indicates LLMs' generation.
<in-context exemplar>
In an undirected graph, the nodes are numbered from 0 to 4, and the edges are: an edge between node 0 and node 4 with weight 4.
an edge between node 0 and node 3 with weight 3,
an edge between node 0 and node 1 with weight 3,
Q: Give the shortest path from node 0 to node 2.
Build-a-Graph Prompting
In an undirected graph, the nodes are numbered from 0 to 4, and the edges are: an edge between node 0 and node 4 with weight 4, ...
Let's construct a graph with the nodes and edges first.
A: All the paths from node 0 to node 2 are: 0,3,2 with a total weight of  $3 + 1 = 4$
0,1,4,2 with a total weight of  $3 + 4 + 2 = 9$  0,4,3,2 with a total weight of  $4 + 1 + 1 = 6$
The weight of path 0,3,2 is the smallest, so the shortest path from node 0 to node 2 is 0,3,2 with a total weight of 4.
Algorithmic Prompting
We can use a Depth-First Search (DFS) algorithm to find the shortest path between two given nodes in an undirected graph.
The basic idea is to start at one of the nodes and use DFS to explore all of its adjacent nodes. At each node you can keep track of the distance it takes to reach that node from the starting node
Once you have explored all the adjacent nodes, you can backtrack and pick the node which has the shortest distance to reach the destination node <in-context exemplar>
In an undirected graph, the nodes are numbered from 0 to 4, and the edges are:
an edge between node 0 and node 4 with weight 4,... Q: Give the shortest path from node 0 to node 2.
0.14.2 with a total weight of  $3 + 4 + 2 = 9$
0,4,3,2 with a total weight of  $4 + 1 + 1 = 6$ .
The weight of path 0,3,2 is the smallest, so the shortest path from node 0 to node 2 is 0,3,2 v total weight of 4.
cases with a performance drop of more than  $40\%$  across various settings. This suggests that LLMs are indeed (un)surprisingly vulnerable to spurious correlations in structured reasoning.
5 Making Language Models Better Graph Reasoners
5.1 Methodology
We propose two instruction-based prompting techniques that improve the graph reasoning ability of LLMs, which can be used together with in-context learning and chain-of-thought prompting.
Build-a-Graph Prompting (BAG) We hypothesize that it might be helpful to map the textual descriptions of graphs to grounded conceptual spaces [Patel et al., 2021] before tackling the specific problem, i.e. visualizing the graph structure on an implicit mental sketchpad. To this end, we propose to use instruction-based prompting by appending "Let's construct a graph with the nodes and edges first." after the textual description of the graph is explicitly given. We envision this straightforward instruction would provide LLMs with the buffer zone to digest graph information, map them to grounded conceptual spaces, and better prepare for the incoming query.
Algorithmic Prompting We hypothesize that in order to generate sound and accurate solutions, LLMs would benefit from revisiting and reciting the relevant algorithm for a given task [Sun et al., 2023]. To this end, we propose to first prepend the algorithm details before the in-context examples by adding "We can use a Depth-First Search (DFS) algorithm ..." for the shortest path task. We similarly provide the algorithm descriptions of other graph reasoning tasks in their respective prompts. We envision algorithmic prompting as empowering LLMs with a general understanding of how to solve the problem before actually solving it.
We illustrate our proposed build-a-graph prompting and algorithmic prompting in Figure 5. In the following, we use these two approaches together with chain-of-thought prompting and evaluate on three representative tasks in the NLGraph benchmark with varying difficulties.
Table 5 shows the results of our proposed prompting methods on three representative tasks with varying complexity. On the two easier tasks, cycle and shortest path, our instruction-based prompting methods resulted in a  $3.07\%$  to  $16.85\%$  performance gain across the two tasks. However, on the more complex task, Hamilton path, the two natural language instructions are largely ineffective. The result suggests that the two simple instruction-based prompting techniques successfully improve LLMs' graph reasoning abilities on relatively easy graph reasoning tasks, while how to make LLMs better
Method	Cycle	Shortest Path	Hamilton Path
Easy	Medium	Hard	Avg.	Easy	Hard	Easy (PC)	Hard (PC)	Avg.	Easy	Hard	Avg.
COT	84.67	63.33	53.25	66.75	63.89	29.50	76.84	35.79	51.51	40.00	8.00	24.00
COT+BAG	86.00	69.33	62.00	72.44	67.78	33.50	79.20	42.56	55.76	38.67	6.00	22.34
COT+ALGORITHM	77.33	74.00	64.00	71.78	63.89	28.00	76.06	38.70	51.66	36.67	7.50	22.09
Table 5: Model performance on the connectivity, cycle, and shortest path tasks with our proposed BAG and ALGORITHM prompting. On the two easier tasks, cycle and shortest path, our instruction-based prompting techniques resulted in a  $3.07\%$  to  $16.85\%$  performance gain across the two tasks. More complex graph reasoning tasks such as Hamilton path remain an open research question.
at reasoning on complex graph reasoning problems remains an open research question. We further explore different variants of the instructions in Appendix D. The NLGraph benchmark also empowers the evaluation of future instructions and solutions towards making LLMs better graph reasoners.
LLMs for tasks with implicit graphical structures. Previous works explored the use of LLMs on tasks with implicit graph structures: Huang et al. [2022] find that LLMs have the ability to ground high-level tasks (e.g. "make breakfast") to a set of actionable steps (e.g. "open fridge") in structured synthetic environments. Valmeekam et al. [2022] explore the possibility of using LLMs for commonsense planning. In theory-of-mind reasoning, Adhikari et al. [2020], Ammanabrolu and Riedl [2021], Sclar et al. [2023] maintain and update structured world representations as the world state change to operate in interactive and situated environments. In structured commonsense reasoning, where LLMs are given a natural language input and then asked to generate a graph such as an event graph [Tandon et al., 2019], a reasoning-graph [Madaan et al., 2021] or an argument explanation graph [Saha et al., 2021], Madaan et al. [2022] find that a code language model (CODEX) with tasks framed as code generation tasks outperforms other strong LMs. In multi-hop question answering or knowledge probing, LLMs implicitly find connections and paths among a vast network of entities and concepts [Creswell et al., 2023, Yu et al., 2022, Zhang et al., 2022a, He et al., 2021]. Recently, Chen et al. [2023] explore the potential of LLMs on the graph node classification task. Together these works demonstrate that LLMs are increasingly adopted for tasks and settings with implicit graphs and structures, while whether LLMs are robust at graph reasoning remains underexplored and may hinder the progress of these structure-aware applications. In this work, we propose the NLGraph benchmark as a comprehensive testbed towards evaluating the graph reasoning abilities of large language models.
LLMs for few-shot reasoning. There is a long line of work on evaluating LLMs' reasoning ability in an in-context learning setting, including arithmetic reasoning, logical reasoning, common sense reasoning, and more. Particularly, simple math problem datasets such as AQUA Ling et al. [2017], GSM8K [Cobbe et al., 2021], and SVAMP [Patel et al., 2021] are used for evaluating arithmetic reasoning [He-Yueya et al., 2023, Touvron et al., 2023, Shi et al., 2023]. Welleck et al. [2021] developed NaturalProofs, a multi-domain dataset for studying mathematical reasoning in natural language, while Welleck et al. [2022] study LLMs' ability to generate the next step in mathematical proof and generate full proofs. LLMs have also been evaluated on logical reasoning tasks, including symbolic tasks like Coin Flip and Last Letter Concatenation [Wei et al., 2022] and Logic Grid Puzzle on the BIG-BENCH [Srivastava et al., 2023]. Commonsense reasoning datasets [Talmor et al., 2019, Geva et al., 2021] are also proposed to evaluate large language models. Most relevant to our work, various proposals to evaluate and augment the algorithm reasoning abilities of large language models are explored [Zhou et al., 2022, Lee and Kim, 2023, Zelikman et al., 2023, Liu et al., 2023]. In this work, we focus on evaluating and enhancing LLMs on graph-based reasoning and algorithmic tasks inspired by the increasing usage of LLMs in tasks with implicit graphs and structures.
In this work, we investigate whether LLMs are capable of explicit graph reasoning, i.e., solving graph algorithm problems in natural language, across various problem categories and prompting techniques. To this end, we curate the NLGraph benchmark, a comprehensive test bed of graph-based reasoning in natural language, with 29,370 problems covering eight tasks with varying complexity. By evaluating
LLMs and prompting approaches on the NLGraph benchmark, we find that 1) LLMs do possess preliminary graph reasoning abilities, 2) the benefit of advanced prompting and in-context learning may diminish on complex reasoning tasks, while 3) LLMs are brittle to spurious correlations in problem settings. We then propose Build-a-Graph and Algorithmic Prompting, two simple instruction-based approaches that bring notable performance gains across multiple tasks. Improving LLMs' graph reasoning abilities on complex and nuanced graph reasoning tasks remains an open research question, and we encourage future work to develop upon our proposed NLGraph benchmark.
We would like to thank the reviewers, the area chair, Jiacheng Liu, Minnan Luo, and members of the UW NLP Group for their comments and feedback. This material is funded by the DARPA Grant under Contract No. HR001120C0124. We also gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, the Alfred P. Sloan Foundation Fellowship, and NSF grant No. IIS2203097. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily state or reflect those of the United States Government or any agency thereof.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
Jacob Andreas. Language models as agent models. In *Findings of the Association for Computational Linguistics: EMNLP* 2022, pages 5769-5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022-findings-emnlp.423.
Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems, 33:3045-3057, 2020.
Prithviraj Ammanabrolu and Mark Riedl. Learning knowledge graph-based world models of textual environments. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=o24k_XfIe6_.
Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. WIQA: A dataset for "what if..." reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6076-6085, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1629. URL https://aclanthology.org/D19-1629.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1384-1403, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.emnlp-main.90.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=3Pf3Wg6o-A4.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL), 2021.
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066, 2022.
Soochan Lee and Gunhee Kim. Recursion of thought: A divide-and-conquer approach to multicontext reasoning with language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 623-658, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023-findings-acl.40. URL https://aclanthology.org/2023.findings-acl.40.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. Parsel: A (de-) compositional framework for algorithmic reasoning with language models. arXiv preprint arXiv:2212.10561, 2023.
Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. Code execution with pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4984-4999, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.308. URL https://aclanthology.org/2023-findings-acl.308.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022b.
NLP tasks and graph tasks alignment. Originally designed for processing language, LLMs are increasingly adopted for NLP tasks with structural implications. For instance, structured commonsense reasoning is similar to topological sort in that they both need to find a solution satisfying diversified constraints between entities. In the belief state graph in theory-of-mind [Adhikari et al., 2020, Ammanabrolu and Riedl, 2021], the final task is to determine the connectivity of nodes, which corresponds to the connectivity task. In multi-hop question answering [Creswell et al., 2023], LLMs implicitly find connections and paths among a vast network of entities and concepts, which resembles the shortest path task and connectivity task. However, we acknowledge that not all graph reasoning tasks in the NLGraph benchmark are clearly aligned with certain applications, thus advanced graph reasoning tasks in the NLGraph benchmark could also be viewed as a test of language model reasoning ability.
The NLGraph benchmark envisioned as reasoning benchmark. In addition to evaluating graph reasoning abilities, the NLGraph benchmark can also be viewed as a structural benchmark to evaluate language model reasoning. While ample LM reasoning datasets exist [Cobbe et al., 2021, Patel et al., 2021, Talmor et al., 2019], it is difficult to avoid train-test overlap since problems such as grade school math [Cobbe et al., 2021] and commonsense reasoning [Talmor et al., 2019] might be readily available in pretraining corpora, making it less convincing for evaluating reasoning abilities. On the contrary, the synthetic graph reasoning problems and their answers are highly unlikely to have exact matches in the training corpora, which makes NLGraph a more robust benchmark towards evaluating language model reasoning.
Fine-tuning to elicit graph reasoning abilities. To further study the graphical thinking abilities of LLMs, we envision fine-tuning as a possible direction for future work. We plan to fine-tune language models on the chain-of-thought reasoning process towards graph-based tasks on both single- and multi-task settings to see if fine-tuning might lead to enhanced graph reasoning abilities.
Tasks in NLGraph benchmark are not complete. While we have incorporated eight graph reasoning tasks with varying complexity in the NLGraph benchmark, there are many more important graph algorithms that test different levels of graph reasoning abilities. It might also be interesting to see how LLMs perform on other graph tasks such as finding the Eulerian path, the minimum spanning tree, and the cut-edge.
Language models that we evaluate are not complete. We only consider four black-box LLMs, TEXT-DAVINCI-003, CODE-DAVINCI-002, GPT-3.5-TURBO, and GPT-4 in our experiments. Since we will make the NLGraph benchmark publicly available, we leave it to future work on evaluating the graph reasoning abilities of other open-source LLMs.
Limited dataset size. Due to monetary costs, we only evaluate LLMs on the standard version of the NLGraph benchmark which has 5,902 problems throughout the paper. We believe evaluating LLMs on the extended version of the NLGraph benchmark, with  $5\mathrm{x}$  more problems, may bring stronger proof to the findings we present.
Methods for improving graph reasoning abilities. The two prompting methods we present are simple instruction-based approaches, which work on easy graph reasoning problems to varying extents. However, on more complex graph reasoning problems, as the overall graph reasoning abilities of LLMs are limited, the instruction-based methods seem to have only marginal effects. In future work, we hope to investigate methods such as asking LLMs to generate and execute code, or simulating steps of algorithmic solutions while maintaining the state variables.
C NLGraph Details
In Table 6, we provide the details of the NLGraph benchmark including edge probabilities and specific values of parameters mentioned in §2.2.
Subset	Connect.	Cycle	Topo. Sort	Shortest Path	Maximum Flow	Bipartite Graph	Hamilton Path	GNNs
# EASY	p: 0.3, 0.7, 1.0	m: 1-4	p: 0.3, 0.5, 0.7	p: 0.5, 0.7, 0.9; w: 1-4; ℓ: 2-6	p: 0.2, 0.3; c: 1-10	p: 0.3-0.7	p: 0.4, 0.6	p: 0.4; ℓ: 1
# MEDIUM	p: 0.3, 0.7, 1.0	m: 1-4	p: 0.3, 0.5, 0.7	/	/	/	/	/
# HARD	p: 0.3, 0.7	m: 1-4	p: 0.3, 0.5	p: 0.2, 0.25; w: 1-10; ℓ: 2-6	p: 0.25; c: 1-20	p: 0.2-0.6	p: 0.4, 0.6	p: 0.2; ℓ: 1
Table 6: Details of the NLGraph benchmark.  $p$  denotes the edge probability. Other characters have the same meaning mentioned in §2.2.
Method	Shortest Path
Easy	Hard	Easy (PC)	Hard (PC)
BAG	67.78	33.50	79.20	42.56
BAG-DOT	65.00	29.00	75.55	37.18
ALGORITHMIC-DOT	66.67	31.00	78.27	39.68
INSTRUCTION #1	66.22	32.50	78.18	40.32
INSTRUCTION #2	67.78	28.50	77.32	37.06
INSTRUCTION #3	67.78	30.50	79.56	37.89
Table 7: The results of variants of instructions on the shortest path task. The BAG prompting approach achieves the best performance across three of the four settings.
D Analysis
Instruction variants To further study the effect of BAG prompting and ALGORITHMIC prompting, we replace the instructions with only a sequence of dots (... equal to the number of characters in the instruction. This is to investigate whether the improved performance is attributed to the natural language instructions or simply increased computation. We also replace the BAG prompting with three other instructions not directly related to graph problems, specifically, Let's think step by step (INSTRUCTION 1), Examine each detail carefully (INSTRUCTION 2), Think it through systematically (INSTRUCTION 3). As illustrated in Table 7, the BAG prompting method generally outperforms the instruction variants, indicating the effectiveness of this method.
Graph definition variants We study how replacing the abstract descriptions of graphs with real-world objects would impact model performance. Specifically, for the shortest path task, we change all the "node"s into "city"s, "edge"s into "road"s, and "weight"s into "distance"s. As shown in Table 9, the model performance improves on the easy subset but drops on the hard subset. While language models are indeed sensitive to the specific instantiations of the graph description, the results are mixed as to which is easier or harder.
Figure 6: Model performance for the shortest path task with different numbers of nodes on the shortest path. The performance steadily drops when the path length increases.
Figure 7: Model performance for the cycle task with varied exemplar difficulty. When the in-context exemplars are based on more difficult problems, the performance becomes worse.
Method	Cycle	Shortest Path	Hamilton Path
Easy	Medium	Hard	Easy	Hard	Easy (PC)	Hard (PC)	Easy	Hard
RANDOM	50.00	50.00	50.00	50.00	50.00	6.07	6.69	0.00	0.00
FEW-SHOT	58.00	50.00	49.75	39.44	28.50	55.10	37.63	36.67	7.00
CoT	60.67	66.83	69.00	41.11	/	45.00	/	50.00	12.00
CoT+SC	59.33	52.67	53.75	42.22	/	49.26	/	34.00	5.00
Table 8: The results of CODE-DAVINCI-002 on the cycle, shortest path, and Hamilton path task.
Method	Shortest Path
Easy	Hard	Easy (PC)	Hard (PC)
RANDOM	50.00	50.00	6.07	6.69
ZERO-SHOT	29.40	14.00	47.53	20.13
FEW-SHOT	41.11	20.50	58.78	29.28
CoT	73.33	28.00	82.57	35.50
0-CoT	6.67	2.00	65.40	51.95
CoT+SC	72.78	27.00	83.50	35.19