Title: Can Language Models Solve Graph Problems in Natural Language?
Venue: NeurIPS 2023
Accepted: yes
Keywords: large language models

graph reasoning

structured reasoning
Abstract: Large language models (LLMs) are increasingly adopted for a variety of tasks with implicit graphical structures, such as planning in robotics, multi-hop question answering or knowledge probing, structured commonsense reasoning, and more. While LLMs have advanced the state-of-the-art on these tasks with structure implications, whether LLMs could explicitly process textual descriptions of graphs and structures, map them to grounded conceptual spaces, and perform structured operations remains underexplored. To this end, we propose NLGraph (Natural Language Graph), a comprehensive benchmark of graph-based problem solving designed in natural language. NLGraph contains 29,370 problems, covering eight graph reasoning tasks with varying complexity from simple tasks such as connectivity and shortest path up to complex problems such as maximum flow and simulating graph neural networks. We evaluate LLMs (GPT-3/4) with various prompting approaches on the NLGraph benchmark and find that 1) language models do demonstrate preliminary graph reasoning abilities, 2) the benefit of advanced prompting and in-context learning diminishes on more complex graph problems, while 3) LLMs are also (un)surprisingly brittle in the face of spurious correlations in graph and problem settings. We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems. Build-a-Graph and Algorithmic prompting improve the performance of LLMs on NLGraph by 3.07% to 16.85% across multiple tasks and settings, while how to solve the most complicated graph reasoning tasks in our setup with language models remains an open research question.

Can Language Models Solve Graph Problems in Natural Language?

Abstract

Large language models (LLMs) are increasingly adopted for a variety of tasks with implicit graphical structures , such as planning in robotics, multi-hop question answering or knowledge probing, structured commonsense reasoning, and more. While LLMs have advanced the state-of-the-art on these tasks with structure implications, whether LLMs could explicitly process textual descriptions of graphs and structures, map them to grounded conceptual spaces, and perform structured operations remains underexplored. To this end, we propose NLGraph (Natural Language Graph), a comprehensive benchmark of graph-based problem solving designed in natural language. NLGraph contains 29,370 problems, covering eight graph reasoning tasks with varying complexity from simple tasks such as connectivity and shortest path up to complex problems such as maximum flow and simulating graph neural networks. We evaluate LLMs (GPT-3/4) with various prompting approaches on the NLGraph benchmark and find that 1) language models do demonstrate preliminary graph reasoning abilities, 2) the benefit of advanced prompting and in-context learning diminishes on more complex graph problems, while 3) LLMs are also (un)surprisingly brittle in the face of spurious correlations in graph and problem settings. We then propose Build-a-Graph Prompting and Algorithmic Prompting, two instruction-based approaches to enhance LLMs in solving natural language graph problems. Build-a-Graph and Algorithmic prompting improve the performance of LLMs on NLGraph by 3.07% to 16.85% across multiple tasks and settings, while how to solve the most complicated graph reasoning tasks in our setup with language models remains an open research question. The NLGraph benchmark and evaluation code are available at https://github.com/Arthur-Heng/NLGraph.

1 Introduction

Originally designed for textual data, large language models (LLMs) are increasingly leveraged for tasks beyond language processing. In robotics and planning, LLMs are adopted to guide agents through structured environments [Huang et al., 2022, Andreas, 2022]. In theory-of-mind reasoning, LLMs are required to maintain and update local and global graphs that reflect the beliefs of different characters [Adhikari et al., 2020, Ammanabrolu and Riedl, 2021]. In structured commonsense reasoning, LLMs are expected to generate graph-based action plans to achieve objectives with diversified prerequisites [Tandon et al., 2019, Madaan et al., 2022]. In multi-hop question answering, LLMs implicitly find connections and paths among a vast network of entities and concepts [Creswell et al., 2023]. Together these works demonstrate that LLMs are widely adopted for tasks with implicit graphical structures while achieving preliminary success. However, one underlying yet crucial question remains underexplored: Can LLMs reason with graphs? More concretely, are LLMs capable of mapping textual descriptions of graphs and structures to grounded conceptual spaces and solving

graph algorithm problems explicitly with natural language? The answer to this question has profound implications for large language model applications with implicit graphs and structures, the reasoning ability of LLMs in advanced and graph-based settings, and more.

To this end, we propose the Natural Language Graph (NLGraph) benchmark, a comprehensive testbed of graph and structured reasoning designed for language models and in natural language. NLGraph contains a total of 29,370 problems, covering eight graph reasoning tasks with varying complexity from intuitively simple tasks such as connectivity , cycle , and shortest path to more complex problems such as topological sort , maximum flow , bipartite graph matching , Hamilton path , and simulating graph neural networks . We control for problem difficulty through generated graph size, network sparsity, numeric range, and more, presenting easy, medium, and hard subsets in each distinct graph reasoning task to enable fine-grained analysis. In addition to using exact match accuracy as a standard metric, we also design several partial credit solutions for various graph reasoning tasks.

With the NLGraph benchmark, we evaluate whether various large language models [Brown et al., 2020, Ouyang et al., 2022, Bubeck et al., 2023] could perform graph-based reasoning and whether different prompting techniques [Brown et al., 2020, Wei et al., 2022, Kojima et al., 2022, Zhou et al., 2023, Wang et al., 2023] improve the graph reasoning abilities of large language models. Extensive experiments on the NLGraph benchmark demonstrate that:

LLMs do possess preliminary graph reasoning abilities. Specifically, large language models demonstrate an impressive level of performance that is 37.33% to 57.82% above the random baseline on simple graph reasoning tasks such as connectivity, cycle, and shortest path. With chain-of-thought prompting, LLMs could generate intermediate steps that are sound and accurate while further improving task performance.

The benefit of advanced prompting methods diminishes with complex problems. On one hand, chain-of-thought [Wei et al., 2022], least-to-most [Zhou et al., 2023], and self-consistency [Wang et al., 2023] successfully enhance the graph reasoning abilities of LLMs on simple tasks such as cycle and shortest path. On the other hand, these approaches are mostly ineffective, even counterproductive in certain settings, on more complex graph reasoning problems such as topological sort and Hamilton path.

Learning from examples did not happen on complex graph reasoning problems. While in-context learning is widely credited for teaching LLMs to learn from examples [Brown et al., 2020], its benefit on more advanced graph reasoning tasks is unclear: Few-shot in-context learning fails to improve over zero-shot prompting across multiple tasks, while increasing the number of exemplars may even be counterproductive for tasks such as Hamilton path.

LLMs are (un)surprisingly brittle to spurious correlations in problem settings. We find that in two special cases in the connectivity task (chain and clique), LLMs perform much worse than the general dataset with a performance drop of more than 40% across various settings. This indicates that large language models are implicitly relying on certain spurious correlations (for example, use node mention frequency to determine connectivity), falling short of performing robust structured reasoning in graph-based contexts.

To improve large language models as better graph reasoners, we propose two instruction-based prompting approaches to better elicit the graph reasoning abilities of large language models. Builda-Graph prompting encourages LLMs to map the textual descriptions of graphs and structures to grounded conceptual spaces [Patel and Pavlick, 2022] before tackling the specific problem through a one-sentence instruction, while Algorithmic prompting instructs LLMs to revisit the algorithmic steps for a given task before learning from in-context exemplars. Experiments demonstrate that builda-graph and algorithmic prompting successfully empower LLMs to better tackle graph reasoning problems, resulting in 3.07% to 16.85% performance gains across multiple tasks, while the most complicated graph reasoning problems remain an open research question.

2 The NLGraph Benchmark

To examine whether language models are capable of reasoning with graphs and structures, we curate and propose the Natural Language Graph (NLGraph) benchmark. Specifically, we first employ a random graph generator to generate graphs and structures while controlling for the network size, graph sparsity, and more. We then adopt the generated graphs as bases to synthetically generate problems for eight graph-based reasoning tasks with varying algorithmic difficulties. We control for

Determine if there is a path between

two nodes in the graph. Note that (i,j)

means that node i and node j are

connected with an undirected edge.

Graph: (0,1) (1,2) (3,4) (4,5)

Q

: Is there a path between node 1 and

1. Connectivity

In an undirected graph, (i,j) means that

node i and node j are connected with an

undirected edge.

The nodes are numbered from 0 to 5,

and the edges are: (3,4) (3,5) (1,0) (2,5)

(2,0)

Q

: Is there a cycle in this graph?

2. Cycle

In a directed graph with 5 nodes

numbered from 0 to 4:

node 0 should be visited before node

4, ...

Q

: Can all the nodes be visited? Give the

solution.

In an undirected graph, the nodes are

numbered from 0 to 4, and the edges are:

an edge between node 0 and node 1 with

weight 2, ...

Q

: Give the shortest path from node 0 to

4. Shortest Path

In a directed graph, the nodes are

numbered from 0 to 3, and the edges

are:

an edge from node 1 to node 0 with

capacity 10,

an edge from node 0 to node 2 with

capacity 6,

an edge from node 2 to node 3 with

capacity 4.

Q

: What is the maximum flow from node

1 to node 3?

5. Maximum Flow

There are 4 job applicants numbered

from 0 to 3, and 5 jobs numbered from

0 to 4. Each applicant is interested in

some of the jobs. Each job can only

accept one applicant and a job

applicant can be appointed for only one

job.

Applicant 0 is interested in job 4, ...

Q

: Find an assignment of jobs to

applicants in such that the maximum

number of applicants find the job they

are interested in.

6. Bipartite Graph Matching

In an undirected graph, (i,j) means that

node i and node j are connected with

an undirected edge.

The nodes are numbered from 0 to 4,

and the edges are: (4,2) (0,4) (4,3) (0,1)

(0,2) (4,1) (2,3)

Q

: Is there a path in this graph that

visits every node exactly once? If yes,

give the path. Note that in a path,

adjacent nodes must be connected

with edges.

2

1

1

1

2

3

2

0

1

4

2

1

1

1

2

3

2

0

1

4

3

2

0

1

4

3

2

0

1

4

3

5

2

0

1

4

3

5

2

0

1

4

0

1

4

5

3

2

0

1

4

5

3

2

10

6

4

3

2

0

1

10

6

4

3

2

0

1

job applicants

jobs

3

2

0

1

3

2

0

1

4

job applicants

jobs

3

2

0

1

3

2

0

1

4

In an undirected graph, the nodes are

numbered from 0 to 4, and every node has an

embedding. (i,j) means that node i and node j

are connected with an undirected edge.

Embeddings: node 0: [1,1],

…

The edges are: (0,1) ...

In a simple graph convolution layer, each

node's embedding is updated by the sum of

its neighbors' embeddings.

Q

: What's the embedding of each node after

one layer of simple graph convolution layer?

8. GNN

3

2

0

1

4

1

1

0

1

0

1

1

1

0

1

3

2

0

1

4

1

1

0

1

0

1

1

1

0

1

3

2

0

1

4

3

2

0

1

4

problem difficulty within each of the eight tasks, resulting in easy, medium, and hard subsets in each graph reasoning task to enable difficulty scaling and fine-grained analysis.

2.1 Random Graph Generator

We first employ a general-purpose random graph generator to generate base graphs while using the number of nodes and graph density to control for complexity. Formally, to generate a graph G = {V , E} where V and E denote the set of nodes and edges, we specify the number of nodes n , thus V = { v 1 , v 2 , . . . , v n } , and |V| = n . We then specify the edge probability p such that all the edges are generated according to P ( e ij ∈ E ) = p , where e ij is an edge from v i to v j . The edges could be directed or undirected depending on the task. We use varying n and p values to control for the complexity of the random graph structure. Building on top of these generated base graphs, we also adopt graph edits and other difficulty control factors tailored for each task.

2.2 Tasks

Armed with the general-purpose random graph generator, we adapt the synthetically generated graphs to eight graph reasoning tasks with varying complexity and describe the problem setups in natural language. Specifically, we first generate easy, medium, and hard subsets of base graphs for each task by controlling for node amount and graph sparsity. We then adapt or edit the base graphs for each task and design queries accordingly. We present an overview of the NLGraph benchmark with specific natural language instructions in Figure 1.

Task 1: Connectivity In an undirected graph G = {V , E} , two nodes u and v are connected if there exists a sequence of edges from node u to node v in E . We randomly select two nodes in the base graphs u, v ∈ V to ask whether node u and node v are connected with a true/false question. We retain a balanced set of questions where half of the node pairs are connected and the other half are not connected by discarding additional questions.

Task 2: Cycle In an undirected graph G = {V , E} , a cycle is a non-empty trail ( e 1 , e 2 , . . . , e n ) with a node sequence ( v 1 , v 2 , . . . , v n , v 1 ) . We present an undirected graph G = {V , E} to ask whether there exists a cycle through true/false questions. We retain base graphs without cycles as the False subset, while we randomly add edges to these base graphs to generate graphs with cycles as the True subset. We retain a balanced set of cyclic and noncyclic graphs in the dataset.

Task 3: Topological Sort A topological sort of a directed graph is a linear ordering of its nodes such that for every directed edge ( u, v ) from node u to node v , u comes before v in the ordering.

The task is to find a valid topological sort given a directed graph and there could be multiple valid solutions. We ask LLMs to generate a valid topological sort for the given directed graph and employ an external program to examine its correctness.

Task 4: Shortest Path The shortest path between two nodes is the path with the sum of edge weights minimized. Given an undirected graph G = {V , E} , a positive weight w for each edge, and two nodes u and v , the task is to find the shortest path between node u and node v and its corresponding path length. We filter the generated base graphs by specifying that the number of nodes on the correct shortest path is as least ℓ , where ℓ is chosen from ℓ min to ℓ max for each question to sever as an additional difficulty control measure. We adopt two metrics: exact match , where the LLM solution is a valid path and optimal, and partial credit , where the LLM solution is valid and is the rank i -th shortest among all the possible paths (i.e. the number of shorter paths plus one). The partial credit score for each problem can be formulated as PC = 1 / rank i .

Task 5: Maximum Flow Let G = {V , E} be a network with two nodes s, t ∈ V being the source and the sink. Each edge is associated with a capacity c , the maximum amount of flow that can pass through the edge. We ask LLMs to generate a plan to route as much flow as possible from the source to the sink. We evaluate in both exact match with the optimal plan and partial credit for this

task, where partial credit can be formulated as PC = { t/s, if t ≤ s 0 , if t > s , where s is the flow value under the optimal plan, and t is the flow value of the solution generated by LLMs.

Task 6: Bipartite Graph Matching In an undirected graph G = {V , E} , a matching is a set of edges without common nodes. A bipartite graph is a graph whose nodes can be divided into two disjoint sets U and V , and in each set no nodes are adjacent to each other. Given a bipartite graph, the task is to find the matching that maximizes the number of edges. We use an external program to evaluate whether the solution generated by LLMs is valid and optimal.

Task 7: Hamilton Path In an undirected graph, a Hamilton path is a path that visits every node exactly once. Given an undirected graph G = {V , E} , the task is to find a valid Hamilton path. We filter generated base graphs to ensure that at least one valid Hamilton path exists and use an external program to evaluate the LLM solution.

Task 8: Graph Neural Networks Given an undirected graph G = {V , E} , and a two-dimension node embedding x i for each node, the task is to perform ℓ layers of message passing, i.e. to update the node embedding with the sum of all the neighbors' embeddings. Formally, x ( ℓ +1) i = ∑ j ∈N i x ( ℓ ) j where N i denotes the neighborhood of node i and ( ℓ ) denotes the ℓ -th layer. We use an exact match with the correct node embeddings and two types of partial credits for this task. Specifically, the first partial credit is the percentage of the nodes whose embedding is correct (PC), and the second is the average of all the values' relative errors for the standard answer (RE). The relative error is formulated as RE = | x -y | max( x,y ) , where x is the value generated by LLMs and y is the value in the standard answer, averaged across all embedding dimensions.

2.3 Benchmark Statistics

Using the above methodology, we generate the NLGraph benchmark with 5,902 problems in a standard version and 29,370 problems in an extended version. Intuitively easier tasks, including connectivity, cycle, and topological sort problems, are further divided into easy, medium, and hard subsets based on graph size, sparsity, among other difficulty control factors. More algorithmically advanced tasks, including shortest path, maximum flow, bipartite graph matching, Hamilton path,

and graph neural networks problems, are divided into easy and hard subsets. We present benchmark statistics in Table 1. Accuracy (whether the true/false answer is correct, whether the proposed solution is valid) is the default evaluation metric, while tasks 4, task 5, and task 8 have additional partial-credit metrics aforementioned. We envision NLGraph as a comprehensive testbed for graph and structured reasoning for large language models.

3 Experimental Settings

Based on the NLGraph benchmark, we aim to investigate whether language models can solve graph algorithm problems in natural language by evaluating large language models and different prompting approaches.

Baselines We adopt a wide range of prompting approaches as baselines. Specifically, zero-shot prompting (ZERO-SHOT), few-shot in-context learning (FEW-SHOT) [Brown et al., 2020], chain-ofthought prompting (COT) [Wei et al., 2022], zero-shot chain-of-thought (0-COT) [Kojima et al., 2022], least-to-most (LTM) [Zhou et al., 2023], and self-consistency (SC) [Wang et al., 2023] are leveraged to tackle various graph reasoning tasks in the NLGraph benchmark.

We also adopt a RANDOM baseline: for true/false questions such as connectivity and cycle, we use RANDOM to denote a baseline that randomly selects an answer from true and false with an expected accuracy of 50%; For the shortest path task, RANDOM denotes a baseline that randomly selects a valid path between the query node pair. For the maximum flow task, RANDOM denotes a baseline that randomly selects a value between 0 and the sum of all the edges' capacities. The performance comparison between different prompting techniques and the RANDOM baseline could indicate whether LLMs are capable of performing graph reasoning instead of giving randomly generated answers.

Models and Settings We use TEXT-DAVINCI-003 as the default large language model, while we also evaluate three additional LLMs (GPT-3.5-TURBO, CODE-DAVINCI-002, and GPT-4), presenting results in Appendix E. For all baselines except self-consistency, we set temperature τ = 0 ; For self-consistency prompting, we sample five chain-of-thought responses with temperature τ = 0 . 7 . For few-shot prompting techniques (i.e., FEW-SHOT, COT, LTM, and COT+SC), the input prompt includes k exemplars selected from the extended version before the problem of interest. For the connectivity task and cycle task, we set k to 4, for the GNN task, we set k to 1 due to the context size limit, while for other tasks k is 5. We evaluate LLMs and various prompting techniques on the standard set of NLGraph due to monetary costs, while we encourage future research to leverage the extended version for enhanced evaluation.

4 Results

4.1 LLMs Have (Preliminary) Graph Reasoning Abilities

Wefirst find that on intuitively simple graph reasoning tasks, large language models achieve impressive performance and demonstrate preliminary graph reasoning abilities. As demonstrated in Table 2, LLM performance on the connectivity, cycle, and shortest path tasks is significantly better than the RANDOM baseline, indicating that LLMs are not giving random answers and they do have preliminary graph reasoning abilities. In the first two tasks with true/false questions, LLM performance is 37.33% to 57.82% higher than random with COT or COT+SC prompting. The performance is also impressive

zero-shot

few-shot

0-CoT

CoT

LTM

0

25

50

75

Acc

51.7

70.6

36.1

54.4

50.5

37.8

easy

zero-shot

few-shot

0-CoT

CoT

0

10

20

Acc

15.3

12.0

9.3

10.0

4.0

easy

zero-shot

few-shot

0-CoT

CoT

LTM

0

10

20

Acc

5.1

18.4

1.1

0.9

1.6

0.2

topological sort

medium

zero-shot

few-shot

0-CoT

CoT

0

5

10

Acc

2.5

2.0

2.0

4.0

1.0

maximum flow

hard

in the ZERO-SHOT setting, as the accuracy is 33.81% and 23.33% higher on the connectivity and shortest path task than random even without any exemplars and chain-of-thought reasoning. Though the shortest path is intuitively harder than the first two true/false tasks since it requires generating the specific shortest path in the response, LLM combined with COT and COT+SC achieves an accuracy 22.81% to 62.83% higher than RANDOM. Together these results demonstrate that large language models do possess preliminary abilities to process graphs and structures in input texts.

4.2 Mixed Results with Advanced Prompting

Table 2 shows that advanced prompting techniques such as chain-of-thought [Wei et al., 2022] and self-consistency [Wang et al., 2023] successfully improve performance on simple graph reasoning tasks. For the task of simulating graph neural networks (Table 3), chainof-thought also significantly improves model performance. However, these approaches can be ineffective, even counterproductive on more complex graph reasoning tasks. From the results on the topological sort and maximum flow task (Figure 2), we observe that COT, COT+SC, and LTM prompting generally underperform FEWSHOT prompting. We believe this may be be-

cause LLMs fail to learn the correct way to generate intermediate steps in the face of complex graph reasoning tasks. This casts doubt on the generalizability of COT, LTM, and self-consistency to more advanced graph reasoning problems.

4.3 In-Context Learning Can be Counterproductive

Although in-context learning is widely attributed to teaching LLMs to learn from in-context exemplars [Brown et al., 2020], we observe that few-shot in-context learning fails to improve LLM performance over complex graph reasoning problems. For tasks such as Hamilton path and bipartite graph matching (Figure 3), zero-shot prompting generally outperforms few-shot learning with in-context exemplars. The results suggest that LLMs fail to learn from the in-context exemplars when the problem involves hundreds of graph-based reasoning steps. In-context exemplars might even distract large language models, evident in that few-shot learning underperforms zero-shot learning by 1.00% to 10.48% on Hamilton path and bipartite graph matching. We further study the correlation between the number of exemplars and model performance on the Hamilton path task. As illustrated in Figure 4, when the number of exemplars increases, model performance is not trending up, and the performance is consistently lower than zero-shot prompting. These results further suggest that in-context learning could be counterproductive in complex structured reasoning.

zero-shot

few-shot

0-CoT

CoT

0

20

40

60

Acc

49.3

46.7

14.7

40.0

34.7

20.0

17.5

4.0

8.0

8.5

easy

hard

zero-shot

few-shot

0-CoT

CoT

0

20

40

60

Acc

46.0

45.0

40.7

48.3

47.0

27.6

17.1

27.1

13.3

11.4

bipartite graph matching

easy

hard

4.4 LLMs are (Un)surprisingly Brittle

Although large language models achieved performance that is significantly better than random on simple graph reasoning tasks (§4.1), we hypothesize that LLMs may be able to reach the correct answer by leveraging certain spurious correlations. For example, on the connectivity task, since nodes with higher degrees are more frequently mentioned and they are more likely to be connected, LLMs might just be counting node mentions instead of actually finding paths. To this end, we design two special cases (chain and clique) for the connectivity task that are the exact opposite of the spurious correlation.

0

2

4

8

12

16

# exemplar

0

20

40

60

Acc

easy

hard

Chain We firstly generate a graph with k components, where each components is a chain. Formally, G = {V , E} can be divided into subgraphs G 1 , G 2 , . . . , G k , where subgraph G i consists of a chain of nodes and edges v i 1 , e i 1 , v i 2 , e i 2 , . . . , e it i , v it i . We then randomly select query node pairs that are at the two ends of a chain, i.e. v i 1 and v it i . These nodes only have a degree of one but they are actually connected through the chain structure. We curate a chain dataset with 120 examples.

̸

Clique We first generate a graph with k densely connected subgraphs. Formally, G = {V , E} can be divided into subgraphs G 1 , G 2 , . . . , G k , where subgraph G i is generated by the random graph generator (§2.1) with a high edge probability p ∈ { 0 . 7 , 1 . 0 } . We then randomly select query node pairs with each pair in two different densely connected subgraphs, G i and G j ( i = j ). These nodes are associated with a large number of edges and thus are frequently mentioned in the natural language prompt, but they belong to two different subgraphs and are not connected. We curate a clique dataset with 120 examples.

We evaluate LLMs with different prompting approaches on the two special cases and compare performances with the general connectivity task in Table 4. LLM performs much worse than ordinary

Algorithmic Prompting

Build-a-Graph Prompting

<in-context exemplar>

In an undirected graph, the nodes are

numbered from 0 to 4, and the edges are:

an edge between node 0 and node 4 with

weight 4,

…

Let's construct a graph with the nodes and

edges first.

Q

: Give the shortest path from node 0 to

node 2.

A

:  All the paths from node 0 to node 2 are:

0,3,2 with a total weight of 3 + 1 = 4,

0,1,4,2 with a total weight of 3 + 4 + 2 = 9,

0,4,3,2 with a total weight of 4 + 1 + 1 = 6.

The weight of path 0,3,2 is the smallest, so

the shortest path from node 0 to node 2 is

0,3,2 with a total weight of 4.

We can use a Depth-First Search (DFS) algorithm to

find the shortest path between two given nodes in an

undirected graph.

The basic idea is to start at one of the nodes and use

DFS to explore all of its adjacent nodes. At each node,

you can keep track of the distance it takes to reach

that node from the starting node.

Once you have explored all the adjacent nodes, you

can backtrack and pick the node which has the

shortest distance to reach the destination node.

<in-context exemplar>

In an undirected graph, the nodes are numbered from

0 to 4, and the edges are:

an edge between node 0 and node 4 with weight 4, ...

Q

: Give the shortest path from node 0 to node 2.

A

:  All the paths from node 0 to node 2 are:

0,3,2 with a total weight of 3 + 1 = 4,

0,1,4,2 with a total weight of 3 + 4 + 2 = 9,

0,4,3,2 with a total weight of 4 + 1 + 1 = 6.

The weight of path 0,3,2 is the smallest, so the

shortest path from node 0 to node 2 is 0,3,2 with a

total weight of 4.

<in-context exemplar>

In an undirected graph, the nodes are

numbered from 0 to 4, and the edges are:

an edge between node 0 and node 4 with

weight 4,

an edge between node 0 and node 3 with

weight 3,

an edge between node 0 and node 1 with

weight 3,

...

Q

: Give the shortest path from node 0 to

node 2.

Standard Prompting

3

4

3

2

0

1

4

3

4

2

1

1

3

4

3

2

0

1

4

3

4

2

1

1

<in-context exemplar>

In an undirected graph, the nodes are

numbered from 0 to 4, and the edges are:

an edge between node 0 and node 4 with

weight 4,

an edge between node 0 and node 3 with

weight 3,

an edge between node 0 and node 1 with

weight 3,

...

Q

: Give the shortest path from node 0 to

node 2.

Standard Prompting

3

4

3

2

0

1

4

3

4

2

1

1

cases with a performance drop of more than 40% across various settings. This suggests that LLMs are indeed (un)surprisingly vulnerable to spurious correlations in structured reasoning.

5 Making Language Models Better Graph Reasoners

5.1 Methodology

We propose two instruction-based prompting techniques that improve the graph reasoning ability of LLMs, which can be used together with in-context learning and chain-of-thought prompting.

Build-a-Graph Prompting (BAG) We hypothesize that it might be helpful to map the textual descriptions of graphs to grounded conceptual spaces [Patel et al., 2021] before tackling the specific problem, i.e. visualizing the graph structure on an implicit mental sketchpad. To this end, we propose to use instruction-based prompting by appending ' Let's construct a graph with the nodes and edges first. ' after the textual description of the graph is explicitly given. We envision this straightforward instruction would provide LLMs with the buffer zone to digest graph information, map them to grounded conceptual spaces, and better prepare for the incoming query.

Algorithmic Prompting We hypothesize that in order to generate sound and accurate solutions, LLMs would benefit from revisiting and reciting the relevant algorithm for a given task [Sun et al., 2023]. To this end, we propose to first prepend the algorithm details before the in-context examples by adding 'We can use a Depth-First Search (DFS) algorithm . . . ' for the shortest path task. We similarly provide the algorithm descriptions of other graph reasoning tasks in their respective prompts. We envision algorithmic prompting as empowering LLMs with a general understanding of how to solve the problem before actually solving it.

We illustrate our proposed build-a-graph prompting and algorithmic prompting in Figure 5. In the following, we use these two approaches together with chain-of-thought prompting and evaluate on three representative tasks in the NLGraph benchmark with varying difficulties.

5.2 Results

Table 5 shows the results of our proposed prompting methods on three representative tasks with varying complexity. On the two easier tasks, cycle and shortest path, our instruction-based prompting methods resulted in a 3.07% to 16.85% performance gain across the two tasks. However, on the more complex task, Hamilton path, the two natural language instructions are largely ineffective. The result suggests that the two simple instruction-based prompting techniques successfully improve LLMs' graph reasoning abilities on relatively easy graph reasoning tasks, while how to make LLMs better

at reasoning on complex graph reasoning problems remains an open research question. We further explore different variants of the instructions in Appendix D. The NLGraph benchmark also empowers the evaluation of future instructions and solutions towards making LLMs better graph reasoners.

6 Related Work

LLMs for tasks with implicit graphical structures. Previous works explored the use of LLMs on tasks with implicit graph structures: Huang et al. [2022] find that LLMs have the ability to ground high-level tasks (e.g. "make breakfast") to a set of actionable steps (e.g. "open fridge") in structured synthetic environments. Valmeekam et al. [2022] explore the possibility of using LLMs for commonsense planning. In theory-of-mind reasoning, Adhikari et al. [2020], Ammanabrolu and Riedl [2021], Sclar et al. [2023] maintain and update structured world representations as the world state change to operate in interactive and situated environments. In structured commonsense reasoning, where LLMs are given a natural language input and then asked to generate a graph such as an event graph [Tandon et al., 2019], a reasoning-graph [Madaan et al., 2021] or an argument explanation graph [Saha et al., 2021], Madaan et al. [2022] find that a code language model (CODEX) with tasks framed as code generation tasks outperforms other strong LMs. In multi-hop question answering or knowledge probing, LLMs implicitly find connections and paths among a vast network of entities and concepts [Creswell et al., 2023, Yu et al., 2022, Zhang et al., 2022a, He et al., 2021]. Recently, Chen et al. [2023] explore the potential of LLMs on the graph node classification task. Together these works demonstrate that LLMs are increasingly adopted for tasks and settings with implicit graphs and structures, while whether LLMs are robust at graph reasoning remains underexplored and may hinder the progress of these structure-aware applications. In this work, we propose the NLGraph benchmark as a comprehensive testbed towards evaluating the graph reasoning abilities of large language models.

LLMs for few-shot reasoning. There is a long line of work on evaluating LLMs' reasoning ability in an in-context learning setting, including arithmetic reasoning, logical reasoning, common sense reasoning, and more. Particularly, simple math problem datasets such as AQUA Ling et al. [2017], GSM8K [Cobbe et al., 2021], and SVAMP [Patel et al., 2021] are used for evaluating arithmetic reasoning [He-Yueya et al., 2023, Touvron et al., 2023, Shi et al., 2023]. Welleck et al. [2021] developed NaturalProofs, a multi-domain dataset for studying mathematical reasoning in natural language, while Welleck et al. [2022] study LLMs' ability to generate the next step in mathematical proof and generate full proofs. LLMs have also been evaluated on logical reasoning tasks, including symbolic tasks like Coin Flip and Last Letter Concatenation [Wei et al., 2022] and Logic Grid Puzzle on the BIG-BENCH [Srivastava et al., 2023]. Commonsense reasoning datasets [Talmor et al., 2019, Geva et al., 2021] are also proposed to evaluate large language models. Most relevant to our work, various proposals to evaluate and augment the algorithm reasoning abilities of large language models are explored [Zhou et al., 2022, Lee and Kim, 2023, Zelikman et al., 2023, Liu et al., 2023]. In this work, we focus on evaluating and enhancing LLMs on graph-based reasoning and algorithmic tasks inspired by the increasing usage of LLMs in tasks with implicit graphs and structures.

7 Conclusion

In this work, we investigate whether LLMs are capable of explicit graph reasoning, i.e. , solving graph algorithm problems in natural language, across various problem categories and prompting techniques. To this end, we curate the NLGraph benchmark, a comprehensive test bed of graph-based reasoning in natural language, with 29,370 problems covering eight tasks with varying complexity. By evaluating

LLMs and prompting approaches on the NLGraph benchmark, we find that 1) LLMs do possess preliminary graph reasoning abilities, 2) the benefit of advanced prompting and in-context learning may diminish on complex reasoning tasks, while 3) LLMs are brittle to spurious correlations in problem settings. We then propose Build-a-Graph and Algorithmic Prompting, two simple instructionbased approaches that bring notable performance gains across multiple tasks. Improving LLMs' graph reasoning abilities on complex and nuanced graph reasoning tasks remains an open research question, and we encourage future work to develop upon our proposed NLGraph benchmark.

References

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning , pages 9118-9147. PMLR, 2022.

Jacob Andreas. Language models as agent models. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 5769-5779, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022. findings-emnlp.423 .

Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems , 33:3045-3057, 2020.

Prithviraj Ammanabrolu and Mark Riedl. Learning knowledge graph-based world models of textual environments. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/ forum?id=o24k_XfIe6_ .

Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. WIQA: A dataset for 'what if...' reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6076-6085, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1629. URL https://aclanthology.org/D19-1629 .

Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 1384-1403, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aclanthology. org/2022.emnlp-main.90 .

Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=3Pf3Wg6o-A4 .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35: 27730-27744, 2022.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id= _VjQlMeSB_J .

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=e2TBb5y0yFf .

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=WZH7099tgfM .

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=1PL1NIMMrw .

Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations , 2022. URL https://openreview.net/ forum?id=gJcEM8sxHK .

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main. 168. URL https://aclanthology.org/2021.naacl-main.168 .

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=-cqvvvb-NkI .

Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop , 2022. URL https://openreview.net/forum?id=wUU-7XTL5XO .

Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 13960-13980, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.780. URL https://aclanthology.org/2023. acl-long.780 .

Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yiming Yang, and Eduard Hovy. Could you give me a hint ? generating inference graphs for defeasible reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 5138-5147, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl.456. URL https: //aclanthology.org/2021.findings-acl.456 .

Swarnadeep Saha, Prateek Yadav, Lisa Bauer, and Mohit Bansal. ExplaGraphs: An explanation graph generation task for structured commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7716-7740, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.emnlp-main.609. URL https://aclanthology.org/2021.emnlp-main.609 .

Xinyan Velocity Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. Crepe: Open-domain question answering with false presuppositions. arXiv preprint arXiv:2211.17257 , 2022. URL https://arxiv.org/abs/2211.17257 .

Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. GreaseLM: Graph REASoning enhanced language models. In International Conference on Learning Representations , 2022a. URL https://openreview. net/forum?id=41e9o6cQPj .

Tianxing He, Kyunghyun Cho, and James Glass. An empirical study on few-shot knowledge probing for pretrained language models, 2021.

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al. Exploring the potential of large language models (llms) in learning on graphs. arXiv preprint arXiv:2307.03393 , 2023.

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 158-167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1015. URL https://aclanthology.org/P17-1015 .

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman. Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning , pages 31210-31227. PMLR, 2023.

Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021. URL https://openreview.net/forum?id=Jvxa8adr3iY .

Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Naturalprover: Grounded mathematical proof generation with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=rhdfTOiXBng .

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research , 2023.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https: //aclanthology.org/N19-1421 .

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics (TACL) , 2021.

Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066 , 2022.

Soochan Lee and Gunhee Kim. Recursion of thought: A divide-and-conquer approach to multicontext reasoning with language models. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 623-658, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.40. URL https://aclanthology.org/2023. findings-acl.40 .

Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. Parsel: A (de) compositional framework for algorithmic reasoning with language models. arXiv preprint arXiv:2212.10561 , 2023.

Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. Code execution with pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 4984-4999, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.308. URL https://aclanthology.org/2023.findings-acl.308 .

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022b.

A Discussion

NLP tasks and graph tasks alignment. Originally designed for processing language, LLMs are increasingly adopted for NLP tasks with structural implications. For instance, structured commonsense reasoning is similar to topological sort in that they both need to find a solution satisfying diversified constraints between entities. In the belief state graph in theory-of-mind [Adhikari et al., 2020, Ammanabrolu and Riedl, 2021], the final task is to determine the connectivity of nodes, which corresponds to the connectivity task. In multi-hop question answering [Creswell et al., 2023], LLMs implicitly find connections and paths among a vast network of entities and concepts, which resembles the shortest path task and connectivity task. However, we acknowledge that not all graph reasoning tasks in the NLGraph benchmark are clearly aligned with certain applications, thus advanced graph reasoning tasks in the NLGraph benchmark could also be viewed as a test of language model reasoning ability.

The NLGraph benchmark envisioned as reasoning benchmark. In addition to evaluating graph reasoning abilities, the NLGraph benchmark can also be viewed as a structural benchmark to evaluate language model reasoning. While ample LM reasoning datasets exist [Cobbe et al., 2021, Patel et al., 2021, Talmor et al., 2019], it is difficult to avoid train-test overlap since problems such as grade school math [Cobbe et al., 2021] and commonsense reasoning [Talmor et al., 2019] might be readily available in pretraining corpora, making it less convincing for evaluating reasoning abilities. On the contrary, the synthetic graph reasoning problems and their answers are highly unlikely to have exact matches in the training corpora, which makes NLGraph a more robust benchmark towards evaluating language model reasoning.

Fine-tuning to elicit graph reasoning abilities. To further study the graphical thinking abilities of LLMs, we envision fine-tuning as a possible direction for future work. We plan to fine-tune language models on the chain-of-thought reasoning process towards graph-based tasks on both single- and multi-task settings to see if fine-tuning might lead to enhanced graph reasoning abilities.

B Limitations

Tasks in NLGraph benchmark are not complete. While we have incorporated eight graph reasoning tasks with varying complexity in the NLGraph benchmark, there are many more important graph algorithms that test different levels of graph reasoning abilities. It might also be interesting to see how LLMs perform on other graph tasks such as finding the Eulerian path, the minimum spanning tree, and the cut-edge.

Language models that we evaluate are not complete. We only consider four black-box LLMs, TEXT-DAVINCI-003, CODE-DAVINCI-002, GPT-3.5-TURBO, and GPT-4 in our experiments. Since we will make the NLGraph benchmark publicly available, we leave it to future work on evaluating the graph reasoning abilities of other open-source LLMs.

Limited dataset size. Due to monetary costs, we only evaluate LLMs on the standard version of the NLGraph benchmark which has 5,902 problems throughout the paper. We believe evaluating LLMs on the extended version of the NLGraph benchmark, with 5x more problems, may bring stronger proof to the findings we present.

Methods for improving graph reasoning abilities. The two prompting methods we present are simple instruction-based approaches, which work on easy graph reasoning problems to varying extents. However, on more complex graph reasoning problems, as the overall graph reasoning abilities of LLMs are limited, the instruction-based methods seem to have only marginal effects. In future work, we hope to investigate methods such as asking LLMs to generate and execute code, or simulating steps of algorithmic solutions while maintaining the state variables.

C NLGraph Details

In Table 6, we provide the details of the NLGraph benchmark including edge probabilities and specific values of parameters mentioned in §2.2.

D Analysis

Instruction variants To further study the effect of BAG prompting and ALGORITHMIC prompting, we replace the instructions with only a sequence of dots (...) equal to the number of characters in the instruction. This is to investigate whether the improved performance is attributed to the natural language instructions or simply increased computation. We also replace the BAG prompting with three other instructions not directly related to graph problems, specifically, Let's think step by step (INSTRUCTION 1), Examine each detail carefully (INSTRUCTION 2), Think it through systematically (INSTRUCTION 3). As illustrated in Table 7, the BAG prompting method generally outperforms the instruction variants, indicating the effectiveness of this method.

Graph definition variants We study how replacing the abstract descriptions of graphs with realworld objects would impact model performance. Specifically, for the shortest path task, we change all the "node"s into "city"s, "edge"s into "road"s, and "weight"s into "distance"s. As shown in Table 9, the model performance improves on the easy subset but drops on the hard subset. While language models are indeed sensitive to the specific instantiations of the graph description, the results are mixed as to which is easier or harder.

3

4

5

6

7

8

0

20

40

60

80

Acc

easy

medium

hard

question

easy

medium

hard

exemplar

84.67

63.33

53.25

62.00

52.67

49.75

49.33

50.67

52.25

20

40

60

80

100

LLMs are brittle to problem scales For the shortest path task, we study the correlation between model performance and the number of nodes on the optimal shortest path. We group problems by the shortest path length and present results in Figure 6 for the COT approach, which shows that the accuracy steadily drops when the number of nodes on the shortest path increases. While the reasoning process towards finding the shortest paths stays the same for optimal solutions with varying lengths, large language models are not robust to changes in graph scales and problem settings.

In-context exemplars and test difficulty We investigate the model performance when the incontext exemplars and test difficulty vary. For the cycle task, we write chain-of-thought solutions for problems selected from the medium and hard subsets, and evaluate TEXT-DAVINCI-003 on the cycle task with the two prompts. We present the results in Figure 7. The performance drops when the exemplar difficulty increases, indicating that LLM struggles to learn from exemplars with more difficulty.

Partial credit for the maximum flow task We provide partial credit for the maximum flow task in Table 10. The performance generally has the same trend as the results provided in Figure 2. Though on the hard subsets, the performance of COT is the highest, we believe this is more like guessing, as when COT is combined with self-consistency, the performance drops significantly.

E Additional Large Language Models

E.1 CODE-DAVINCI-002 Results

We evaluate CODE-DAVINCI-002 on cycle, shortest path, and Hamilton path. The results are provided in Table 8. The findings are mostly similar to TEXT-DAVINCI-003: language models are preliminary graph reasoners on easy tasks while the benefit of in-context learning and advanced prompting is limited on more complex problems. However, in the COT and COT+SC settings, CODE-DAVINCI002 generates too many tokens before reaching the final answer, so we do not present the results in Table 8.

E.2 OPT Results

We fine-tune OPT-2.7B [Zhang et al., 2022b] and evaluate the model on the NLGraph benchmark. We randomly divide the standard version of the NLGraph benchmark into 4902 training problems and 1,000 test problems. We generate zero-shot solutions for the training problems, combine the solutions with the problems, and attain nine corpus of text ( i.e. one for each task and one that concatenating all the text). We then fine-tune OPT on the nine corpus separately. However, we find that fine-tuning did not elicit the graph reasoning abilities of OPT-2.7B: the generated text is mostly meaningless, falling short of answering questions and following formats. We leave it to future work on fine-tuning larger language models to elicit graph reasoning abilities due to our limitations in computational resources.

E.3 Qualitative Analaysis with GPT-3.5-TURBO and GPT-4

Due to monetary costs, we only conduct qualitative analysis on GPT-3.5-TURBO and GPT-4 with 19 representative problems across the eight tasks. We provide the results in Table 11, Table 12, Table 13, Table 14, Table 15. Results demonstrate that while GPT-4 does present better graph reasoning abilities, it is still far from being a robust reasoner towards graph-based problems.

Question In an undirected graph, the nodes are numbered from 0 to 6, and the edges are: An edge between node 0 and node 2 with weight 1, An edge between node 3 and node 0 with weight 4, An edge between node 5 and node 0 with weight 2, An edge between node 4 and node 0 with weight 3, An edge between node 0 and node 1 with weight 4, An edge between node 0 and node 6 with weight 2, An edge between node 2 and node 1 with weight 2, An edge between node 3 and node 1 with weight 3, An edge between node 4 and node 1 with weight 3, An edge between node 1 and node 6 with weight 2, An edge between node 2 and node 5 with weight 4, An edge between node 2 and node 4 with weight 3, An edge between node 6 and node 2 with weight 2, An edge between node 3 and node 5 with weight 1, An edge between node 4 and node 3 with weight 2, An edge between node 3 and node 6 with weight 4, An edge between node 4 and node 5 with weight 3, An edge between node 4 and node 6 with weight 1, An edge between node 6 and node 5 with weight 2. Q: Give the shortest path from node 2 to node 3.

GPT-3.5-TURBO

All the paths from node 2 to node 3 are: 2,0,3 with a total weight of 1 + 4 = 5, 2,1,3 with a total weight of 2 + 3 = 5, 2,4,3 with a total weight of 3 + 2 = 5. The weight of path 2,0,3 is the smallest, so the shortest path from node 2 to node 3 is 2,0,3 with a total weight of 5. %

All the paths from node 2 to node 3

are: 2,0,3 with a total weight of 1 + 4

= 5, 2,1,3 with a total weight of 2 + 3

= 5, 2,1,0,3 with a total weight of 2 +

4 + 4 = 10, 2,1,6,3 with a total weight

of 2 + 2 + 4 = 8, 2,4,3 with a total

weight of 3 + 2 = 5, 2,4,0,3 with a

total weight of 3 + 3 + 4 = 10, 2,4,1,3

with a total weight of 3 + 3 + 3 = 9,

2,4,1,0,3 with a total weight of 3 +

3 + 4 + 4 = 14, 2,4,1,6,3 with a total

weight of 3 + 3 + 2 + 4 = 12, 2,5,0,3

with a total weight of 4 + 2 + 4 = 10,

2,5,4,3 with a total weight of 4 + 5 +

2 = 11, 2,5,4,0,3 with a total weight

of 4 + 5 + 3 + 4 = 16, 2,5,4,1,3 with

a total weight of 4 + 5 + 3 + 3 = 15,

2,5,6,3 with a total weight of 4 + 6

+ 4 = 14, 2,6,1,3 with a total weight

of 2 + 2 + 3 = 7, 2,6,1,0,3 with a

total weight of 2 + 2 + 4 + 4 = 12,

2,6,5,0,3 with a total weight of 2 +

5 + 2 + 4 = 13, 2,6,5,4,3 with a total

weight of 2 + 5 + 5 + 2 = 14.

The weight of paths 2,0,3, 2,1,3, and

2,4,3 is the smallest.

So, there are

three shortest paths from node 2 to

node 3, which are 2,0,3; 2,1,3; and

2,4,3, each with a total weight of 5.

%

In a directed graph with 5 nodes numbered from 0 to 4:

node 2 should be visited before node 4

node 2 should be visited before node 1

node 2 should be visited before node 0

node 4 should be visited before node 1

node 4 should be visited before node 0

Q: Can all the nodes be visited? Give the solution.

This is not possible as there is a cycle between nodes 0, 1, and 4. Node 2 cannot be visited before nodes 0, 1, and 4, and nodes 0, 1, and 4 cannot be visited before node 2. Therefore, it is not possible to visit all the nodes in this graph. %

Since no nodes are required to be vis- ited before node 2 and node 3, node 2 and node 3 can be visited first. Since visiting node 4 requires visiting node 2 first, after node 2 is visited, node 4 can be visited. Since visiting node 0 requires visiting node 2, node 4 first, after node 2 and node 4 are visited, node 0 can be visited. Since visiting node 1 requires visiting node 2, node 4 first, after node 2 and node 4 are visited, node 1 can be visited. So the solution is: 2,3,4,0,1. "

In a directed graph with 5 nodes numbered from 0 to 4:

node 0 should be visited before node 1

node 1 should be visited before node 2

node 3 should be visited before node 1

node 3 should be visited before node 2

node 4 should be visited before node 3

node 4 should be visited before node 0

node 4 should be visited before node 1

Q: Can all the nodes be visited? Give the solution.

This graph has a cycle, which means it is not possible to visit all nodes in a valid order. Specifically, there is a cycle between nodes 1, 2, and 3, which cannot be resolved based on the given constraints. %

Since no nodes are required to be vis- ited before node 4, node 4 can be visited first. Since visiting node 0 and node 3 requires visiting node 4 first, after node 4 is visited, node 0 and node 3 can be visited. Since vis- iting node 1 requires visiting node 0 and node 3 first, after node 0 and node 3 are visited, node 1 can be vis- ited. Since visiting node 2 requires visiting node 1 and node 3 first, after node 1 and node 3 are visited, node 2 can be visited. So the solution is: 4,0,3,1,2. "

Question

GPT-3.5-TURBO

In a directed graph with 6 nodes numbered from 0 to 5:

node 0 should be visited before node 5

node 0 should be visited before node 1

node 2 should be visited before node 1

node 3 should be visited before node 0

node 3 should be visited before node 5

node 3 should be visited before node 1

node 4 should be visited before node 0

node 4 should be visited before node 2

node 5 should be visited before node 2

Q: Can all the nodes be visited? Give the solution

Since no nodes are required to be vis-

ited before node 3 and node 4, node 3

and node 4 can be visited first. Since

visiting node 0 requires visiting node

3 first, after node 3 is visited, node 0

can be visited. Since visiting node 1

requires visiting node 0, node 2, node

3 first, after node 0, node 2, and node

3 are visited, node 1 can be visited.

Since visiting node 5 requires visit-

ing node 0, node 3 first, after node

0 and node 3 are visited, node 5 can

be visited. Since visiting node 2 re-

quires visiting node 4, node 5 first,

after node 4 and node 5 are visited,

node 2 can be visited. So the solution

is: 3,4,0,2,1,5.

%

Since no nodes are required to be vis-

ited before node 3 and node 4, node 3

and node 4 can be visited first. Since

visiting node 0 requires visiting node

3 and node 4 first, after node 3 and

node 4 are visited, node 0 can be vis-

ited.

Since visiting node 5 requires

visiting node 0 and node 3 first, after

node 0 and node 3 are visited, node 5

can be visited. Since visiting node 1

requires visiting node 0, node 2, and

node 3 first, node 2 must be visited

before node 1.

Since visiting node

2 requires visiting node 4 and node

5 first, after node 4 and node 5 are

visited, node 2 can be visited. Now,

since node 0, node 2, and node 3

have been visited, node 1 can be vis-

ited.

So the solution is: 3,4,0,5,2,1.

"

In a directed graph, the nodes are numbered from 0 to

5, and the edges are: an edge from node 1 to node 3

with capacity 4,

an edge from node 1 to node 0 with capacity 9,

an edge from node 2 to node 1 with capacity 4,

an edge from node 3 to node 1 with capacity 4,

an edge from node 4 to node 3 with capacity 1,

an edge from node 4 to node 5 with capacity 2,

an edge from node 5 to node 1 with capacity 4,

an edge from node 5 to node 0 with capacity 7.

Q: What is the maximum flow from node 4 to node 0?

There is no path from node 4 to node

0, so the maximum flow from node

4 to node 0 is 0.

%

From the source (node 4), we can

send 1 unit of flow to node 3, and 2

units of flow to node 5.

1 unit of flow arrives at node 3, then

we can send 1 unit of flow from node

3 to node 1.

2 units of flow arrive at node 5, then

we can send 2 units of flow from

node 5 to node 0.

1 unit of flow arrives at node 1, then

we can send 1 unit of flow from node

1 to node 0.

The total unit of flow arrives at node

0 is 3. So the maximum flow from

node 4 to node 0 is 3.

"

In a directed graph, the nodes are numbered from 0 to

7, and the edges are: an edge from node 0 to node 6

with capacity 1,

an edge from node 0 to node 7 with capacity 10,

an edge from node 2 to node 1 with capacity 9,

an edge from node 2 to node 7 with capacity 8,

an edge from node 3 to node 1 with capacity 9,

an edge from node 3 to node 6 with capacity 10,

an edge from node 3 to node 4 with capacity 7,

an edge from node 4 to node 6 with capacity 1,

an edge from node 5 to node 3 with capacity 7,

an edge from node 5 to node 2 with capacity 9,

an edge from node 6 to node 4 with capacity 1,

an edge from node 7 to node 5 with capacity 8,

an edge from node 7 to node 6 with capacity 5.

Q: What is the maximum flow from node 2 to node 4?

From the source (node 2), we can

send 8 units of flow to node 7, and 9

units of flow to node 1.

8 units of flow arrive at node 7, then

we can send 5 units of flow from

node 7 to node 6, and 3 units of flow

from node 7 to node 5.

9 units of flow arrive at node 1, then

we can send 9 units of flow from

node 3 to node 1.

9 units of flow arrive at node 6, then

we can send 1 unit of flow from node

6 to node 4.

7 units of flow arrive at node 3, then

we can send 7 units of flow from

node 3 to node 4.

The total unit of flow arrives at node

4 is 8. So the maximum flow from

node 2 to node 4 is 8.

"

From the source (node 2), we can

send 9 units of flow to node 1, and 8

units of flow to node 7.

9 units of flow arrive at node 1, then

we can send 9 units of flow from

8 units of flow arrive at node 7, then

we can send 8 units of flow from

node 7 to node 5, and 5 units of flow

from node 7 to node 6.

8 units of flow arrive at node 5, then

we can send 7 units of flow from

node 5 to node 3, and 1 unit of flow

from node 5 to node 2.

14 units of flow arrive at node 3, then

we can send 14 units of flow from

node 3 to node 4.

The total unit of flow arrives at node

4 is 14. So the maximum flow from

node 2 to node 4 is 14.

%

Question

GPT-3.5-TURBO

There are 5 job applicants numbered from 0 to 4, and 6

jobs numbered from 0 to 5. Each applicant is interested

in some of the jobs.

Each job can only accept one

applicant and a job applicant can be appointed for only

one job.

Applicant 0 is interested in job 3.

Applicant 4 is interested in job 5. Applicant 4 is inter-

ested in job 4.

Applicant 3 is interested in job 0.

Applicant 0 is interested in job 5.

Applicant 1 is interested in job 0.

Applicant 0 is interested in job 2.

Applicant 0 is interested in job 4.

Applicant 4 is interested in job 0.

Applicant 4 is interested in job 2.

Applicant 2 is interested in job 3.

Q: Find an assignment of jobs to applicants in such that the maximum number of applicants find the job they are interested in.

Applicant 0 is interested in job 2, job 3, job 4, job 5. Applicant 1 is inter- ested in job 0. Applicant 2 is inter- ested in job 3. Applicant 3 is inter- ested in job 0. Applicant 4 is inter- ested in job 0, job 2, job 4, job 5. We can make the following assignments: applicant 0: job 5 applicant 1: job 0 applicant 2: job 3 applicant 3: job 0 applicant 4: job 2 5 applicants can find the job they are interested in. %

Applicant 0 is interested in job 2, job 3, job 4, job 5. Applicant 1 is inter- ested in job 0. Applicant 2 is inter- ested in job 3. Applicant 3 is inter- ested in job 0. Applicant 4 is inter- ested in job 0, job 2, job 4, job 5. We can make the following assignments: applicant 0: job 3 applicant 1: job 0 applicant 4: job 4 3 applicants can find the job they are interested in. %

n an undirected graph, the nodes are numbered from 0 to 5, and every node has an embedding. (i,j) means that node i and node j are connected with an undirected edge.

Embeddings:

node 1: [0,1]

node 2: [1,0]

node 3: [1,1]

node 4: [0,0]

node 5: [0,1]

The edges are: (0,1) (5,0) (1,4) (1,5) (2,4) (4,3)

In a simple graph convolution layer, each node's em- bedding is updated by the sum of its neighbors' embed- dings.

Q: What's the embedding of each node after one layer of simple graph convolution layer?

Layer 1: node 0: sum of embeddings of node 1, node 5, [0,1] + [0,1] = [0,2].

node 1: sum of embeddings of node 0, node 4, node 5, [1,1] + [0,0] + [0,1] = [1,2].

node 2: sum of embeddings of node 4, [0,0] = [0,0].

node 3: sum of embeddings of node 4, [0,0] = [0,0].

node 4: sum of embeddings of node 1, node 2, node 3, [1,2] + [1,0] + [0,0] = [2,2].

node 5: sum of embeddings of node 0, node 1, [0,2] + [1,2] = [1,4].

So the answer is:

node 2: [0,0]

node 3: [0,0]

node 4: [2,2]

node 5: [1,4] %

Layer 1:

node 0: sum of embeddings of node 1, node 5, [0,1] + [0,1] = [0,2].

node 1: sum of embeddings of node 0, node 4, node 5, [1,1] + [0,0] + [0,1] = [1,2].

node 2: sum of embeddings of node 4, [0,0] = [0,0].

node 3: sum of embeddings of node 4, [0,0] = [0,0].

node 4: sum of embeddings of node 1, node 2, node 3, [0,1] + [1,0] + [1,1] = [2,2].

node 5: sum of embeddings of node 0, node 1, [1,1] + [0,1] = [1,2].

So the answer is:

node 2: [0,0]

node 3: [0,0]

node 4: [2,2]

node 5: [1,2]

"

In an undirected graph, the nodes are numbered from 0 to 7, and every node has an embedding. (i,j) means that node i and node j are connected with an undirected edge.

Embeddings:

node 2: [1,0]

node 3: [1,1]

node 4: [1,1]

node 5: [0,0]

node 7: [1,0]

The edges are: (0,3) (4,0) (6,0) (5,1) (1,4) (5,2) (2,3) (2,6) (3,5) (3,6) (5,4) (7,5)

In a simple graph convolution layer, each node's em- bedding is updated by the sum of its neighbors' embed- dings.

Q: What's the embedding of each node after one layer of simple graph convolution layer?

Layer 1: node 0: sum of embeddings of node 3, node 4, node 6, [1,1] + [1,1] + [1,0] = [3,2].

node 1: sum of embeddings of node 4, [1,1] = [1,1].

node 2: sum of embeddings of node 3, node 5, node 6, [1,1] + [0,0] + [1,0] = [2,1].

node 3: sum of embeddings of node 0, node 2, node 5, node 6, [0,1] + [1,1] + [0,0] + [1,0] = [2,2].

node 4: sum of embeddings of node 0, node 1, node 5, [3,2] + [1,1] + [0,0] = [4,3].

node 5: sum of embeddings of node 2, node 3, node 4, node 7, [2,1] + [2,2] + [4,3] + [1,0] = [9,6].

node 6: sum of embeddings of node 0, node 2, node 3, [3,2] + [1,1] + [1,0] = [5,3].

node 7: sum of embeddings of node 5, [9,6] = [9,6].

So the answer is:

node 2: [2,1]

node 3: [2,2]

node 4: [4,3]

node 5: [9,6]

node 7: [9,6]

%

Layer 1: node 0: sum of embeddings of node 3, node 4, node 6, [1,1] + [1,1] + [1,0] = [3,2].

node 1: sum of embeddings of node 5, node 4, [0,0] + [1,1] = [1,1].

node 2: sum of embeddings of node 5, node 3, node 6, [0,0] + [1,1] + [1,0] = [2,1].

node 3: sum of embeddings of node 0, node 2, node 5, node 6, [0,1] + [1,0] + [0,0] + [1,0] = [2,1].

node 4: sum of embeddings of node 0, node 1, node 5, [0,1] + [1,1] + [0,0] = [1,2].

node 5: sum of embeddings of node 1, node 2, node 3, node 4, node 7, [1,1] + [1,0] + [1,1] + [1,1] + [1,0] = [5,3].

node 6: sum of embeddings of node 0, node 2, node 3, node 7, [0,1] + [1,0] + [1,1] + [1,0] = [3,2].

node 7: sum of embeddings of node 5, [0,0] = [0,0].

So the answer is:

node 2: [2,1]

node 3: [2,1]

node 4: [1,2]

node 5: [5,3]

node 7: [0,0]

%

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe applications of Large Language Models (LLMs) have been extended beyond natural language, covering more complex tasks that might have an implicit graph structure, such as planning for robots. This work introduces a benchmark named Natural Language Graph (NLGraph) to examine the explicit graph reasoning of advanced LLMs.\n\nNLGraph contains eight basic graph reasoning task, varying from easy (connectivity, cycle, topological, ...) to hard (maximum flow, Hamilton path, ...). Based on NLGraph, this work has several findings:\n\n* Current LLM (text-davinci-003) shows preliminary graph reasoning abilities. It achieves much better results than random guess in the easy task, for example, connectivity and cycle.\n* Advanced prompting fails in more complex tasks.\n* LLM are not robust to spurious correlation.\n\nThey also propose two simple prompts to enhance the LLM in easy graph reasoning task.\n\nSTRENGTHS:\n* This paper is well-motivated and finely written. As large language models (LLMs) grow increasingly proficient, we are in need of novel, challenging benchmarks to thoroughly assess their capabilities. The graph structure exemplifies data structures that are prevalent in real life.\n* This benchmark is fairly comprehensive, encompassing the eight fundamental types of graph problems. It also forms subsets of varying difficulty levels by manipulating factors such as the number of nodes and density.\n* The findings are both intriguing and illuminating. For instance, the revelation that current advanced prompt strategies cannot generalize to complex tasks could potentially motivate research into exploring innovative techniques.\n\nWEAKNESSES:\nThe evaluations are only conducted on text-davinci-003, therefore, some conclusions may be limited to this model.\n\nQUESTIONS:\nWhat is the correspondence between, train, main, and test splits provided in the supplementary material and the standard and extended set in the text?\n\nLIMITATIONS:\nThe authors discuss limitations comprehensively in the supplementary material, partly as future work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper curates a benchmark dataset called NLGraph that contains 8 types of graph reasoning problems. Given that various previous works have used LLMs to solve real-world tasks that implicitly require some form of simple graph reasoning, this dataset aims to isolate and analyze the ability of LLMs to graph reasoning problems. The paper tests various existing prompting strategies and also proposes two new strategies specifically useful for solving graph reasoning problems. The newly proposed strategies are evaluated on the proposed NLGraph dataset and are shown to improve the reasoning performance.\n\nSTRENGTHS:\n### originality\nThe paper is the first to curate a dataset aimed at isolating the graph reasoning abilities of LLMs in the abstract. This is complementary to all the works that use LLMs to solve tasks that implicitly require graph reasoning, and thus can provide valuable insights that can be useful in real applications.\n### quality\n1. The discussion about the limitations of the work (in the appendix) clearly states the scope of applicability of their finding.\n2. The experiments are well-designed, especially the ones demonstrating the brittleness of LLMs.\n### clarity:\nThe paper is well-organized, clearly written, and easy to follow.\n### significance\nThe curated dataset and the experimental setup proposed in the paper can be a good starting point for further exploration of the graph reasoning abilities of LLMs, making this work useful for the community.\n\nWEAKNESSES:\n### quality: \n1. The performance of the model depends heavily on how the prompts are constructed. While Figure 1 shows the instructions used in the prompts clearly, to ensure reproducibility, the complete prompts for more complex strategies like COT should be provided. \n### Significance: \n1. The motivation behind choosing the eight specific graph reasoning tasks is lacking. The ultimate goal of performing graph reasoning using LLMs is to use it to solve simple real-world tasks where graph reasoning is required. Given this motivation, the authors should provide additional justification for picking the reasoning tasks used in the paper.\n2. For the findings to apply to real-world data, the proposed prompting strategies should be tested with graphs that have real-world semantics associated with notes and edges. For example, representing nodes as numbers might allow the model to easily decouple the inherent meaning of the nodes from the meaning in the context of the given graph. This might not be an easy task if the nodes were a set of words, phrases, or actions.\n\nQUESTIONS:\n1. Line 160 states how the easier tasks were further subdivided into easy, medium, and hard. How are the more algorithmically challenging tasks categorized (Line 162)? Does it make sense to use the number of valid solutions for categorizing difficulty?\n2. How are the exemplars selected? How does the performance vary with the selection? Does is show high or low variance?\n3. In the zero-shot case, when there is not way to ensure the proper output formatting, how do you perform automated evaluation?\n\nLIMITATIONS:\nThe limitations discussed in the appendix are satisfactory. However, the authors should provide a reference to this discussion in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether large language models (LLMs) are able to solve graph algorithm problems in natural language. A benchmark NLGraph contains 29,370 problems, covering 8 graph reasoning tasks with varying complexity from simple tasks such as connectivity, cycle, and shortest path to more complex problems such as topological sort, maximum flow, bipartite graph matching, Hamilton path, and simulating graph neural networks. Various prompting techniques have been applied to evaluate the capabilities of LLMs for performing complex reasoning on graphs. Several valuable insights are provided based on the experimental results and analyses. Two prompting techniques Build-a-Graph and Algorithmic Prompting are further introduced for enhancing graph reasoning.\n\nSTRENGTHS:\n1. The proposed NLGraph benchmark is comprehensive, covering intuitively simple to more sophisticated tasks. Also, as the benchmark is synthetic, answers are unlikely to appear in the pretraining corpus of LLMs, making it a more robust benchmark for evaluating complex reasoning with LLMs. I think it would be a challenging and valuable testbed for evaluating the graph reasoning abilities of large language models.\n2. Extensive experiments and analyses are conducted on the proposed benchmark. Specifically, results of text-davinci-003 model with various prompting techniques (e.g. Chain-of-Thoughts, Least-to-Most, Self-Consistency) are reported.  Such results and analyses help the research community to better understand the capabilities/behavior of LLMs for graph reasoning or even complex reasoning. The four key phenomenons summarized in the paper are quite counter-intuitive and thought-provoking.\n3. The paper is clear and well-organized.\n\nWEAKNESSES:\nI only have several minor concerns/questions about this paper as follows:\n1. Using a programming style of prompting techniques (e.g. PAL[1], PoT[2]) for solving these graph reasoning tasks is more intuitive. It could potentially address the problem of generating too many tokens of code-davinci-002.  \n2. Except for the results of text-davinici-003, only part of the results of code-davinci-002 are reported (i.e. cycle, shortest path, and hamilton path).  For GPT-3.5-turbo and GPT4, only 19 tasks across 8 tasks are provided. I am not sure the four key findings listed are universal for the LLMs. It would be interesting to know the overall performance of other LLMs on the benchmark. It would shed light on this problem.  \n3. It seems to me that including the Graph Neural Networks (GNNs) task is not well-motivated. Why do we want the LLM to perform graph convolution on a two-dimension node embedding? We already have an efficient way to calculate it even for node embeddings that have hundreds or thousands of dimensions. If the LLM can use tools (e.g. calculator, python interpreter), why do we want it to perform large number calculations itself? Similarly, LLMs can write the code of different GNNs to perform message propagation.\n\n\nReferences:\n\n[1]PAL: Program-aided Language Models. \n\n[2]Program of Thoughts Prompting.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tried to answer the question, \"Are LLMs capable of mapping textual descriptions of graphs and structures to grounded conceptual spaces and solving graph algorithm problems explicitly with natural language?\" The answer to this question has profound implications for large language model applications with implicit graphs and structures, the reasoning ability of LLMs in advanced and graph-based settings, and more. This paper proposes the Natural Language Graph (NLGraph) benchmark, a comprehensive testbed of graphs and structured reasoning designed for language models and in natural language. Extensive experiments on the NLGraph benchmark demonstrate that:\n1. LLMs do possess preliminary graph reasoning abilities.\n2. The benefit of advanced prompting methods diminishes with complex problems.\n3. Learning from examples did not happen on complex graph reasoning problems.\n4. LLMs are (un)surprisingly brittle to spurious correlations in problem settings.\n\nTo improve large language models as better graph reasoners, this work proposes two instruction-based prompting approaches to better elicit the graph reasoning abilities of large language models. Build-a-Graph prompting encourages LLMs to map the textual descriptions of graphs and structures to grounded conceptual spaces before tackling the specific problem through a one-sentence instruction, while Algorithmic prompting instructs LLMs to revisit the algorithmic steps for a given task before learning from in-context exemplars.\n\nSTRENGTHS:\n1. The motivation of this work is to answer the question, \"Are LLMs capable of mapping textual descriptions of graphs and structures to grounded conceptual spaces and solving graph algorithm problems explicitly with natural language?\" is very interesting.\n\n2. The extensive experiments are comprehensive. Experiments demonstrate that build-a-graph and algorithmic prompting successfully empower LLMs to better tackle graph reasoning problems, resulting in 3.07% to 16.85% performance gains across multiple tasks.\n\n3. This paper provides the Natural Language Graph (NLGraph) benchmark, a comprehensive testbed of graphs and structured reasoning designed for language models and in natural language.\n\nWEAKNESSES:\nThis work is mainly on testing different prompt engineering strategies and in-context learning results. If we can include more LLM fine-tuning results, it will be of great help.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper evaluates and studies the performance of state-of-the-art LLMs on graph reasoning-based tasks. For this, the authors construct a testbed of graph and structured reasoning tasks, comprising of 29K problems on 8 graph reasoning tasks such as topological sorting and max flow. Each task has three subsets based on problem difficulty and the default metric is exact match accuracy. The comprehensive evaluation conducted by the authors shows that LLMs perform well on graph reasoning tasks but advanced-prompting algorithms such as CoT, L2M, Self-Consistency based COT, etc. do not yield performance improvements on complex tasks. The authors also show that the LLMs evaluated (mainly text-davinci-003, but also others from OpenAI) are brittle to spurious correlations in the problem settings. Finally, the authors propose two very simple, domain-specific prompting methods, named Build-a-Graph and Algorithmic Prompting, which improves upon zero-shot performance of text-davcinci-003 on the tasks.\n\nSTRENGTHS:\n1. Comprehensive and Interesting: I really like this work. This is primarily an evaluation paper, and the authors do a great job of conducting a systematic and comprehensive study of LLM performance on graph reasoning tasks. \n2. A Useful Resource for the Community: A number of reasoning problems could be reduced to implicit reasoning over graphs and the proposed benchmark, NLGraph will help the research community evaluate LLMs on such reasoning problems.\n\nWEAKNESSES:\n1. Lack of Systematic Comparisons between LLMs: Although I am empathetic to the plight of researchers conducting research on black-box models, only through an API, I feel the work would have been more valuable had the researchers also focused on a comparison between 3 generations of LLMs: GPT-3, GPT-3.5 and GPT-4. It would have enumerated a few things: the impact of instruction finetuning since the chat models are more aggressively instruction finetuned, the gains in performance across these models and the impact of advanced prompting with scale/model generation. The qualitative analysis in Appendix E on 19 problems is insufficient in ascertaining any of the performance jumps.\n\nQUESTIONS:\n1. From Appendix E, it appears that GPT-4 is considerably more capable than GPT-3.4-Turbo. A number of other works have found GPT-4 to show considerably higher performance, thereby hitting the thresholds of practical utility. Can you comment on any of this behavior wrt the graph reasoning tasks?\n2. I am surprised that zero-shot is the best performing method in Figure 2 (right, max-flow task). Similarly, I am finding it hard to understand why CoT underperforms zero-shot. How much time was spent iterating on CoT? Can you comment of this exact trend holding on for GPT-4 as well?\n3. To arrive at the Build-a-Graph and Algorithmic prompting methods, how many negative prompts did you iterate through? It is clear that domain specific contextual knowledge is being leveraged by LLMs to solve this task, but I am interested in the meta-question of how did you arrive these methods?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-15"}