Title: Evaluating Cognitive Maps and Planning in Large Language Models with CogEval
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Language Models

LLM evaluation

model comparison

GPT-4

graph analysis

cognitive science

cognitive map

hippocampus

planning

multi-step planning

reasoning

community graph
Abstract: Recently an influx of studies claims emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and falling in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.

Evaluating Cognitive Maps and planning in Large Language Models with CogEval

Abstract

Recently an influx of studies claims emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and falling in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.

1 Introduction

Large language models (LLMs) are generatively pre-trained and display apparent competence on some cognitive tasks [8]. This has led to a recent surge in studies claiming LLMs have emergent human-level cognitive abilities, which may encourage applications that interact with LLMs in a zero-shot or few-shot manner with expectations of human-level cognition. However, most claims of competence are based on anecdotes rather than systematic evaluation. In response, we make two contributions. First, we propose CogEval, a Cognitive Science-Inspired [12, 5, 39] protocol for Measurement and Evaluation of cognitive abilities in LLMs (Figure 1, top), such as planning, theory

of mind, causal inference, or other abilities. Second, we apply this evaluation protocol to the domain of cognitive maps and planning, and systematically evaluate these capacities across eight LLMs. We build our task prompts according to established human experiments, but our goal is not a comparison with human performance nor any assumptions of LLMs being "human-like" [27]. We evaluate LLMs' functional as opposed to formal linguistic abilities [21], and by that we have both a functionalist and multiple-realizability-based notion of cognitive ability [9] in mind.

We investigated whether LLMs (OpenAI GPT-4, GPT-3.5-175B, and davinci-003-175B, Google Bard, Cohere-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B) understand the latent structure of planning problems (cognitive maps). We hypothesized that failure in planning may relate to cognitive map deficits. To address these questions, we followed the CogEval protocol (Figure 1). First, we operationalized the latent ability (cognitive map and planning) in terms of multiple tasks with variations in three factors: (a) the latent structure of the tasks' environment (different Markov decision processes (MDPs) or graphs), (b) the domain (spatial vs. social ties vs. object relations), and (c) multiple planning tasks for each latent graph structure (c.f. Section 2 for detail). These domains were selected due to their prevalence in everyday problems as well as the cognitive science literature on cognitive maps [4, 44, 35]. We then generated repeated measurements across small and large LLMs (c.f. Section 2 for choice of LLMs) and conducted statistical analysis to compare the results. We found that LLMs only show apparent competence in simpler tasks, where route memorization was sufficient to find a solution, but fail on closer systematic observation. Our evidence suggests against out-of-the-box emergent planning capacities in recently introduced LLMs.

What is a cognitive map? A cognitive map is a representation of latent relational structures that underlies a task or environment, and facilitates planning, reasoning, and inference in biological and artificial problems [46, 4, 26, 7]. The concept originated from Tolman's latent learning experiments, demonstrating rodents' ability to learn maze structures without rewards [46]. This challenged the dominant behaviorist view that learning only occurs with reinforcement; and paved the way for a cognitive revolution. Decades later, discoveries of hippocampal place cells [31, 30, 32] and entorhinal cortex grid cells [13, 15, 29], together referred to as "the brain's GPS," further substantiated cognitive maps and earned the 2014 Nobel Prize [1]. Cognitive maps have since been studied behaviorally, computationally, and neurally; revealing that multi-step, multi-scale, and compressed neural representations are crucial for inference in both memory and planning [4, 26, 7]. Over the past decades, a number of reinforcement learning (RL) and deep neural network models have been proposed to capture the computations involved in cognitive maps and planning in the hippocampus and the prefrontal cortex of humans, rodents, bats, monkeys, and birds [4, 37, 7].

Why would LLMs plan with a cognitive map? It has been suggested that the transformer architecture and its learned representations, which lie at the heart of modern LLMs, are comparable to the hippocampus of the brain and the representations it learns [55]. Other studies show that GPT-3 is capable of event segmentation of narrative transcripts similar to human evaluators [23], and evaluate some cognitive capacities of GPT-3 using cognitive science and psychological methods applied in the evaluation of human cognition [5, 38, 48, 56]. Other cognitive scientists have distinguished formal linguistic ability (e.g., the ability to form grammatically correct sentences) from functional cognitive capacities (e.g., theory of mind, sequential planning, etc) and call for a meticulous evaluation of LLMs' functional competence without conflating them with their formal linguistic competence much like the dissociation of language and thought [21]. Taken together, these studies raise the hypothesis that LLMs would be able to extract and use cognitive maps from text, and second, that LLMs' failure in capturing cognitive maps could underlie failure modes in planning.

To test these hypotheses, we designed prompts to measure behavioral signatures of extraction and use of cognitive maps in a set of tasks adapted from existing human behavioral experiments [25, 24, 28, 34, 36]. We operationalized cognitive maps and planning with a number of tasks (Figure 1) with variations in environmental structures or graphs, varying items from different domains (spatial, social, object relations), and across a number of different conditions (e.g., value-based planning, reward and transition revaluation, shortcut, and detour).

Notably, the corresponding human experiments that inspired our prompts were never in linguistic form, and this is the first adaptation of them to prompts to the best of our knowledge. This is an important consideration since contamination of the training data with the test set is one of the most challenging obstacles to testing LLM capacities. To prevent any possible contamination, we avoided BIG-bench [40], which has been flagged by OpenAI for contamination [2], and a planning benchmark

Operationalization

Task & prompt generation

Data generation & evaluation

Graph

(cognitive map,

Latent ability

structure

Task

prompt

planning, etc)

S-M飄-

Item

measurement

domain

+

30X

Response

Results

Tasks/

per

Evaluation

prompt

+

Conditions

Task 1

Statistics

Task3

√1

Instruction

Task 4

Task 2

Latent structures of environments(with varying domains)

A

B

C

O+O

C

OOIO

D

E

F

Exampletaskprompt

Imagine abuilding withsixrooms.Fromthelobbyyou have twochoices,youcangotoroom1 orroom2.You enterroom1,at theotherendofroom1there'sadoorthatleadstoroom3, androom3leadstoroom5.There'sachest inroom5.Youopen it and there's10dollars,but youdonottake anymoney.Thenyouexit andstartover.Thistimeinthelobbyyouchoose room 2,which has a doortoroom4,and room4 has a door thatleads toroom 6.You find a chestwith50 dollarsinroom6,but youdonot take anymoney. You return to the lobby.You will onlybe able to choose onepath that leads to the most money. Whichroomfromthelobbywillead tothepathwhereonecanmakethemostmoney?

Graph:A,Domain:spatial

Cond:Value-based planning

for GPT-3 [49] as both pre-date GPT-4 and raise data contamination issues. Here we introduce and generate novel prompts inspired by human experiments with established validity in cognitive science. To our knowledge, a systematic evaluation of planning and cognitive map capacities in GPT-4 and comparison to other LLMs remain unexplored. In what follows we elaborate on a protocol and two related experiments to address this.

2 Methods

The CogEval protocol. In order to evaluate cognitive-map-related planning and navigation in LLMs, we propose and use the CogEval protocol (Figure 1). Please note that CogEval is not a benchmark nor limited to cognitive maps, it is a general protocol for evaluating any cognitive capacity, such as planning, theory of mind, causal reasoning, etc. As an example, here we have applied it to the domain of cognitive maps and planning.

CogEval adheres to four methodological best practices suggested by cognitive scientists [12]. First, the latent construct or ability : here we evaluate cognitive maps, which are representations that capture a model of the task structure, and adaptive planning, which requires an internal representation of task structures (similar to model-based RL [42] or task-oriented model-free RL [16-19]). Second, operationalization with construct validity : we operationalize planning ability by generating unique variations of established experimental tasks that measure the comprehension and use of cognitive maps in multi-step planning behavior [25, 36, 24]. Third, multiple tasks and multiple response generations : we generated many tasks for each condition of interest varying graph structure, and domain (spatial with ordered states such as room numbers, spatial with unordered states, social ties, object relations). Most task conditions include a partial change in the environment to test adaptive planning (e.g., changing the location of rewards or the structure of the environment, see Table 1). Collectively, these tasks allow us to robustly measure the latent construct: cognitive map and planning ability. Fourth, including multiple task conditions allows us to control for multiple factors when making inference about the ability.

Thus, we evaluate the construct using multiple environments with different graph structures (based on existing human experiments on cognitive maps [25, 36, 24], see graphs in Figure 1), controlling for robustness to variations in graphs, task conditions, and item domains (e.g., rooms, people, objects, random letters), using multiple generations (30 generations per condition), and across different temperatures ( 0 , 0 . 5 , and 1 ).

LLMs evaluated. We compared the following LLMs: GPT-4-* [2], GPT-3.5-turbo-175B [33], text-Davinci-3-175B [6] (Azure OpenAI API), Bard-* [45], Anthropic Claude-1-52B [3], LLaMA13B [47], Cohere-52.4B [10], Alpaca-7B [43] (nat.dev API), where * means the number of parameters is undisclosed.

Experiments. We conducted planning experiments to systematically compare the performance of all LLMs across task conditions created with 3 factors of graph structure (6 graphs), domain (3 domains), and tasks (15 tasks) over 3 temperatures (0, 0.5, 1)).

2.1 A cognitive science inspired evaluation of cognitive maps and planning capacity in LLMs

We designed our experiment prioritizing robustness and control conditions. Model performance on cognitive tasks can be influenced by various factors beyond the primary cognitive capacity, such as the specific prompts, the temperature parameter, experimental conditions (Table 1, Figure 1, bottom), the specific items the task is presented with or domain (e.g., spatial connections vs. social ties), and the specific relational graph underlying the problem (e.g., this could be a graph structure such as line graphs, trees, community graphs with different size and density). For instance, perhaps an LLM performs better when the items in a task are rooms that are numbered in order in a line graph (item or domain effect), or when the graph structure is finite rather than a community graph with potential loops (graph effect). Thus, we implemented measures to mitigate such effects, like potential performance variations due to task item selection or its underlying graph structure. We measured the results for each combination of factors and parameters 30 times for OpenAI models (for which we had API access) and once for the remaining models with no API access. We compared the results across 10 LLMs.

Why vary temperature? Temperature in LLMs determines randomness in the generated response, by manipulating the probabilities of the next word in a sequence. Thus, temperature can be thought of as a parameter controlling the diversity of the output. A temperature of 0 results in deterministic or greedy responses with less variance (Note: OpenAI has made it known that even a temperature of 0 is not entirely deterministic). With higher temperatures, especially closer to 1, the LLM creates more diverse and varied text upon repetition, akin to exploration. While a higher temperature may be helpful for tasks that require varied responses or creativity, it could go either way for planning: on the

one hand, precision in planning trajectories may seem more in line with a deterministic temperature, and on the other, a higher temperature leads to exploration, which may improve behavior by getting out of local minima. Repeating the experiments with varying temperature can help address its possible effect in either direction.

Statistical analysis. We evaluated the robustness of each LLM's performance by applying a statistical model of how each of the factors and their combinations contribute to variance in performance. Specifically, we fit a logistic regression analysis with domain, condition, and graph types as categorical regressors, and included second and third-order interaction terms between these three terms. We made sure that each combination of domain, condition, and graph had several replicates, though the approach is robust to imbalance issues. We included model version and temperature as separate independent variables that account for technical variation distinct from our conditions of interest. See supplement for full details on analysis and results.

2.1.1 Task prompts

Navigating cognitive maps requires adaptive multi-step planning using compressed representations of the environment, not mere memorization of all routes. Thus, cognitive map experiments test flexible adaptivity to local changes in the environment to evaluate biological and reinforcement learning agents [25, 25, 26, 14]. Latent learning experiments found that rodents who explored a maze with no reward could quickly find the shortest route to a newly introduced reward, i.e., find an optimal policy in RL context. This was taken as their ability to learn the cognitive maps of their mazes [46], but various additional experimental conditions were then designed and evaluated to confirm that they could flexibly adapt their cognitive map and planning to local environment alterations such as reward relocation (revaluation), changes to the map (transition revaluation) [25], or the introduction of shortcuts and detours [41]. Previous research has adapted these experiments to investigating the robustness and flexibility of deep model-based RL in the face of local changes to the reward structure (LoCA), and shown that deep model-based RL agents such as Dreamer v2, muZero, and PlaNet failed at flexible planning in reward revaluation scenarios [52]. Here we operationalized our tasks inspired by similar conditions in human reinforcement learning and deep MBRL experiments on learning, updating, and using cognitive maps for adaptive and flexible planning [25, 52].

Importantly, the corresponding human experiments were never conducted using texts, but were presented either as videos or a sequence of images that human participants moved forward by choosing an action (e.g. pressing left, right, up, or down). We believe this mitigates the risks of contamination. Moreover, when possible, we set the date earlier than our pilot studies to avoid potential contamination due to our experiments in the past month. To also ensure that the model cannot infer any answers from the papers, we asked GPT-4 to explain the experimental paradigm and draw the map of the environments after providing it a reference to a specific figure in a corresponding paper, and it failed. Thus, we believe our prompts have a negligible to no chance of having contaminated the training sets.

Below we provide examples of task prompts for graph A and a spatial domain (number ordered rooms). All prompts are available in the supplementary material and on https://github.com/cogeval/cogmaps.

I. Value-based or goal-driven planning. Below is an example prompt for value-driven or goaldirected planning in graph A in Figure 1. Success requires an understanding of the start and goal positions, comparison of the paths to find the shortest path that leads to the highest rewards, and planning a multi-step navigation or traversal of the underlying graph structure of the task.

Imagine a world with six rooms. From the lobby you have two choices, room 1 and room 2. You enter room 1, at the end there's a door that leads to room 3, and room 3 leads to room 5. There's a chest in room 5. You open it and there's 10 dollars. Then you exit and start over. This time in the lobby you choose room 2, then enter room 4, which leads to room 6. There's a chest with 50 dollars. You return to the lobby. Which room will you choose to make the most money?

II. Transition Revaluation, after prompt I. This condition occurs when the structure of the environment (e.g., an edge of the graph or Markov decision process) locally changes, and planning requires integrating or 'piecing together' different parts of the cognitive map to update one's plan or policy.

Now you're dropped in room 3 and the door at its end suddenly leads to room 6, and then

you're dropped in room 4 and the door at its end suddenly leads to room 5. you return to the

lobby. Which room will lead to more rewards?

III. Reward Revaluation, after prompt I. A common local change in any environment is when the location of rewards or goals change, without any changes to the map or structure of the states (or the cognitive map). This is known as Reward Revaluation or retrospective revaluation of rewards [25].

Now you're dropped into room 3, then you enter room 5 and the chest has 100 dollars. Then you're taken out, and dropped into room 4, then you enter room 6 and the chest has the same amount as before. When you return to the lobby, which room do you choose to make the most reward?

V. Shortcut prompts with and without teleportation, after prompt I.

Tolman's experiments on

cognitive maps [46] included a condition evaluating the animal's ability to discover shortcuts. Since

the early 1990s, evaluating the ability of various Dyna architectures [42] in discovering shortcuts has

been an important part of evaluating planning behavior. Below are two different shortcut prompts.

In the lobby you're presented with a portal, and you can choose which room to teleport into.

Which room do you choose to maximize rewards?

In the lobby you're presented with a new door which leads to a new room, room 7. Room

7's door leads directly to room 6. Remember that you will only be able to choose one path

that leads to the most money. Which room from the lobby will lead to the path where one can

make the most money?

V. Detour prompts with and without Teleportation, after prompt I.

You enter the lobby and this time you encounter a new room, room 7. Room 7's door leads to room 8, and room 8 leads to room 9. From room 9 you can teleport anywhere. You return to the lobby, and choose the room that leads to the most reward, but the door to the next room is blocked. You go back to the lobby. Which room do you choose to reach the most rewards?

You enter the lobby and this time you encounter a new room, room 7. Room 7's door leads to room 8, and room 8 leads to room 6. When you return to the lobby and choose the previous path that led to the most reward, you discover that the regular door to the room with the most money is now blocked. You go back to the lobby. You will only be able to choose one path that leads to the most money. Which room from the lobby will lead to the path where one can make the most money?

3 Results

3.1 Repeated measures comparison of planning across LLMs

We evaluated out-of-the-box emergent or native ability of different LLMs on the cognitive map tasks. Table 2 shows the statistical analysis highlighting the contributions of each factor to a logistic regression model's fit of LLM model performance. The magnitude of chi-square test statistics indicate contribution to overall model fit. Figure 2 compares the performance of all LLMs across all latent graph structures. Table 3 shows mean and standard error for planning performance across tasks and LLMs.

The results in Table 2 indicate that the LLM ( χ 2 (11) = 2357 . 87 , p < .001), graph ( χ 2 (11) = 3431 . 53 , p < .001), condition ( χ 2 (11) = 2080 . 04 , p < .001), and domain ( χ 2 (11) = 304 . 06 , p < .001) each yielded significant chi-squared statistics. This means that not only did different LLMs performed

Condition

Description

Group

valuePath

The optimal solution is to find the optimal policy, or shortest path, which yields the highest reward



















Traversal

1stepPath

The optimal solution is a 1-hop policy, i.e., goal is adjacent to the starting state

2stepPath

The optimal solution is a 2-step policy

3stepPath

The optimal solution is a 3-step policy

nstepPath

The optimal solution is an n-step policy, where max n is the diameter of the graph (longest shortest path)

rewardReval

Upon a local change in the

reward structure

, the goal has changed and the optimal solution requires finding a new path

}

RewReval

policyReval

Upon a local change in the

reward structure

, the optimal solution requires finding a new policy

transReval

Upon a local change in the

transition structure

, the goal is the same but the optimal solution requires finding a new policy

}

TransReval

transRevalStochastic

Upon a local change in the

transition structure

, the goal is the same but the optimal solution requires finding a new policy

in a stochastic environment

Upon a change in the graph structure, the optimal solution requires finding a shortcut



















Shortcut

nonteleShortcutCoT

Upon a change in the graph structure, the optimal solution requires finding a shortcut, an additional CoT prompt is given

teleShortcut

Upon a local change in the

transition structure

, the optimal solution requires finding a shortcut using a teleportation portal

teleShortcutCoT

Upon a local change in the

graph or transition structure

, the optimal solution requires finding a shortcut using

a teleportation portal, an additional CoT prompt is given

nonteleDetour

Upon a change in the graph structure, the optimal solution requires finding a detour

}

Detour

teleDetour

Upon a local change in the

transition structure

, the optimal solution requires finding a detour using a teleportation step

1.0

TA

LLMS

1.0

collection

0.8

gpt-4-32k

gpt-35-turbo

0.8

Detour

Traversal

text-davinci-003

Shortcut

score

0.6

工

真

replicate-llama-13b

replicate-alpaca-7b

anthropic-claude-v1

score

0.6

0.4

RewReval

TransReval

cohere-xlarge

0.4

bard

0.2

0.2

王

0.0

0.0

gpt-35

cohere

bard

A

C

B

E

D

F

gpt-4-32k

davinci-003

Ilama-13b

alpaca-7b

claude-v1

graph

LLMs

differently, but performance varied as a result of varying graphs, domains, and conditions. Conversely, the temperature showed a non-significant chi-squared statistic ( χ 2 (11) = 1 . 28 , p = . 53 ) and the interaction between the LLM and temperature was also non-significant ( χ 2 (11) = 10 . 69 , p = . 71 ). Noteworthy, the interactions among graph-domain, graph-condition, domain-condition, and graph-domain-condition were all significant (all p's < .001). The interactions among graph-domain ( χ 2 (11) = 334 . 41 , p < . 001 ), graph-condition ( χ 2 (50) = 1651 . 33 , p < . 001 ), domain-condition ( χ 2 (39) = 310 . 53 , p < . 001 ), and graph-domain-condition ( χ 2 (108) = 1133 . 16 , p < . 001 ) were all significant. A full table of regression coefficient estimates is included in the supplement.

In summary, while the 'temperature' and the interaction of 'LLM' and 'temperature' do not show significant effects on planning tasks, all other factors and their interactions significantly contribute to the variations in the dependent variable. These effects show that LLM performance on cognitive map and planning tasks was not robust to the graph structure of the problems, the domain, nor the task conditions, and it also varied across models (see Tables 2 and 3 and Figure 2).

3.2 Failure modes

We note three main failure modes when the task had an underlying graph with a dense community structure. Notably, when we probe the LLMs to list connected rooms or items as (state , actions , state) tuples, they do well (e.g., (room1 , opendoor , room3) is a tuple for graph A in Figure 1). However, when asked to do any tasks with a community graph structure using this tuple knowledge, LLMs display the following failure modes; (1) hallucinate edges that do not exist, or (2) produce longer trajectories instead of shortest paths, or (3) produce trajectories that fall in loops. For example in the task of finding the shortest path to a state that is 1 cluster away, out of 30 runs GPT-4 has a success rate of 0 at temperature 0 . Even with changing the temperature to 0 . 5 or

1 and repeating the same 30 runs its success rate can not exceed 10% . Please refer to Figure 3 for examples of above failure modes.

4 Discussion and future directions

This paper makes two main contributions. First, we introduce CogEval, a cognitive science inspired protocol for systematic and robust evaluation of functional [21] cognitive abilities in LLMs. Second, we follow the CogEval protocol to evaluate multiple LLMs' native or emergent ability to extract cognitive maps for sequential planning, navigation, or graph inference. All tasks and prompts are based on non-linguistic human cognitive science experiments that we adapted into text prompts for the first time. We test for robustness of the findings by varying task conditions, graph structure, domains (spatial, social), and LLM temperature. Our systematic evaluation reveals that while LLMs display apparent competence on some tasks in simpler graphs, they do not have out-of-the-box zero-shot emergent cognitive map comprehension or planning competence for other graphs.

Methodological contribution . CogEval , our proposed cognitive-science inspired protocol [12] for systematic evaluation of LLMs, affords the following contributions. (1) We avoid the reuse of contaminated standardized benchmarks by creating novel prompts based on non-text-based experiments that are known to evaluate cognitive maps and planning in humans, animals, and RL. (2) We use multiple tasks to probe the cognitive constructs (cognitive maps and planning) and repeat each interaction multiple times and across different temperatures. (3) We use statistical analysis to evaluate the robustness and reliability of each effect, with three main factors of graph structure, item domain (spatial vs. social), and task condition (e.g., value-based decision making, shortcut, detour, see Table 1). (4) We employ chain of thought and instruction prompts to evaluate the limits of out-of-the-box cognitive abilities of LLMs (Supplementary Experiment 2), and (5) analyze and categorize different types of failure modes. Please note that CogEval is not a benchmark nor limited to evaluating cognitive maps and planning, it is a general protocol for evaluating any cognitive capacity in LLMs. As an example, in this paper we have applied it to the domain of cognitive maps and planning.

No evidence for understanding cognitive maps or planning. Our systematic and incremental evaluations reveal limited to no cognitive map capacities in the current generation of LLMs - including

What is the shortestpath from

Whatistheshortestpathfrom

What is the shortestpathfrom

room1toroom12?

room 1to room 7?

room1toroom12?

Answer:1,7,10,13,12

Answer: 1,13, 10, 7

Answer: 1,7,4,7,13,12

GPT-4, temp=0, graph D, n-step

GPT-35, temp=1, graph D,1-step

GPT-4, temp=0, graph D, n-step

GPT-4. Specifically, we find that LLMs only show apparent competence on simple sequential inference tasks where route memorization can help, and given LLMs have received all possible trajectories in the text prompt. We also observe that the sparsity of graph structure drove performance. However, when 1-step and multi-step traversal and planning require understanding the underlying relational structure of the environment graph, LLMs including GPT-4 fail by hallucinations, suboptimally long routes, or falling in loops.

How did LLMs solve the simpler tasks? Without access to full architectures or training sets, we can only form hypotheses based on our behavioral observations. We observe that LLMs do better in problems where the entire trajectories are explicitly available in the text prompts, and they only need to retrieve and piece together partial changes. However, planning behavior in more complex graphs is far worse, and this is not just due to graph size: performance was worse for graph B (7-node tree) than C (12 node, parallel lines). Performance on the 15-node graph with 3 dense clusters was worse than the 16-node (4-cluster) graph that has better cross-cluster connectivity and more paths among clusters.

These observations suggest that LLMs may fail at planning problems where they need to use the transition structure to unroll the trajectories and find the correct path, which is closer to the notion of planning in model-based RL and in cognitive science. Capturing the underlying structure and using it to unroll trajectories are quintessential to cognitive maps and planning ability. Thus, the apparent competence in simpler tasks may be due to using cached or memorized routes rather than understanding the cognitive map, planning, or inference ability.

LLMs may do better in smaller and simpler graphs because the prompt already expands all the possible paths or trajectories. When there is a change in the rewards or transition structure, LLMs only need to change one step in an already laid out path. However, in more complex graphs only the one-step connections are laid out in the prompt, but not all paths or trajectories between any given two nodes. We observed that failures significantly increase in tasks with these larger graphs with community structures, even when an LLM can list the pairwise tuples of connected states (see failure modes, Figure 3).

Interpreting the results. The experiments in the paper are not meant to be interpreted as a benchmark for planning. They probe the same construct in different ways, evaluating the ability to use information about (state , action , state) tuples, e.g., (room1 , openleftdoor , room3) , to piece together policies in response to task prompts. A reader may wonder why we claim that LLMs do not display emergent planning in spite of non-zero performance for some tasks in our experiments (Figure 2. We interpret the findings as such due to failure modes and various inconsistencies in success cases (Figure 3). For instance, a common failure mode is generating sequences with hallucinated (state , actions , state) tuples that do not exist. For GPT-4 this constitutes over 20% of 4-step plans. Another common failure mode is that they fall into loops when prompted to find the shortest path between two states (Figure 3, left and right). LLMs sometimes even fail to identify 1-step paths or suggest multi-hop trajectories for traversing to an adjacent state (Figure 3, middle).

These observations seem counter-intuitive given some LLMs can generate a list of tuples when asked, but fail to use the tuples to make (even 1-step) valid plans. This shows that, while LLMs appear to solve planning problems when given simple routes that can be explicitly memorized, they cannot generalize from route memory solutions (trajectories directly in the prompt) to using the tuples to adaptively generate branching plans. Together, these inconsistent observations are in line with the hypothesis that LLMs do not understand cognitive maps and therefore cannot consistently plan. Elsewhere propose black box architectures that improve planning performance [54]. However, our findings point to a lack of out-of-the-box zero-shot emergent planning ability.

Limitations. First, we lack knowledge of LLMs like GPT-4's architecture or training. To address this limitation, we did not use existing text-based benchmarks and instead generated novel prompts not in their training sets. Second, in the human experiments that influenced our prompts participants learn gradually, experiencing states one-by-one, only tested after they showed signs of learning, similar to a model-based RL agent having the transition structure and using it for inference and planning. To address this difference, we present the environment's structure in linguistic format. The LLM had to extract the cognitive map, identify the goal location based on instructions. and infer the policy towards the goal. Third, we do not have a human baseline for the language-based tasks, so a comparison with human behavior on the exact language prompts remains the topic of future studies.

Implication for applications. LLMs are expected to be applied in fields like gaming, planning, and social reasoning, with tasks that require understanding the inherent relational structure of the problem from the input for flexible reasoning and planning. However, here we show various failure modes in the understanding of the underlying cognitive maps or planning abilities, including hallucination and falling in loops. Even when provided instructions and Chain of Thought (CoT) prompts like breadth-first search (BFS), we observe that GPT-4 struggles to process multi-hop paths it has not experienced (Supplementary Experiment 2). These findings suggest caution in the application of LLMs in problems that involve planning or complex structures. Below we discuss future directions that may mitigate these challenges for problems with simpler structures.

LLMs as programmable machines rather than emergent intelligence? While some regard LLMs as agents with emergent intelligence comparable to humans and animals, our findings are more consistent with the view that LLMs are programmable machines where natural language is their programming language [20]. Thus, here we evaluated planning in LLMs in a functionalist and multiple-realizability sense rather than making any assumptions of them being "human-like" [27].

Future directions. A future direction is to analyze embedding representations and attention in LLMs, and test hypotheses about representations underlying success and failure in planning. This mirrors how neuroscience analyzes neural data to understand representations in model-based and predictive planning and decision-making [25, 7]. A recent study suggests a promising future for this direction [57]. Another interesting direction is to study the limits of LLMs' transitive inference using pairwise associations [37, 34], given we observed that while some LLMs could list pairwise tuples or recognize the goal, they still struggled with planning. A further direction is to study whether the use of schemas , i.e., overused, generalized cognitive maps such as "airport" [50, 22, 11, 51], can improve performance on real-world scenarios by evoking helpful structures, given LLMs have shown promise with analogical reasoning tasks [53]. Finally, some have suggested ways to improve planning by augmenting LLMs with algorithms that enable executive control, an interesting direction that can contribute to the future of augmenting both larger and especially smaller language models (see [54]).

LLMs need a hippocampus and prefrontal cortex. The hippocampus and prefrontal cortex in the brain extract the relational structure or cognitive maps from sequential data to flexibly plan at multiple scales [7, 26]. Their functions can inspire memory, planning, and executive control augmentations to LLMs in order to mitigate failure modes such as hallucinating edges. Ongoing research shows that indeed prefrontal cortex-inspired solutions can improve planning performance in an LLM-based architecture with multiple calls to GPT-4 [54], pointing at a promising future direction.

Summary. We introduced CogEval, a cognitive science inspired protocol for systematic and robust evaluation of LLMs. We applied CogEval to evaluate planning performance across 8 LLMs and found poor performance.

References

The nobel prize in physiology or medicine 2014. https://www.nobelprize.org/prizes/ medicine/2014/press-release/ . Accessed: 2023-5-10.

OpenAI 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Anthropic. Introducing Claude. https://www.anthropic.com/index/ introducing-claude , 2023. [Online].

Timothy E J Behrens, Timothy H Muller, James C R Whittington, Shirley Mark, Alon B Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson. What is a cognitive map? organizing knowledge for flexible behavior. Neuron , 100(2):490-509, October 2018.

Marcel Binz and Eric Schulz. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences , 120(6):e2218523120, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Iva K Brunec and Ida Momennejad. Predictive representations in hippocampal and prefrontal hierarchies. J. Neurosci. , November 2021.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. March 2023.

Rosa Cao. Multiple realizability and the spirit of functionalism. Synthese , 200(6):506, December 2022.

Cohere. Introducing Cohere. https://txt.cohere.com/ cohere-launches-extremely-large-beta-2 , 2022. [Online].

Delaram Farzanfar, Hugo J Spiers, Morris Moscovitch, and R Shayna Rosenbaum. From cognitive maps to spatial schemas. Nat. Rev. Neurosci. , 24(2):63-79, February 2023.

Michael C Frank, Mika Braginsky, Julie Cachia, Nicholas Coles, Tom Hardwicke, Robert Hawkins, Maya Mathur, and Rondeline Williams. Experimentology. https:// experimentology.io/ , 2023. Accessed: 2023-5-9.

Marianne Fyhn, Sturla Molden, Menno P Witter, Edvard I Moser, and May-Britt Moser. Spatial representation in the entorhinal cortex. Science , 305(5688):1258-1264, August 2004.

Mona M Garvert, Raymond J Dolan, and Timothy Ej Behrens. A map of abstract relational knowledge in the human hippocampal-entorhinal cortex. Elife , 6, 2017.

Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I Moser. Microstructure of a spatial map in the entorhinal cortex. Nature , 436(7052):801-806, August 2005.

Hosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Logically-constrained neural fitted Q-iteration. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pages 2012-2014. International Foundation for Autonomous Agents and Multiagent Systems, 2019.

Hosein Hasanbeig, Yiannis Kantaros, Alessandro Abate, Daniel Kroening, George J Pappas, and Insup Lee. Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees. In Proceedings of the 58th Conference on Decision and Control , pages 5338-5343. IEEE, 2019.

Hosein Hasanbeig, Daniel Kroening, and Alessandro Abate. Certified reinforcement learning with logic guidance. Artificial Intelligence , page 103949, 2023.

Hosein Hasanbeig, Natasha Yogananda Jeppu, Alessandro Abate, Tom Melham, and Daniel Kroening. Symbolic task inference in deep reinforcement learning. Journal of Artificial Intelligence Research (JAIR) , 2023.

Ana Jojic, Zhen Wang, and Nebojsa Jojic. GPT is becoming a turing machine: Here are some ways to program it. March 2023.

Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: a cognitive perspective. January 2023.

Rolando Masís-Obando, Kenneth A Norman, and Christopher Baldassano. Schema representations in distinct brain networks support narrative memory during encoding and retrieval. Elife , 11, April 2022.

Sebastian Michelmann, Manoj Kumar, Kenneth A Norman, and Mariya Toneva. Large language models can segment narrative events similarly to humans. ArXiv , January 2023.

I Momennejad, A R Otto, N D Daw, and K A Norman. Offline replay supports planning in human reinforcement learning. Elife , 2018.

I Momennejad, E M Russek, J H Cheong, M M Botvinick, N D Daw, and S J Gershman. The successor representation in human reinforcement learning. Nat Hum Behav , 1(9):680-692, September 2017.

Ida Momennejad. Learning structures: Predictive representations, replay, and generalization. Current Opinion in Behavioral Sciences , 32:155-166, April 2020.

Ida Momennejad. A rubric for human-like agents and NeuroAI. Philos. Trans. R. Soc. Lond. B Biol. Sci. , 378(1869):20210446, December 2022.

Ida Momennejad, Ajua Duker, and Alin Coman. Bridge ties bind collective memories. Nat. Commun. , 10(1):1578, April 2019.

Edvard I Moser, Emilio Kropff, and May-Britt Moser. Place cells, grid cells, and the brain's spatial representation system. Annu. Rev. Neurosci. , 31:69-89, 2008.

J O'Keefe. Place units in the hippocampus of the freely moving rat. Exp. Neurol. , 51(1):78-109, April 1976.

J O'Keefe and J Dostrovsky. The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat. Brain Res. , 34(1):171-175, November 1971.

John O'Keefe and Lynn Nadel. The Hippocampus as a Cognitive Map . Oxford: Clarendon Press, 1978.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Athula Pudhiyidath, Neal W Morton, Rodrigo Viveros Duran, Anna C Schapiro, Ida Momennejad, Demitrius M Hinojosa-Rowland, Robert J Molitor, and Alison R Preston. Representations of temporal community structure in hippocampus and precuneus predict inductive reasoning decisions. J. Cogn. Neurosci. , 34(10):1736-1760, September 2022.

Matthew Schafer and Daniela Schiller. Navigating social space. Neuron , 100(2):476-489, October 2018.

Anna C Schapiro, Timothy T Rogers, Natalia I Cordova, Nicholas B Turk-Browne, and Matthew M Botvinick. Neural representations of events arise from temporal community structure. Nat. Neurosci. , 16(4):486-492, April 2013.

Anna C Schapiro, Nicholas B Turk-Browne, Matthew M Botvinick, and Kenneth A Norman. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Philos. Trans. R. Soc. Lond. B Biol. Sci. , 372(1711), 2017.

Richard Shiffrin and Melanie Mitchell. Probing the psychology of AI models. Proc. Natl. Acad. Sci. U. S. A. , 120(10):e2300963120, March 2023.

Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. March 2023.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karaka¸ s, B Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Koco´ n, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B Tenenbaum, Joshua S Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia ContrerasOchando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem ¸ Senel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Sw˛ edrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Varma T Mukund, Nanyun Peng,

Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R Bowman, Samuel S Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T Piantadosi, Stuart M Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. June 2022.

K L Stachenfeld, M Botvinick, and others. Design principles of the hippocampal cognitive map. Adv. Neural Inf. Process. Syst. , 2014.

Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT Press, November 2018.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following LLaMA model, 2023.

Rita Morais Tavares, Avi Mendelsohn, Yael Grossman, Christian Hamilton Williams, Matthew Shapiro, Yaacov Trope, and Daniela Schiller. A map for social navigation in the human brain. Neuron , 87(1):231-243, July 2015.

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239 , 2022.

E C Tolman. Cognitive maps in rats and men. Psychol. Rev. , 55(4):189-208, July 1948.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Tomer Ullman. Large language models fail on trivial alterations to Theory-of-Mind tasks. February 2023.

Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). June 2022.

Marlieke T R van Kesteren, Sarah F Beul, Atsuko Takashima, Richard N Henson, Dirk J Ruiter, and Guillén Fernández. Differential roles for medial prefrontal and medial temporal cortices in schema-dependent encoding: from congruent to incongruent. Neuropsychologia , 51(12):2352-2359, October 2013.

Marlieke Tina Renée van Kesteren and Martijn Meeter. How to optimize knowledge construction in the brain. NPJ Sci Learn , 5:5, May 2020.

Yi Wan, Ali Rahimi-Kalahroudi, Janarthanan Rajendran, Ida Momennejad, Sarath Chandar, and Harm van Seijen. Towards evaluating adaptivity of Model-Based reinforcement learning methods. April 2022.

Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. December 2022.

Taylor Webb, Shanka Subhra Mondal, Chi Wang, Brian Krabach, and Ida Momennejad. A prefrontal cortex-inspired architecture for planning in large language models, 2023.

James C R Whittington, Joseph Warren, and Timothy E J Behrens. Relating transformers to models and neural representations of the hippocampal formation. December 2021.

Eunice Yiu, Eliza Kosoy, and Alison Gopnik. Imitation versus innovation: What children can do that large language and language-and-vision models cannot (yet)? May 2023.

Safoora Yousefi, Leo Betthauser, Hosein Hasanbeig, Akanksha Saran, Raphaël Millière, and Ida Momennejad. In-context learning in large language models: A neuroscience-inspired analysis of representations, 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a test battery to test for the emergence of cognitive maps and planning abilities in LLM. The tests are based on existing cogsci experiments converted into text prompts. For example, to test for a planning ability, the prompt first describes an apartment layout, and then asks the model to plan a path through several rooms to retrieve something. The authors show that LLM are poor at completing such tasks.\n\nSTRENGTHS:\nI think the adaptation of cogsci planning experiments to text prompts is a great idea. There needs to be more tests like this to map the scope of LLM capabilities.\n\nThe authors evaluate multiple prompts from the same generative model and compute statistics of LLM responses. This approach stands in contrast to many arguments for emergent intelligence in LLM, which derive from anecdotal examples. It is important to do actual experiments with LLM.\n\nWEAKNESSES:\nThis is a presentation issue, but it affects my ability to understand and evaluate the paper. The writing is unedited. There are a lot of unnecessarily long sentences that can be condensed for pragmatics and readability. The grammar and punctuation are weird, like this might be an initial draft. I had to read the same content multiple times, and missed important details throughout the paper. \n\nFigures are odd, seem to be just thrown in together from unrelated bits. For example Figure 1 shows examples of graphs A-E. Are these all the graphs used in Experiment 1? If so, this is not stated. Weirdly, each graph is shown in a different pictorial style. I'm sorry, but this looks like someone just downloaded different types of graphs from google search, panel E has a random red outline around it.\n\nExperiment design is not given. I can not check the statistics without the experiment design. What are these degrees of freedom? It looks like multiple regression models were built? I think the intention is a good one, but it is not clear what was done\n\nQUESTIONS:\n1.\n\"\nWe evaluated the robustness of our findings by regenerating the results for each combination of factors\nand parameters multiple times and applying a statistical model of how each of these factors contribute\nto variance in LLM performance.\n\"\nWhat does this mean? Where is the exact experimental design? How many trials were simulated? Which conditions were evaluated?\n\n2. \nWhere is Experiment design for Experiment 1? Where are the graphs?\n\n3.\n\"198 For the graph community block model, example graphs are shown in Figure 2 ...\"\nWhat does this mean? Does Figure 2 show all graphs, or a subset of them? If this is a subset, then how many graphs of each kind were shown?\nWhere is the \"community block model\" described?\n\nLIMITATIONS:\nI can not make out what was done exactly, but am happy to read the next draft. The experiments and statistical analysis need to be described so that people could reproduce them.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper evaluates LLMs on a set of tasks that could be solved by cognitive maps, such as goal-oriented planning, or incorporating shortcuts. The work finds that LLMs generally perform poorly at these tasks, and their performance is affected by features such as graph sparsity.\n\nSTRENGTHS:\n* The paper is admirably thorough with experiments and analyses:\n    - Assessing a range of LLMs (including varying parameters such as temperature).\n    - Evaluating across a range of different graph structures, task paradigms, etc.\n    - Creating new stimuli to avoid dataset contamination. \n    - Performing regression analyses to determine how different features contribute to model performance, and describing the failure modes observed.\n    - These thorough results and analyses suggest that the conclusions are likely to roughly generalize to some extent, and help the reader to understand which features will affect performance.\n* The results are interesting, there are a variety of patterns that could be investigated further.\n\nWEAKNESSES:\nMy primary concern is that the overall framing of the paper is misleading. In particular, the work is motivated with references to the cognitive and neuroscience literature on cognitive maps. This work performs  draw strong conclusions from its experiments such as \"no evidence for understanding cognitive maps or planning.\" Are conclusions such as these justified?\n\nA key issue in analyzing AI in comparison to human or animal capabilities is determining where a performance failure originates: is it a lack of an underlying capability (such as the ability to form a cognitive map), or a more superficial performance issue? Several recent papers have emphasized this point from a cognitive science perspective, and argued that it is essential to ensure fair comparisons between AI and natural intelligence to draw accurate conclusions (https://www.pnas.org/doi/abs/10.1073/pnas.1905334117; for LLMs specifically see: https://arxiv.org/abs/2210.15303).\n\nIn that context, it's worth noting that the animal and human experiments cited involved a great deal more experience before the map was tested than the present experiments do. For example, Tolman's latent learning experiments involved the rats fully exploring the maze for multiple days before they were tested with a food reward at the end (and even then, performance continued to improve well after the rewards were first introduced). Or Schapiro's temporal community structure paper involved half an hour of exposure to transitions from the graph; that is, thousands of transitions from a graph with only 15 nodes. This a much denser sampling of experience than the current LLM experiments afford, and it is quite possible that some degree of repeated experience contributes to the ability of natural intelligences to form a cognitive map. The difference in learning conditions is briefly mentioned to in the limitations, but is quickly dismissed; however, the difference in experimental conditions is a fundamental challenge to concluding that LLMs are failing to form cognitive maps like animals/humans do.\n\nLikewise, it is typical in cognitive science to report comparisons to chance-level performance, and an underlying ability is usually inferred from better-than-chance performance, even if that performance is imperfect. For example, in Tolman & colleague's studies, the rats continued to improve for several days after the reward was introduced (that is, their paths were not optimal on the first test), and the rats were clearly stitching together trajectories that they had observed (since there was only a single route through the maze, everything else lead to dead ends), but we still interpret their performance as showing latent learning. It would be useful to report chance-level performance across all conditions, and, in the case that a model performs better than chance across a broad range of conditions, that would suggest some underlying ability, even if it is imperfect. For example, GPT-4's performance seems reasonably high across most conditions in Table 2 (though certainly imperfect).\n\nIn addition, explicit comparisons to human performance on these tasks (presented exactly as they are presented to the language model), would strengthen the claim that language models are failing in a fundamental way that humans or animals would not.\n\nThese points seem critical to the overall framing of the paper, and also to much of the discussion. I therefore think the paper would be substantially improved by:\n1) providing a more nuanced framing of the very interesting results, that suggests they emphasize some limitations of planning in LLMs, without making overly strong claims such as \"no emergent planning\" or \"no evidence of cognitive maps\"\n2) Providing explicit comparisons to chance-level (and ideally human) performance to help contextualize the results.\n\nQUESTIONS:\n* In the discussion, what explicit experiment is referred to by \"We observe that LLMs do better in problems where the entire trajectories are explicitly available in the text prompts, and they only need to piece together partial changes\"?\n\nLIMITATIONS:\nSee weaknesses; I believe that the limitations of the experimental paradigm not matching the inspiration are not fully discussed, and that more generally the conclusions are not fully supported by the experiments presented.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

comments: SUMMARY:
The paper presents CogEval, a set of best practices ported from cognitive science on how to do behavioral evaluations. The authors also transcribe new tasks from human reinforcement learning and planning into text, such that LLMs can be tested on them. On these tasks, the authors do not observe evidence for an emergent capability for planning in large language models.

STRENGTHS:
- The presentation of CogEval is clear and potentially useful for the ML community. I’m excited to use CogEval in my own work.
- Thorough and thoughtful discussion.
- Exhaustive experimental setup—a feature perhaps enabled by the principled CogEval framework!
- The paper is lucid overall and easy to read.

WEAKNESSES:
There’s some mild overselling in the paper, given the empirical results, which only demonstrate the failure of an existence proof.  Defining the conditions under which we can declare definitively that there is no emergent planning. However, given that we have not rigorously defined these conditions, for example, “No Emergent Planning” in the title seems too strong.

QUESTIONS:
- Are you planning to release these tasks transcribed to text as a new testbed?
- Does transcribing these tasks to text make them harder? Perhaps a vision-and-language model like GPT-4 or Flamingo might perform better on pictures of the task, in which case, perhaps your conclusions would be different. 
- Why is “Measurement and Evaluation for Large Language Models” capitalized in line 6?

Nits:
- Line 14: Stay in active voice here for consistency?
- Figure 3: Can you make this less blurry?

LIMITATIONS:
The authors have adequately addressed limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an evaluation of large language models with respect to their ability to solve problems that require use of latent cognitive maps. Evaluation focuses on different underlying graph structures, and the influence of chain-of-thought inference on performance.\n\nSTRENGTHS:\n* I really liked the very clear outline of the motivation of the research design in the introduction of Section 2.\n* The experiments are extremely thorough, and the goal of designing experiments with statistical robustness in mind is good.\n* Good discussion around the capabilities and limitations of LLMs\n\nWEAKNESSES:\nSome of the presentation could be refined:\n* The description of Figure 1 is somewhat difficult to understand. References to future aspects of the paper (e.g., \"Experiment 3\") are undefined, which makes it more difficult to understand.\n* The figures / tables should appear closer to where they are referenced in the text.\n* I'd suggest reordering 2.1, so that the tasks (i.e., maze learning) are described before the experimental setup.\n* Some details could use more context. e.g., what is temperature? What are the graph structures A/B/C... etc? \n* The discussion on BFS/DFS prompting should be moved to the experimental setup section (2.3). I also didn't quite understand the distinction between these two; an example would help.\n* Formatting of Figure 3 can be improved\n* What is \"dialogue\" referring to throughout the paper? I don't believe an actual multi-turn dialogue is taking place during the evaluation\n* Text of Table 2 is really tiny\n\nQUESTIONS:\n* I am not completely familiar with the term \"cognitive maps\", and I'm not sure if this more of a metaphor, or usually applied to actual spatial reasoning tasks (hence \"map\"). Since the examples in the paper are about navigation, I am wondering if this means the domains studied are mostly about literal maps, or if there are other domains usually studied with the framework of cognitive maps. Are there other domains covered in this study? If so, what are they?\n* If there are different domains, how does performance vary across them?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}