Evaluating Cognitive Maps and Planning in Large Language Models with CogEval
Authors: Ida Momennejad
Hosein Hasanbeig
Felipe Vieira Frujeri
Hiteshi Sharma
Nebojsa Jojic
Hamid Palangi
Robert Ness
Jonathan Larson
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Models
LLM evaluation
model comparison
GPT-4
graph analysis
cognitive science
cognitive map
hippocampus
planning
multi-step planning
reasoning
community graph
TL;DR: We evaluate LLMs' ability for inferring the structure of the environment and using it in multi-step planning

Abstract: Recently an influx of studies claims emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohere-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and falling in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.

Evaluating Cognitive Maps and planning in Large Language Models with CogEval
Ida Momennejad"
Hosein Hasanbeig" Microsoft Research New York, NY hosein hasannbasabbigg
Felipe Vieira Frujeri* Microsoft Redmond, WA feiipe.fujjer
Hiteshi Sharma Microsoft Redmond, WA hiteshi. sharma
Microsoft Research New York, NY idamo
Robert Osazuwa Ness Microsoft Research Redmond, WA robertness
Nebojsa Jojic Microsoft Research Redmond, WA jojic
Hamid Palangi Microsoft Research Redmond, WA hpalangi
Jonathan Larson Microsoft Research Redmond, WA jolarso
@mi amicrosoft.com
Abstract
Recently an influx of studies claims emergent cognitive abilities in large language models (LLMs). Yet, most rely anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here make two major contributions. First, we propose CogEval, cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMss. The Cogl protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo- 175B, davinci-003-175B Google Bard, Johere-xlarge-52.44 Anthropic Claude-1-52B, LLaMA-13B and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks. including hallucinations of invalid trajectories and falling in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs.. This could be because L.LMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail unrolling goal-directed trajectories based the underlying structure. Implications for application and future directions are discussed.
Introduction
Large language models (LLMs) are generatively pre-trained and display apparent competence on some cognitive tasks [8. This has led to recent surge in studies claiming LLMs have emergent human-level cognitive abilities, which may encourage applications that interact with LLMs a zero-shot or few-shot manner with expectations of human-level cognition. However, most claims of competence are based on anecdotes rather than systematic evaluation. In response, we make two contributions. First, we propose CogEval, Cognitive Ecience-Inspired [12 5 39 protocol for
Measurement and Evaluation of cognitive abilities LLMs (Figure [ top), such as planning, theory
'Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of mind, causal inference, or other abilities. Second, we apply this evaluation protocol to the domain of cognitive maps and plamning, and systematically evaluate these capacities across eight LLMs. We build our task prompts according to established human experiments, but our goal is not a comparison with human performance nor any assumptions of L.LMs being "human-like" [27]. We evaluate L.L.s'' functional as opposed to formal linguistic abilities 221. and by that we have both functionalist and multtille-rallizablity based notion of cognitive ability (9] in mind.
We investigated whether LLMs (OpenAI GPT-4, GPT-3.5-175B, and davinci-003-175B. Google Bard, Cohere-522 -5.44B, Anthropic Claude-1-52B, LLaMA-13B. and Alpaca-7B) understand the latent structure of planning problems (cognitive maps). We hypothesized that failure in planning may relate to cognitive map deficits. To address these questions, we followed the CogEval protocol (Figure [T]. First, we operationalized the latent ability (cognitive map and planning) in terms of multiple tasks with variations in three factors: (a) the latent structure of the tasks' environment (different Markov decision processes (MDPs) or graphs), (b) the domain (spatial vs. social ties vs. object relations), and (c) multiple planning tasks for each latent graph structure (c.f. Section 2 for detail). These domains were selected due to their prevalence in everyday problems as well the cognitive science literature on cognitive maps 4 35]. We then generated repeated measurements across small and large LLMs (c.f. Section 2 for choice of LLMs) and conducted statistical analysis to compare the results. We found that LLMs only show apparent competence in simpler tasks, where route memorization was sufficient to find solution, but fail closer systematic observation. Our evidence suggests against oft-oof-he-bbo emergent planning capacities in recently introduced LLMs.
What is a cognitive map? A cognitive map is a representation of latent relational structures that underlies a task or environment, and facilitates planning, reasoning, and inference in biological and artificial problems [46] 4 226 [7]. The concept originated from Tolman's 's latent learning experiments,
demonstrating rodents' ability to learn maze structures without rewards [46). This challenged the dominant behaviorist view that learning only occurs with reinforcement; and paved the way for a cognitive revolution. Decades later, discoveries of hippocampal place cells [3 30 32 and entorhinal cortex grid cells [13] 15 22), together referred to as "the brain's 's GPS," " further substantiated cognitive maps and earned the 2014 Nobel Prize M. Cognitive maps have since been studied behaviorally, computationally. and neurally; revealing that multi-step, multi-scale, and compressed neural representations are crucial for inference in both memory and planning [4] (6) 77. Over the past decades, number of reinforcement learning (RL) and deep neural network models have been proposed to capture the computations involved in cognitive maps and planning in the hippocampus and the prefrontal cortex of humans, rodents, bats, monkeys, and birds
7]
Why would LLMs plan with a cognitive map? It has been suggested that the transformer architecture and its learned representations, which lie at the heart of modern LLMs, are comparable to the hippocampus of the brain and the representations it learns (55). Other studies show that GPT-3 is capable of event segmentation of narrative transcripts similar to human evaluators [23], and evaluate some cognitive capacities of GPT-3 using cognitive science and psychological methods applied in the evaluation of human cognition 5 38 48 [66.. Other cognitive scientists have distinguished formal linguistic ability (e.g., the ability to form grammatically correct sentences) from funftionaaal cognitive capacities (e.g., theory of mind, sequential planning, etc) and call for meticulous evaluation of LLMs' functional competence without conflating them with their formal linguistic competence much like the dissociation of language and thought (2I). Taken together, these studies raise the hypothesis that LLMs would be able to extract and use cognitive maps from text, and second, that LLMs' failure in capturing cognitive maps could underlie failure modes in planning.
To test these hypotheses, we designed prompts to measure behavioral signatures of extraction and use of cognitive maps in set of tasks adapted from existing human behavioral experiments 25 24 28 34 B6]. We operationalized cognitive maps and planning with a number of tasks (Figure with variations in environmental structures graphs, varying items from different domains (spatial, social, object relations), and across a number of different conditions (e.g., value- planning, reward and transition revaluation, shorteut, and detour).
Notably, the corresponding human experiments that inspired our prompts were never in linguistic form, and this is the first adaptation of them to prompts to the best of our knowledge. This is an important consideration since contamination of the training data with the test set one of the most challenging obstacles to testing LLM capacities. To prevent any possible contamination, we avoided BIG- BIG-bench [40), which has been flagged by OpenAl for contamination (2), and a planning benchmark
2

---

Operationalization
Task & prompt generation
Data generation & evaluation
Graph structure
Latent abilitt (cognitiiee map, planning, etc)
Task prompt
Item domain
measurement
30 x prompt
Response Evaluation
Results
Tasks/
Conditions
Statistics
Instruction
Task 2
Task
Latent structures of environments (with varying domains)
A
B
D
E
Example task prompt Graph: A, spatial
Imagine a building with six rooms. From the lobby have two choices, you can go to room or room 2. You enter room and room leads to room 5, There's chest in room 5. You open it and there's a 10 dollars, but you do not take any money. Then you exit and start over. This time in the lobby you choose foom which has door to room 4, and room 4 has door that leads to toom 6, You find chest with 50 dollars in room 6, but you do not take any money.
enterfoom 1, at the other end of room there's door that leads to room
Cond: Value planning
You return lobby. You will only be able to choose one path that leads to the most money. Which room from the lobby will lead the path where one can make the most money?
The CogEval protocol, task structure, and example task prompt. (top) In the CogEval protocol, a latent ability can be evaluated by first, being operationalized as tasks, and second, be measured multiple times and with variations and controls. We followed this protocol to evaluate cognitive map and planning. To To robustly evaluate these abilities, multiple task prompts were generated with varying task structures (graph), the item domains (e.g., spatial social), and task conditions (e.g., value-based path, detour). L.M responses were generated 30 times per task prompt and temperature for the three OpenAl models studied in this work and once per task and temperature for other LLMs. The results were compared across task configurations, LLMs, and temperatures using statistical analysis. (middle) The prompts' underlying task structures were six graphs based on human experiments. A: simple line graph from (25). B: simple tree graphs based on [24]. C: graph A with double depth and stochastic transitions. D. E, and F represent community graphs from (36 and [34] respectively. (bottom) An example prompt for graph A. This procedure evaluates planning behavior value-based navigation (see T..le The colored transitions in the figure are for clarity, showing different stages of the latent transition structure (cognitive map or graph).
Figure 1:
as both pre- GPT-4 and raise data contamination issues. Here we introduce and generate novel prompts inspired by human experiments with established validity in cognitive science. To our knowledge, systematic evaluation of planning and cognitive map capacities in GPT-4 and comparison to other L.LMs remain unexplored. what follows we elaborate on a protocol and two related experiments to address this.
for GPT-3 49

---

2 Methods
The CogEval protocol. In order to evaluate reapreivemap-rrllatedd planning and navigation in LLMs, we propose and use the CogEval protocol (Figure[) Please note that CogEval is not benchmark nor limited to cognitive maps, it is general protocol for evaluating any cognitive capacity, such as planning, theory of mind, causal reasoning, etc. As an example, here we have applied it to the domain of cognitive maps and planning.
CogEval adheres to four methodological best practices suggested by cognitive scientists 12 First, the latent construct or abiliry: here we evaluate cognitive maps, which are representations that capture a model of the task structure, and adaptive planning, which requires an internal representation of task structures (similar model-based RL [2] or task-oriented model-free RL [96. Second, operationalization with construct validity: we operationalize planning ability by generating unique variations of established experimental tasks that measure the comprehension and use of cognitive maps in multi-step planning behavior [25] 36 24]. Third, multiple tasks and multiple response generations: we generated many tasks for each condition of interest varying graph structure, and domain (spatial with ordered states such as room numbers, spatial with unordered states, social ties, object relations). Most task conditions include partial change in the environment to test adaptive planning (e.g., changing the location of rewards or the structure of the environment, see Table [ Collectively, these tasks allow us to robustly measure the latent construct: cognitive map and planning ability. Fourth, including multiple task conditions allows us to control for multiple factors when making inference about the ability.
Thus, we evaluate the construct using multiple environments with different graph structures (based on existing human experiments cognitive maps [25 36 24], see graphs in Figure D. controlling for robustness to variations in graphs, task conditions, and item domains (e.g., rooms, people, objects, random letters), using multiple generations (30 generations per condition), and across different
temperatures (0, 0.5, and 1).
LLMs evaluated. We compared the following LLMs: GPT-4-* (2) GPT-3.5-turbo-177 175B 33]
text- Sott-Dainnci--1177 [6] (Azure OpenAl API), Bard- * (45). Anthropic Claude- 1-52B [3] LLaMA- 13B [47). Cohere-52 52.4B [10], Alpaca-7B 43 (nat.dev APD, where * means the number of parameters
is undisclosed.
Experiments. We conducted planning experiments to systematically compare the performance of all LLMS across task conditions created with factors of graph structure (6 graphs), domain (3 domains), and tasks (15 tasks) over 3 temperatures (0, 0.5, 1)).
2.1 A cognitive science inspired evaluation of cognitive maps and planning capacity in LLMs
We designed our experiment prioritizing robustness and control conditions. Model performance on cogmitive tasks can be influenced by various factors beyond the primary cognitive capacity, such as the specific prompts, the temperature parameter, experimental conditions (Table [ Figure [ bottom), the specific items the task is presented with or domain (c.g., spatial connections vs. social ties), and the specific relational graph underlying the problem (e.g.. this could be a graph structure such as line graphs, trees, community graphs with different size and density). For instance, perhaps an task are rooms that are numbered in order in a line graph
LLM performs better when the items (item or domain effect), or when the graph structure is finite rather than community graph with potential loops (graph effect). Thus, we implemented measures to mitigate such effects, like potential performance variations due to task item selection or its underlying graph structure. We measured the results for each combination of factors and parameters 30 times for OpenAl models (for which we had API access) and once for the remaining models with no API access. We compared the results across 10 LLMs.
Why vary temperature? ? Temperature in LLMs determines randomness in the generated response, by manipulating the probabilities of the next word a sequence. Thus, temperature can be thought of as parameter controlling the diversity of the output. A temperature of 0 results in deterministic or greedy responses with less variance (Note: OpenAl has made it known that even a temperature of 0 is not entirely deterministic). With higher temperatures, especially closer to 1, the LLM creates more diverse and varied text upon repetition, akin to exploration. While a higher temperature may be helpful for tasks that require varied responses or creativity, it could go either way for planning: on the

---

one hand, precision in planning trajectories may seem more in line with a deterministic temperature, and on the other, higher temperature leads to exploration, which may improve behavior by getting out of local minima. Repeating the experiments with varying temperature can help address its possible effect in either direction.
Statistical analysis. We evaluated the robustness of each LLM's performance by applying a statistical model of how each of the factors and their combinations contribute variance in performance. Specifically, we fit a logistic regression analysis with domain, condition, and graph types as categorical regressors, and included second and third-order interaction terms between these three terms. We made sure that each combination of domain, condition, and graph had several replicates. though the approach is robust to imbalance issues. We included model version and temperature as separate independent variables that account for technical variation distinct from our conditions of interest. See supplement for full details on analysis and results.
2.1.1 Task prompts
Navigating cognitive maps requires adaptive multi-step planning using compressed representations of the environment, not mere memorization of all routes. Thus, cognitive map experiments test flexible adaptivity to local changes in the environment to evaluate biological and reinforcement learning agents [25 25 26 [4]. Latent learning experiments found that rodents who explored a maze with no reward could quickly find the shortest route to a newly introduced reward, i.e., find an optimal policy in RL context. This was taken as their ability to learn the cognitive maps of their mazes [46], but various additional experimental conditions were then designed and evaluated to confirm that they could flexibly adapt their cognitive map and planning to local environment alterations such as reward relocation (revaluation), changes to the map (transition revaluation) (25], or the introduction of shortcuts and detours [4]. Previous research has adapted these experiments to investigating the robustness and flexibility of deep model-based RL in the face of local changes to the reward structure (LoCA), and shown that deep model-based RL agents such as Dreamer v2, muZero, and PlaNet failed flexible planning in reward revaluation scenarios [52]. Here we operationalized our tasks inspired by similar conditions in human reinforcement learning and deep MBRL experiments on learning, updating, and using cognitive maps for adaptive and flexible planning 2 52)
Importantly, the corresponding human experiments were never conducted using texts, but were presented either videos or sequence of images that human participants moved forward by choosing an action (c.g. pressing left, right, up, or down). We believe this mitigates the risks of contamination. Moreover, when possible, we set the date earlier than our pilot studies to avoid potential contamination due to our experiments in the past month. To also ensure that the model cannot infer any answers from the papers, we asked GPT-4 to explain the experimental paradigm and draw the map of the environments after providing it reference to a specific figure in a corresponding paper, and it failed. Thus, we believe our prompts have a negligible no chance of having contaminated the training sets.
Below we provide examples of task prompts for graph A and spatial domain (number ordered rooms). All prompts are available in the supplementary material and on htttps://gthuub..om//t com//oogeaal/coomaps
I. Value-based or goal-driven planning. Below is an example prompt for value-driven or goal- directed planning graph A in Figure T Success requires an understanding of the start and goal positions, comparison of the paths to find the shortest path that leads to the highest rewards, and planning multi-step navigation or traversal of the underlying graph structure of the task.
Imagine a world with six rooms. From the lobby you have two choices, room I and room 2. You enter room 1, at the end there's 's door that leads to room 3, and room 3 leads to room There's 's a chest in room 5. You open it and there's 's 10 dollars. Then you exit and start over. This time in the lobby you choose room then enter room which leads to room 6. There's a chest with 50 dollars. You return to the lobby. Which room will you choose to make the most money?
II. Transition Revaluation, after prompt I. This condition occurs when the structure of the environment (e.g., an edge of the graph or Markov decision process) locally changes, and planning requires integrating or 'piecing together' different parts of the cognitive map to update one's 'ss plan or policy.
5

---

Now you 're dropped in room 3 and the door at its end suddenty leads to room 6, and then you'l re dropped in room 4 and the door at its end suddenly leads to room 5. you return to the lobby. Which room will lead to more rewards?
III. Reward Revaluation, after prompt I. A common local change in any environment is when the location of rewards or goals change, without any changes to the map or structure of the states (or the cognitive map). This is known Reward Revaluation or retrospective revaluation of rewards (25
Now you re dropped into room 3, then you enter room 5 and the chest has 100 dollars. Then you't 're taken out, and dropped into room 4, then you enter room 6 and the chest has the same amount as before. When you return to the lobby, which room you choose to make the most reward?
V. Shortcut prompts with and without teleportation, after prompt Tolman's experiments on cognitive maps 46) included condition evaluating the animal's ability to discover shortcuts. Since the early 1990s, evaluating the ability of various Dyna architectures [42 in discovering shortcuts has been an important part of evaluating planning behavior. Below two different shortcut prompts.
In the lobby you 'e presented with a portal, and you can choose which room to teleport into. Which room do you choose to maximize rewards?
In the lobby you' re presented with new door which leads to a new room, room 7. Room 7's door leads directly to room Remember that you will only be able to choose one path that leads to the most money. Which room from the lobby will lead to the path where one can make the most money?
Detour prompts with and without Teleportation, after prompt
You enter the lobby and this time you encounter new room, room 7. Room 7's door leads to room 8, and room 8 leads room 9. From room you can teleport anywhere. You return to the lobby, and choose the room that leads to the most reward, but the door to the next room is blocked. You go back to the lobby. Which room do you choose to reach the most rewards?
You enter the lobby and this time you encounter new room, room Room 7's door leads to room 8, and room 8 leads to room 6. When you return to the lobby and choose the previous path that led to the most reward, you discover that the regular door to the room with the most money is now blocked. You go back to the lobby. You will only be able to choose path that leads to the most money. Which room from the lobby will lead to the path where one can make the most money?
3 Results
3.1 Repeated measures comparison of planning across LLMs
We evaluated out-of-the-bou emergent or native ability of different LLMs the cognitive map tasks. Table Sable[]show shows the statistical analysis highlighting the contributions each factor to logistic regression model's fit of LLM model performance. The magnitude of chi-square test statistics indicate contribution to overall model fit. Figure 2 compares the performance of all LLMs across all latent graph structures. Table shows mean and standard error for planning performance across tasks and LLMs.
The results in Table indicate that the LLM (x1 2357.87, p< ,001), graph (x1 3431.53, p < .001), condition (x1( = 2080.04, p < .001), and domain (x1( = 904.06, p < .001) each yielded significant chi-squared statistics. This means that not only did different LLMs performed

---

Table 1: Brief descriptions of the task conditions applied to varying graphs and domains
Condition
Description
Croup
vatnePatth
The optional solution find the optimal policy, or shootest path, which yiekss the hightst reward
IstepPath
The opiimal wolution a (-hop pollcy, ie., gool is affascet to the starting state
2stepPath
The optimal solution 2 stpp policy
Traversal
SetepPathh
The optinal solution is 3-step policy
nstepPPath
The optimal solution
polccy, where
the diameter the graph (lungett shortest pulh)
rewardRRovl
Upon local change the owand
the gool has changod and the eptinal solution requines finding new path Upon change reward streeure the opcimal
policy Reval
requires finding new policy
RewRoval
IransReval
Upon change the monstioo strecture, the gool some but the optimal solution roquires finding a new policy Upon local change the mrustioo stracture, the goal the same but the optimal solution zequires finding new policy in a stochastic emvironmnnt
transkeralStoccuuiie
TransReval
monteleShorteut montiishorteutCaT
Upon change in the graph scrueture, the optimal solution requires finding shortcur
Upon change in the graph senuctore, the optimal solution requires finding shrvicur, sdditiinaal CaT preempt given Upon local change the monsttion
telShhrrcct
the uptimal solution optimal solution roquires
finding shuncut seleportation shorteut using
Shortcut
I
Upon change
gropl truvuilio teleportition porial,
prompt given
nonteleDetonr
Upon change in graph scrustare, the optinal solution eequires linding detoorr
teleDetour
Upon local change the trnnmmio stracnure, the uptimal solution requires finding a detour using a teleportation step
Detour
1,0
LLMS
collection
gpi-4
Traversal
ge1-35 turbo
Dalou
text davinci 003
Shotleut
08
ropilictte alpp---l--------
spore
RewRoual
TranaRavil
arthiople-dduuee
0.4
ocharo-elarge
02
0.
094433
gal-35
davioi 183
demat Eit
dbpons 70
cahere
hand
A
B
grash
L/4b
Figure 2: Results for planning experiments in 8 LLMs. (left) Mean and standard error of perfor-
mance on all tasks for each of the different graphs (see Figure [ for graph details) across different LLMs studied in this work. (right) Mean performance compared across per main task category (see Table 3 for details).
differently, but performance varied as a result of varying graphs, domains, and conditions. Conversely,
the temperature showed a non-significant chi-squared statistic (x1 1.28. p = .53) and the interaction between the LLM and temperature was also non-significant (x1( = 10.69,
711. Noteworthy, the interactions among graph-domain, graph-condition, domain-condition. and graph-domain--onditon were all significant (all p's < .001). The interactions among graph-domain
6x (11) = 334.41, p < .001), graph-condition (x3 = 1651.33, p < ,001), Nomain-condition (39) 310.53, p < .001), and graph-domain-conditee (x2 (108) 1133.16, .001) were all significant. A full table regression coefficient estimates is included in the supplement.
In summary, while the 'temperature' and the interaction of 'LLM' and 'temperature' do not show significant effects on planning tasks, all other factors and their interactions significantly contribute to the variations in the dependent variable. These effects show that LLM performance on cognitive map and planning tasks was not robust to the graph structure of the problems, the domain, nor the task conditions, and it also varied across models (see Tables 2 and 3 and Figure 2)
3.2 Failure modes
We note three main failure modes when the task had an underlying graph with dense com- munity structure. Notably, when we probe the LLMs to list connected rooms or items as (state, actions, state) tuples, they do well (e.g., (rooml, opendoor, (rooml,opendooo: room3) is tuple for graph A in Figure []. However, when asked to do any tasks with a community graph structure using this tuple knowledge, LLMS display the following failure modes; (1) hallucinate edges that not exist, or (2) produce longer trajectories instead of shortest paths. or (3) produce trajectories that fall in loops. For example the task of finding the shortest path to a state that is cluster away, out of 30 runs GPT-4 has success rate of 0 at temperature 0. Even with changing the temperature to 0.5 or
7

---

Table 2: Step-wise contribution of adding each factor to the logistic regression fit of LLM model performance (number of successes out of max possible successes in dialog).
term
Chi-squared Stat (Deviance)
dt
p vvauu
I.I.
2357.85
<0.001
2 graph
3431 .33
5
<0.001
domain
458.74
2
<0.001
lemperature
1.28
0.53
condition
2080.04
<0.001
LLM and temperature
10.69
14
0.71
7
graph and domain
334.41
10
<0.001
graph and condition
1651.33
<0.001
domain and condition
310.53
8
<0.001
10
graph, domain, condition
1133.16
<0.001
Table 3: Mean and standard errors for planning performance across all task conditions in all 10 L.Mss. ARI scores closer to zero represent poor performance by the L.M and ARI scores reaching 1.0 represent performance matching Leiden.
spt-4-32k
RpM-35
davinci-003
claude-vi
pythia-20b
cohere
Ilama-133 13b
alpaca-7b
bard
Condition
IstepPath
0.99,
0.76.
0.52, 0.45
0.37.0.37
0.64,
0.27.
0.23,
0.41
0.05,
2stepPath
0.82, 0.35
0.73, 0.38
0.16, 0.25
0.61.041
0.67. 0.42
0.29. 0.44
0.22, 0.37
0.35, 0.47
0.25, 0,50
3stepPath
0.38
0.37.0.37
0.58, 0.43
0.27.0.11
0.35, 0.49
0.04. 0.11
0.04, 0.07
0.06, 0.20
0.11. 0.10
notteleDetour
0.55, 0.39
0.31.
0.55, 0.43
0.50, 0.41
0.51, 0.37
0.35.
0.19,
0.26, 0.38
0.29, 0.48
nonteleShortcut
0,56,
0.52. 0.39
0.49, 0.40
0.62, 0.43
0.40, 0.36
0.16,
0.11,
0,20, 0.30
0.29. 0.48
sonteleShortcutCoo
1.00,
0.09.0.07
0.58,
0.36,
0.37.
0.17.0.29
0.37, 0.37
nstepPath
0.47, 0.38
0.34
0.17.0.27
0.33.0.37
0.27, 0.42
0.05,0.11 0.11
0.06, 0.08
0.12, 0.32
0.00, 0.00
pelicyReval
0.11.
0.18.
0.13,0.04
0.28, 0.30
0.00, 0.00
0.00
0.07
0.05, 0.22
0.00,
rewardReval
0.67, 0.40
0.36
0.34, 0.25
0.48, 0.35
0.60, 0.45
0.41,
0.28,
0.33, 0.44
0.14,
teleDetour
0.35
0.34
0.53, 0.44
0.33,
0.44, 0.41
0.21.
0.23
0.23, 0.38
0.29. 0.48
teleShortcut
0.54, 0.39
0.35.
0.44.0441
0.27,
0.16.
0.16,
0.12, 0.24
0.29, 0.48
teleShortcuCool
0,50, 0,00
0.50, 0.00
0.94, 0.01
0.50, 0.50
0.39, 0,36
0.19
0.83, 0.29
0,35, 0.36
transReval
0,60,0,42
0.59. 0.40
0.39.
0.55.0.34
0.47
0.19.
0,22, 0.33
0.27, 0.37
0.08, 0.17
trans Revalttcttuut
0.73, 0.36
0.52,
0.24
0.78, 0.34
0.11, 0.19
valuePath
0.58,
0.66. 0.40
0.66, 0.39
0.44.0.41
0.49,0.46
0.31
0.27, 0.39
0.33, 0:45
0.29.0.48 0.48
1 and repeating the same 30 runs its success rate can not exceed 10%. Please refer to Figure 3 for examples of above failure modes.
Discussion and future directions
This paper makes two main contributions. First, we introduce CogEval, a cognitive science inspired protocol for systematic and robust evaluation of functional [1] cognitive abilities in LLMs. Second, we follow the CogEval protocol to evaluate multiple LLMs' native or emergent ability to extract cognitive maps for sequential planning, navigation, or graph inference. All tasks and prompts are based on non-linguistic human cognitive science experiments that we adapted into text prompts for the first time. We test for robustness of the findings by varying task conditions, graph structure, domains (spatial, social), and LLM temperature. Our systematic evaluation reveals that while LLMs display apparent competence on some tasks in simpler graphs, they do not have tut-of-the-bos zero-shot emergent cognitive map comprehension or planning competence for other graphs.
Methodological contribution. CogEval, our proposed cognitive-science inspired protocol [12 for systematic evaluation of LLMs, affords the following contributions. (1) We avoid the reuse of contam- inated standardized benchmarks by creating novel prompts based non-text-based experiments that are known to evaluate cognitive maps and planning in humans, animals, and RL. (2) We use nultiple tasks to probe the cognitive constructs (cognitive maps and planning) and repeat each interaction multiple times and across different temperatures. (3) We use statistical analysis to evaluate the robust- ness and reliability of each effect, with three main factors of graph structure, item domain (spatial vs. social), and task condition (c.g., value-based decision making, shortcut, detour, see Table I]. (4) We employ chain of thought and instruction prompts to evaluate the limits of out-of-the-box cognitive abilities of LLMs (Supplementary Experiment 2), and (5) analyze and categorize different types of failure modes. Please note that CogEval is not benchmark nor limited to evaluating cognitive maps and planning, it is a general protocol for evaluating any cognitive capacity in LLMs. As an example, in this paper we have applied it to the domain of cognitive maps and planning.
No evidence for understanding cognitive maps or planning. Our systematic and incremental evalu- ations reveal limited to no cognitive map capacities in the current generation of LLMs including
8

---

What is the shortest path from room to room 12?
What is the shortest path from room room
What is the shortest path from room to room
Answer: 1, 7, 10,
Answer: 13,
7,
GPT-4 temp=0, graph n-step
GPT-35, temp=1, graph 1-step
GPT-4, temp=0, graph n-step
Figure 3: Examples of three failure modes. (left) Edge hallucination. (middle) Failure at finding a 1-step policy within the same cluster. (right) Failure multi-hop path by both falling in loop and hallucinating edges. In each example the blue box is the task prompt, the grey box shows the model response, and the green arrows demonstrate the correct response on the graph.
GPT-4. Specifically, we find that LLMs only show apparent competence on simple sequential infer- ence tasks where route memorization can help, and given LLMs have received all possible trajectories in the text prompt. We also observe that the sparsity of graph structure drove performance. However, when 1-step and multi-step traversal and planning require understanding the underlying relational structure of the environment graph, LLMs including GPT-4 fail by hallucinations, suboptimally long routes, or falling in loops.
How did LLMS solve the simpler tasks? Without access to full architectures or training sets, we can only form hypotheses based on our behavioral observations. We observe that LLMs do better in problems where entire trajectories are explicitly available in the text prompts, and they only need to retrieve and piece together partial changes. However, planning behavior in more complex graphs is far worse, and this not just due to graph size: performance was worse for graph B (7-node tree) than C (12 node, parallel lines). Performance the 15-node graph with 3 dense clusters worse than the 16-node (4-cluster) graph that has better cross-cluster connectivity and more paths among clusters.
These observations suggest that LLMs may fail at planning problems where they need to use the transition structure to unroll the trajectories and find the correct path, which is closer to the notion of planning in model-based RL and in cognitive science. Capturing the underlying structure and using it to unroll trajectories are quintessential to cognitive maps and planning ability. Thus, the apparent competence in simpler tasks may be due to using cached or memorized routes rather than understanding the cognitive map. planning, inference ability.
LLMs may do better in smaller and simpler graphs because the prompt already expands all the possible paths or trajectories. When there is change in the rewards or transition structure, LLMs only need to change one step an already laid out path. However, more complex graphs only the one-step connections are laid out in the prompt, but not all paths or trajectories between any given two nodes. We observed that failures significantly increase in tasks with these larger graphs with community structures, even when an LLM can list the pairwise tuples of connected states (see failure modes, F).
Interpreting the results. The experiments in the paper are not meant to be interpreted as a benchmark for planning. They probe the same construct different ways, evaluating the ability to use information about (state, action, state) tuples, e.g., (rooml, openleftdoor, room3), to piece together policies in response to task prompts. A reader may wonder why we claim that LLMs do not display emergent planning in spite of non-zen non-zero performance for some tasks in our experiments (Figure 2 We interpret the findings as such due to failure modes and various inconsistencies success cases (Figure For instance, common failure mode is generating sequences with hallucinated (state, actions, state) tuples that do not exist. For GPT-4 this constitutes over 20% 4-step plans. Another common failure mode is that they fall into loops when prompted to find the shortest path between two states (Figure 3 left and right). L.LMs sometimes even fail to identify 1-step paths or suggest multi-hop trajectories for traversing to adjacent state (Figure 3 middle).

---

These observations seem counter-intuitive given some LLMs can generate list of tuples when asked, but fail to use the tuples to make (even 1-step) valid plans. This shows that, while LLMs appear to solve planning problems when given simple routes that can be explicitly memorized, they cannot generalize from route memory solutions (trajectories directly the prompt) to using the tuples to adaptively generate branching plans. Together, these inconsistent observations are in line with the hypothesis that LLMs do not understand cognitive maps and therefore cannot consistently plan. Elsewhere propose black box architectures that improve planning performance (54). However, our findings point to a lack of out-of-the--oe zero-shot emergent planning ability.
Limitations. First, we lack knowledge of LLMs like GPT-4's architecture or training. To address this limitation, we did not use existing text-based benchmarks and instead generated novel prompts not in their training sets. Second, in the human experiments that influenced our prompts participants learn gradually, experiencing states one-by-one, only tested after they showed signs of learning, similar to model- RI. agent having the transition structure and using it for inference and planning. To address this difference, we present the environment's s structure in linguistic format. The LLM had to extract the cognitive map, identify the goal location based on instructions. and infer the policy towards the goal. Third, we do not have a human baseline for the language-bare tasks, so a comparison with human behavior on the exact language prompts remains the topic of future studies.
Implication for applications. LLMs are expected to be applied in fields like gaming, planning, and social reasoning, with tasks that require understanding the inherent relational structure of the problem from the input for flexible reasoning and planning. However, here we show various failure modes in the understanding of the underlying cognitive maps or planning abilities, including hallucination and falling in loops. Even when provided instructions and Chain of Thought (CoT) prompts like breadth- first search (BFS), we observe that GPT-4 struggles to process multi-hop paths it has not experienced (Supplementary Experiment 2). These findings suggest caution in the application of LLMs in problems that involve planning or complex structures. Below we discuss future directions that may mitigate these challenges for problems with simpler structures.
LLMs as programmable machines rather than emergent intelligence? While some regard LLMs as agents with emergent intelligence comparable to humans and animals, our findings are more consistent with the view that LLMs are programmable machines where natural language is their
programming language (0). Thus, here we evaluated planning in LLMS in a functionalist and multiple-eellzzabiiiy sense rather than making any assumptions of them being "human-like" [27]
Future directions. A future direction is to analyze embedding representations and attention in LLMs, and test hypotheses about representations underlying success and failure in planning. This mirrors how neuroscience analyzes neural data understand representations in model-based and predictive planning and decision-making (25 77. A recent study suggests a promising future for this direction [57]. Another interesting direction is to study the limits of LLMs' transitive inference using pair- wise associations [37] 34), given we observed that while some LLMs could list pairwise tuples or recognize the goal, they still struggled with planning. A further direction is to study whether the use of schemas, i.e., overused, generalized cognitive maps such as "airport" [50 22 I1 51). can improve performance on real- world scenarios by evoking helpful structures, given LLMS have shown promise with analogical reasoning tasks [53]. Finally, some have suggested ways to improve planning by augmenting LLMs with algorithms that enable executive control, an interesting direction that can contribute to the future of augmenting both larger and especially smaller language models (see [54))
LLMs need a hippocampus and prefrontal corlex. The hippocampus and prefrontal cortex in the brain extract the relational structure or cognitive maps from sequential data to flexibly plan at multiple scales 7 [66] Their functions can inspire memory, planning, and executive control augmentations to LLMs in order to mitigate failure modes such as hallucinating edges. Ongoing research shows that indeed prefrontal cortex- cortex-inspired inspired solutions can improve planning performance an LLLM-bssed architecture with multiple calls to GPT-4 [54], pointing at a promising future direction.
Summary. We introduced CogEval, a cognitive science inspired protocol for systematic and robust evaluation of LLMs. We applied CogEval to evaluate planning performance across 8 LLMs and found poor performance.
10

---

Acknowledgement
We are extremely grateful to Peter Lee, Michael Frank, John Krakauer, Joshua Tenenbaum, Paul Bennett, Alison Gopnik, and Melanie Mitchell for early feedback on methods and results. We would also like acknowledge Derek Worthen and Richard Ciapala for engineering support.

---

References
[1] The nobel prize in physiology or medicine 2014. http:///eeee11//eeeseeeeeeeeeeee
Accessed: 2023-5-10.
[2] OpenAI 2023. GPT-4 technical report. arXiv preprint
[3] Anthropic. Introducing Claude. poppduucnn-cllaude 2023. [Online].
[4] Timothy E Behrens, Timothy Н Muller, James C Whittington, Shirley Mark, Alon B Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson. What is cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2)4490-50, October 2018.
[5] Marcel Binz and Eric Schulz. Using cognitive psychology to understand GPT-3, Proceedings of the National Academy of Sciences, 120(6))22118523122) 2023.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
[7] Iva Brunec and Ida Momennejad. Predictive representations hippocampal and prefrontal hierarchies. J. Neurosci., November 2021.
[8] Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. March 2023.
[9] Rosa Cao. Multiple realizability and the spirit of functionalism. Synthese, 200 December 2022.
[10] Cohere.
Introducing Cohere.
https://txt.come wwweree.ttcoooecccoo com/
cohere- laarceeluuches--ee
-areelll--aree-bbee 2022. [Online].
[11] Delaram Farzanfar, Hugo Spiers, Morris Moscovitch, and R Shayna Rosenbaum. From cognitive maps to spatial schemas. Nat. Rev. Neurosci., 24(2):63-79, February 2023.
[12] Michael C Frank, Mika Braginsky, Julie Cachia, Nicholas Coles, Tom Hardwicke, Robert Hawkins, Maya Mathur, and Rondeline Williams. Experimentology. https:// experimentology. 1o/ 2023. Accessed: 2023-5-9.
[13] Marianne Fyhn, Sturla Molden, Menno P Witter, Edvard Moser, and May-Britt Moser. Spatial representation in the entorhinal cortex. Science, 305(5688) August 2004.
[14] Mona M Garvert, Raymond J Dolan, and Timothy Ej Behrens. A map of abstract relational knowledge the human hippocampal-enttrhinall cortex. Elife, 6, 2017.
[15] Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I Moser. Mi- crostructure of spatial map in the entorhinal cortex. Nature, 436(7052):801-800 August 2005.
[16] Hosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Logically-conntrainee neural fitted Q-iteration. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2012-2014, International Foundation for Autonomous Agents and Multiagent Systems, 2019.
[17] Hosein Hasanbeig, Yiannis Kantaros, Alessandro Abate, Daniel Kroening, George J Pappas, and Insup Lee. Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees. In Proceedings of the 58th Conference on Decision and Control, pages 5338-5343. IEEE, 2019.
[18] Hosein Hasanbeig, Daniel Kroening, and Alessandro Abate. Certified reinforcement learning with logic guidance, Artificial Intelligence, page 103949, 2023.
12

---

[19] Hosein Hasanbeig, Natasha Yogananda Jeppu, Alessandro Abate, Tom Melham, and Daniel Kroening. Symbolic task inference in deep reinforcement learning. Journal of Artificial Intelligence Research (JAIR), 2023.
[20] Ana Jojie, Zhen Wang, and Nebojsa Jojie. GPT is becoming a turing machine: Here are some ways to program it. March 2023.
[21] Kyle Mahowald, Anna Ivanova, Idan Blank, Nancy Kanwisher, Joshua Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: cognitive perspective. January 2023.
[22] Rolando Masís-Obando Kenneth A Norman, and Christopher Baldassano. Schema representa- tions in distinct brain networks support narrative memory during encoding and retrieval. Elife, 11, April 2022.
[23] Sebastian Michelmann, Manoj Kumar, Kenneth A Norman, and Mariya Toneva. Large language models segment narrative events similarly to humans. ArXiv, January 2023.
[24] I Momennejad, A R Otto, N Daw, and K A Norman. Offline replay supports planning in human reinforcement learning. Elife, 2018.
[25] I Momennejad, E M Russek, J H Cheong, M M Botvinick, N D Daw, and S Gershman. The successor representation in human reinforcement learning. Nat Hum Behav, 1(9):680-692. September 2017.
[26] Ida Momennejad. Learning structures: Predictive representations, replay, and generalization. Current Opinion in Behavioral Sciences, 32:155-166, April 2020.
[27] Ida Momennejad. A rubric for human-like agents and NeuroAI. Philos. Trans. R. Soc. Lond. Biol. Sci., (78(1866)200210444 December 2022.
[28] Ida Momennejad, Ajua Duker, and Alin Coman. Bridge ties bind collective memories. Nat. Commun., 10(1):1578, April 2019.
[29] Edvard Moser, Emilio Kropff, and May-Britt Moser. Place cells, grid cells, and the brain's spatial representation system. Annu. Rev. Neurosci., 31:69-89, 2008
[30] Keefe. Place units the hippocampus of the freely moving rat. Exp. Neurol., 51(1):88-109 April 1976.
[31] J Keefe and J Dostrovsky. The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat. Brain Res., 34(1):171-17) November 1971.
[32] John O' Keefe and Lynn Nadel. The Hippocampus as a Cognitive Map. Oxford: Clarendon Press, 1978.
[33] Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[34] Athula Pudhiyidath, Neal W Morton, Rodrigo Viveros Duran, Anna Schapiro, Ida Momenne- jad, Demitrius M Linojosa-Rowlane Robert J Molitor, and Alison Preston. Representations of temporal community structure in hippocampus and precuneus predict inductive reasoning decisions. Cogn. Neurosci., 34(10):1736-174) 1760. September 2022.
[35] Matthew Schafer and Daniela Schiller. Navigating social space. Neuron, 100(2):476-48) October 2018
[36] Anna C Schapiro, Timothy T Rogers, Natalia I Cordova, Nicholas B Turk-Broone,, and Matthew M Botvinick. Neural representations of events arise from temporal community structure. Nat. Neurosci., 1664))486-492, April 2013.
13

---

[37] Anna C Schapiro, Nicholas B Burk-Browne, Matthew M Botvinick, and Kenneth A Norman. Complementary learning systems within the hippocampus: neural network modelling approach to reconciling episodic memory with statistical learning. Philos. Trans. R. Soc. Lond. B Biol. Sci., 372(111))) 1), 2017.
[38] Richard Shiffrin and Melanie Mitchell. Probing the psychology of Al models. Proc. Natl Acad. Sci. U. S. A., 120( 20010))x2300963100 March 2023.
[39] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. March 2023.
[40]
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alona Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexan- der W Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alícia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S Iyer, An- ders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B Ryan Roberts, Bao Sheng Loe, Barret Zoph. Barthomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekei. Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt. Christo- pher D Manning, Christopher Potts, Cindy Ramirez, Clara E Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan. Dimitri Coelho Mollo, Diyi Yang. Dong-Ho Ho Lee, Ekaterina Shutova. Ekin Dogus Cubuk, Elad Segal, Elcanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellic Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang. Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambat- tista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galjasevvcc Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong. Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac. James B Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jillian Tang. Joan Waweru, John Burden, John Miller, John U Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo. Joseph Boudeman, Joseph Jones, Joshua B Tenenbaum, Joshua S Rule, Joyce Chua, Kamil Kanclerz, Karen Liveseu, Karl Krauth, Karthik Gopalakr- ishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble. Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel. Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Faroogi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Varma T Mukund, Nanyun Peng,
14

---

Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar. Niveditha Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy I Liang. Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Bares,, Rif A Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi. Ryan Lee, Ryan Stovall, Ryan Techan, Rylan Yang, Sahib Singh, Saif M Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruet- ter, Samuel R Bowman, Samuel S Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sehastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu. Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanis- las Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T Piantadosi, Stuart M Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu. Tariq Ali. Tatsu Hashimoto, Te-Lin Wu, Théo Deshordes, Theodore Rothschild, Thomas Phan, Tianle Wang. Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J Wang. Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. June 2022.
[41] K Stachenfeld, M Botvinick, and others. Design principles of the hippocampal cognitive map. Adv. Neural Inf. Process. Syst., 2014.
[42] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, November 2018
[43] Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cedtructionfoollwwee LLaMA model, 2023.
[44] Rita Morais Tavares, Avi Mendelsohn, Yael Grossman, Christian Hamilton Williams, Matthew Shapiro, Yaacov Trope, and Daniela Schiller. A map for social navigation in the human brain. Neuron, 87(1):331-243, July 2015.
[45] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201 220.0..90,,.,,99, 2022.
[46] Tolman. Cognitive maps in rats and men. Psychol. Rev,, 55(4)) July 1948.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ama Lachaux, Timo- thée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint 13X1V 2023.
[48] Tomer Ullman. Large language models fail on trivial alterations Theory-of-Mind tasks. February 2023.
[49] Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for L.MSs on planning and reasoning about change). June 2022.
15

---

[50] Marlicke T R van Kesteren, Sarah F Beul, Atsuko Takashima, Richard N Henson, Dirk Ruiter, and Guillén Fernández. Differential roles for medial prefrontal and medial temporal cortices in schema- echema-dependent encoding: from congruent to incongruent. Neuropsychologia, 51(12):2352-2333 October 2013.
[51] Marlicke Tina Renée van Kesteren and Martijn Meeter. How to optimize knowledge construction in the brain. NPJ Sci Learn, 5:5, May 2020.
[52] Yi Wan, Ali and Harm van Seijen. Towards evaluating adaptivity of Model-Based reinforcement learning methods. April 2022.
Janarthanan Rajendran, Ida Momennejad, Sarath Chandar,
[53] Taylor Webb, Keith Holyoak, and Hongjing Lu. Emergent analogical reasoning large language models. December 2022.
[54] Taylor Webb, Shanka Subhra Mondal, Chi Wang, Brian Krabach, and Ida Momennejad. A prefrontal cortex-inspired architecture for planning in large language models, 2023.
[55] James Whittington, Joseph Warren, and Timothy E J Behrens. Relating transformers to models and neural representations of the hippocampal formation. December 2021.
[56] Eunice Yiu, Eliza Kosoy, and Alison Gopnik. Imitation versus innovation: What children can do that large language and language-and-viiinn models cannot (yet)? May 2023.
[57] Safoora Yousefi, Leo Betthauser, Hosein Hasanbeig, Akanksha Saran, Raphaël Millière, and Ida Momennejad. In-context learning in large language models: A cuuroscience-isspiin analysis of representations, 2023.
16


Reviews:
[{"review_id": "xzkVQhHemf", "reviewer": "Reviewer_UmDW", "summary": "This paper proposes an evaluation of large language models with respect to their ability to solve problems that require use of latent cognitive maps. Evaluation focuses on different underlying graph structures, and the influence of chain-of-thought inference on performance.", "questions": "* I am not completely familiar with the term \"cognitive maps\", and I'm not sure if this more of a metaphor, or usually applied to actual spatial reasoning tasks (hence \"map\"). Since the examples in the paper are about navigation, I am wondering if this means the domains studied are mostly about literal maps, or if there are other domains usually studied with the framework of cognitive maps. Are there other domains covered in this study? If so, what are they?\n* If there are different domains, how does performance vary across them?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* I really liked the very clear outline of the motivation of the research design in the introduction of Section 2.\n* The experiments are extremely thorough, and the goal of designing experiments with statistical robustness in mind is good.\n* Good discussion around the capabilities and limitations of LLMs", "weaknesses": "Some of the presentation could be refined:\n* The description of Figure 1 is somewhat difficult to understand. References to future aspects of the paper (e.g., \"Experiment 3\") are undefined, which makes it more difficult to understand.\n* The figures / tables should appear closer to where they are referenced in the text.\n* I'd suggest reordering 2.1, so that the tasks (i.e., maze learning) are described before the experimental setup.\n* Some details could use more context. e.g., what is temperature? What are the graph structures A/B/C... etc? \n* The discussion on BFS/DFS prompting should be moved to the experimental setup section (2.3). I also didn't quite understand the distinction between these two; an example would help.\n* Formatting of Figure 3 can be improved\n* What is \"dialogue\" referring to throughout the paper? I don't believe an actual multi-turn dialogue is taking place during the evaluation\n* Text of Table 2 is really tiny", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive evaluation of our work as well as their careful and constructive questions and suggestions. Please see our responses below. We hope that we have addressed any concerns and are happy to address further questions in the discussion period as well.\n\nWeaknesses:\n\n1.\tWe have now improved Figure 1 caption for clarity. \n2.\tGreat suggestion, we reorganized the figures and hope to show it in the camera ready.\n3.\tAppreciated, we have now accommodated this.\n4.\tTemperature in LLMs determines randomness in the generated response through the softmax function. This manipulates the probabilities of the next word in a sequence. Because of this, temperature can be thought of as a parameter controlling the diversity of the output. When temperature is set to 0, this results in deterministic or greedy responses with less variance (Note: OpenAI has made it known that even temperature=0 is not entirely deterministic, though this is as close as you can get). Higher temperatures, especially closer to 1, create more diverse and varied text upon repetition. While this is helpful for tasks that may require varied responses or creativity, it’s not great for responses that require precision such as planning trajectories. We are more than happy to add this to the supplementary or integrate a summary in the main text if the reviewer sees fit.\nThe graph structures are the latent structures of the problems discussed in the paper. \n5.\tThank you. Breadth First Search and Depth First Search instructions are provided in the supplementary (lines 55-75), and we have pasted them below to address the reviewer’s question. \n\n5.1.\tBFS (Breadth First Search) instruction: \n“Think carefully before you respond. You can try using Breadth-first search (BFS), it is a graph traversal algorithm that visits all the vertices of a graph in breadth-first order, starting from a given source vertex. In BFS, vertices are visited in layers, where the vertices at distance 1 from the source vertex are visited first, followed by the vertices at distance 2, and so on. BFS uses a queue data structure to keep track of the vertices to be visited, and it ensures that no vertex is visited more than once. BFS is useful for finding the shortest path between two vertices in an unweighted graph, or for exploring all the vertices in a graph.”\n\n5.2.\tDFS (Depth First Search) instruction: \n“Think carefully before you respond. You can try using Depth-first search (DFS), it is a graph traversal algorithm that visits all the vertices of a graph in depth-first order, starting from a given source vertex. In DFS, the algorithm traverses as far as possible along each branch before backtracking. DFS uses a stack data structure to keep track of the vertices to be visited, and it ensures that all vertices connected to a visited vertex are explored before backtracking. DFS is useful for finding cycles in a graph, for exploring all the vertices in a graph, or for finding a path between two vertices. However, unlike BFS, DFS does not guarantee that the shortest path is found.”\n\n6.\tWe agree, and have updated Figure 3 accordingly, which you can find in the PDF. Additionally, we have audited some of the prompts and can provide further explanation as needed. All prompts can be found in the interactive tool with the prompts Chatbot Visualization (cogeval.github.io).\n7.\tWhile we don’t use the term \"dialogue\" in the manuscript, in tasks that test robustness of LLMs to changes in rewards or transition structures, there is a 2-step question. Following Momennejad et al. 2017, 2018, First, the graph is explained and the LLM is probed for the optimal policy, then a partial change is described and the LLMs is probed a second time for the optimal policy. The correct response to the second question requires the integration of information in the first and the second prompts.\n8.\tWe have now changed the table size. \nThanks again for the helpful suggestions, and we are more than happy to address any further feedback.\n\nQuestions:\n\n1.\tThank you for the question. In lines 39-51 in the original manuscript, we discuss what a cognitive map is and in 52-62 we explain why LLMs would show that capacity. Briefly, the term was coined in cognitive science by Tolman’s 1948, where he reviewed decades of planning and navigation research using mazes. It refers to the relational structures that are stored in memory, representing a map of the state-space (which can be spatial or non-spatial, e.g., social relations) that is held in the mind rather than externally, hence the phrase \"cognitive map\". The specific structure of this map, whether it is one-step, multi-step, multi-scale, etc. has all been a matter of research over the past century. Since Tolman, the neural underpinnings of cognitive maps won a Nobel prize and as Tolman intended it, many studies discuss cognitive maps in terms of non-spatial maps (e.g., social maps, associative relational structures) as well. Many models have been proposed, with representation learning and RL being of special relevance, including debates over whether the map is Euclidean or not. We are happy to clarify further if lines 39-62 (and references 5, 1, 9) are unclear or add further sections in the supplementary.\n2.\tThank you for this helpful question. We have now provided a figure to address this constructive question in the attached PDF.  \n"}, {"review_id": "SfQLBZ2I9Q", "reviewer": "Reviewer_ydCJ", "summary": "The paper presents CogEval, a set of best practices ported from cognitive science on how to do behavioral evaluations. The authors also transcribe new tasks from human reinforcement learning and planning into text, such that LLMs can be tested on them. On these tasks, the authors do not observe evidence for an emergent capability for planning in large language models.", "questions": "- Are you planning to release these tasks transcribed to text as a new testbed?\n- Does transcribing these tasks to text make them harder? Perhaps a vision-and-language model like GPT-4 or Flamingo might perform better on pictures of the task, in which case, perhaps your conclusions would be different. \n- Why is “Measurement and Evaluation for Large Language Models” capitalized in line 6?\n\nNits:\n- Line 14: Stay in active voice here for consistency?\n- Figure 3: Can you make this less blurry?", "limitations": "The authors have adequately addressed limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The presentation of CogEval is clear and potentially useful for the ML community. I’m excited to use CogEval in my own work.\n- Thorough and thoughtful discussion.\n- Exhaustive experimental setup—a feature perhaps enabled by the principled CogEval framework!\n- The paper is lucid overall and easy to read.", "weaknesses": "There’s some mild overselling in the paper, given the empirical results, which only demonstrate the failure of an existence proof.  Defining the conditions under which we can declare definitively that there is no emergent planning. However, given that we have not rigorously defined these conditions, for example, “No Emergent Planning” in the title seems too strong. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful that the reviewer finds CogEval clear and thorough and are delighted to read they may potentially try it or use it in their work. We hope that we have addressed their helpful and constructive suggestions below and are more than happy to address any further feedback.\n\nWeaknesses:\n•\tWe appreciate the reviewer’s suggestion and find it a fair assessment. In the camera-ready version, we plan to remove “no emergent planning” from the title. We have already begun editing the text to make the conclusions more nuanced in various sections. Our interim new title is “Evaluating Cognitive Maps and Planning in Large Language Models with CogEval”.\n\nQuestions:\n•\tAll the prompts are already available in the anonymous link we provided for the reviewers.\n•\t•\tThis is an excellent question, and we hope very much to test this in the future. Given our multiple tests and checks we know that GPT4 can extract the correct tuples from the instructions, both spatial and social. However, GPT4 fails at using these same tuples during planning with three main failure modes, please see Supplementary figure 2, also included in the rebuttal PDF. If accepted, this figure will be moved to the main manuscript. These failure modes include 1) hallucinating edges that are non-existent, 2) taking unnecessary moves that unnecessarily lengthen the path, even when there’s a 1-step transition available that GPT4 has captured from the description, and 3) falling into loops. The question of the influence of a visual LFM on each and all these failure modes is surely of interest. Would visual reasoning improve all, some, or worsen? We think pursuing the reviewer's suggestions could be an excellent follow-up study.\n•\tThe reviewer is correct, this is inherited from an older abbreviation prior to CogEval. We have fixed it per the reviewer’s suggestion.\n\n"}, {"review_id": "2TTUx6YG3K", "reviewer": "Reviewer_Vmb5", "summary": "This paper evaluates LLMs on a set of tasks that could be solved by cognitive maps, such as goal-oriented planning, or incorporating shortcuts. The work finds that LLMs generally perform poorly at these tasks, and their performance is affected by features such as graph sparsity. ", "questions": "* In the discussion, what explicit experiment is referred to by \"We observe that LLMs do better in problems where the entire trajectories are explicitly available in the text prompts, and they only need to piece together partial changes\"?", "limitations": "See weaknesses; I believe that the limitations of the experimental paradigm not matching the inspiration are not fully discussed, and that more generally the conclusions are not fully supported by the experiments presented.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "* The paper is admirably thorough with experiments and analyses:\n    - Assessing a range of LLMs (including varying parameters such as temperature).\n    - Evaluating across a range of different graph structures, task paradigms, etc.\n    - Creating new stimuli to avoid dataset contamination. \n    - Performing regression analyses to determine how different features contribute to model performance, and describing the failure modes observed.\n    - These thorough results and analyses suggest that the conclusions are likely to roughly generalize to some extent, and help the reader to understand which features will affect performance.\n* The results are interesting, there are a variety of patterns that could be investigated further.", "weaknesses": "My primary concern is that the overall framing of the paper is misleading. In particular, the work is motivated with references to the cognitive and neuroscience literature on cognitive maps. This work performs  draw strong conclusions from its experiments such as \"no evidence for understanding cognitive maps or planning.\" Are conclusions such as these justified?\n\nA key issue in analyzing AI in comparison to human or animal capabilities is determining where a performance failure originates: is it a lack of an underlying capability (such as the ability to form a cognitive map), or a more superficial performance issue? Several recent papers have emphasized this point from a cognitive science perspective, and argued that it is essential to ensure fair comparisons between AI and natural intelligence to draw accurate conclusions (https://www.pnas.org/doi/abs/10.1073/pnas.1905334117; for LLMs specifically see: https://arxiv.org/abs/2210.15303).\n\nIn that context, it's worth noting that the animal and human experiments cited involved a great deal more experience before the map was tested than the present experiments do. For example, Tolman's latent learning experiments involved the rats fully exploring the maze for multiple days before they were tested with a food reward at the end (and even then, performance continued to improve well after the rewards were first introduced). Or Schapiro's temporal community structure paper involved half an hour of exposure to transitions from the graph; that is, thousands of transitions from a graph with only 15 nodes. This a much denser sampling of experience than the current LLM experiments afford, and it is quite possible that some degree of repeated experience contributes to the ability of natural intelligences to form a cognitive map. The difference in learning conditions is briefly mentioned to in the limitations, but is quickly dismissed; however, the difference in experimental conditions is a fundamental challenge to concluding that LLMs are failing to form cognitive maps like animals/humans do.\n\nLikewise, it is typical in cognitive science to report comparisons to chance-level performance, and an underlying ability is usually inferred from better-than-chance performance, even if that performance is imperfect. For example, in Tolman & colleague's studies, the rats continued to improve for several days after the reward was introduced (that is, their paths were not optimal on the first test), and the rats were clearly stitching together trajectories that they had observed (since there was only a single route through the maze, everything else lead to dead ends), but we still interpret their performance as showing latent learning. It would be useful to report chance-level performance across all conditions, and, in the case that a model performs better than chance across a broad range of conditions, that would suggest some underlying ability, even if it is imperfect. For example, GPT-4's performance seems reasonably high across most conditions in Table 2 (though certainly imperfect).\n\nIn addition, explicit comparisons to human performance on these tasks (presented exactly as they are presented to the language model), would strengthen the claim that language models are failing in a fundamental way that humans or animals would not.\n\nThese points seem critical to the overall framing of the paper, and also to much of the discussion. I therefore think the paper would be substantially improved by:\n1) providing a more nuanced framing of the very interesting results, that suggests they emphasize some limitations of planning in LLMs, without making overly strong claims such as \"no emergent planning\" or \"no evidence of cognitive maps\"\n2) Providing explicit comparisons to chance-level (and ideally human) performance to help contextualize the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough engagement with our work and thoughtful notes on the strengths and constructive suggestions.\nWeaknesses:\n\n1- Framing. We agree that nuanced language is more productive. Our goal was to test zero-shot planning behavior in LLMs. If accepted the camera-ready is “Evaluating Cognitive Maps and Planning in Large Language Models with CogEval”, removing \"no emergent planning\". We have also introduced nuance in various sections. We referred to “no emergent planning”, given LLM failure modes suggest they don’t “understand” how to use a cognitive map for planning, e.g., they hallucinate edges that don't exist, they fall into loops, or fail to use one-step paths (Supplementary figure 3, now in the main text). We did not intend to create a benchmark to compare against human behavior or chance but to compare planning across LLMs. We evaluate robustness in the face of changes against the baseline of simpler tasks and identify major failure modes.\n\n2- Comparison to rodent/human experiments: We are thrilled that the reviewer has a deep understanding of cognitive maps and rodent experimental work. \nWe'd like to note that our goal here was to test whether LLMs have *zero-shot* planning behavior. While we agree with making the wording more nuanced, please note that our goal was not a comparison to human behavior.\nMoreover, comparison to rodent learning would not be fair given rodents can learn but LLMs with frozen weights can’t learn, and zero-shot planning with language is inherently not comparable to rodent learning.\nMeanwhile, in ongoing studies we are using non-zero-shot approaches to LLMs, in context learning (ICL) & scratchpad/CoT, to improve planning in LLMs. That said, we believe that these studies are not in the scope of the present paper. We acknowledge that investigating all these tasks in human behavior would be a fantastic but separate future contribution.  We hope the reviewer appreciates our response.\n\n3- Comparison to chance: \nWe thank the reviewer for this suggestion. We agree that providing an impression of a chance baseline to LLM performance would be an interesting contribution. However, we believe doing so would be non-trivial given our specific design.\n\n3.1. defining chance is non-trivial: The examples the reviewer provided rely on binary & multiple-choice responses, where defining chance at 50% makes sense. However, it is less clear how to define chance level when we have asked for a trajectory to optimal reward. If we had used multiple-choice responses, we could quantify random chance as 1/k where k was the number of choices. We pose open questions to the LLMs and evaluate their responses. It is not clear how to enumerate a sample space of possible answers.\nHowever, we believe that comparison of the same LLM across graphs and domains, as well as the comparison of different LLMs' performances on the same tasks/graphs/domains offer ample novel contributions to the field and satisfy cognitive- and neuro- science standards (some authors have worked in cog sci for decades). However, we have considered a number of possibilities below in hopes to align the reviewer’s thoughts with our thinking process on the non-triviality of a chance level.\nConsider traversal from a given node to a destination node. For given a graph, one possibility is to calculate all pairs of “shortest paths”, but this reduces down to the algorithm used for betweenness centrality (which is n^2 * (log n) to compute) and it would produce an exponential number of possible paths. This means that the likelihood of randomly choosing the correct path will almost always be zero, which makes the use of random chance less meaningful.\n\n3.2. We could also use a random walk algorithm with the following constraints:\n•\tRandomly walk through the graph and report a success if the goal node is reached, failure otherwise\n•\tNo backtracking & no revisiting nodes.  If a previously visited node is encountered again in the random walk, terminate walk as a failure. \n•\tApply uniform transition probabilities based on what transitions are permitted in each condition.\n•\tRun repeated random walks, count the proportion of successes as a Monte Carlo estimator of the probability of success by random chance.\nTo illustrate, for the value-based planning conditions and transition revaluation conditions on graph A, a random walker that didn’t backtrack would have a *50% chance* of achieving the goal (the room with the most reward). However, once we introduce the shortcut/detour/teleportation conditions, the state graph becomes complicated. The simulation would become more complex for more complex graphs but this random walk Monte Carlo estimation approach for random success probability may be a sound baseline for evaluating LLM performance.\n\nQuestions: We appreciate the opportunity to expand on this. \nIn smaller graphs, the prompt already expands all the possible paths or trajectories. When there’s a change in the rewards or transition structure, the LLM only needs to change one thing in an already laid out path. However, in more clustered graphs only the one-step connections are laid out in the prompt, but not all paths or trajectories between any given two nodes. This means that the LLM needs to use the transition structure to unroll the trajectories and find the correct path, which is closer to the notion of planning in model-based RL and in cognitive science. \nAn observation that speaks to this is that performance on larger graphs is far worse than the smaller ones. It is not just the graph size, LLMs often perform worse on the graph with 15 nodes and 3 dense clusters compared to the 16-node (4-cluster) graph that has more nodes, but better cross-cluster connectivity. The difference: there are fewer paths among clusters in the Schapiro graph, making “planning” more relevant here. The difference is robust to prompt variation. Taken together, these findings support the claim and we are happy to discuss further."}, {"review_id": "cmTKxGBaaM", "reviewer": "Reviewer_xxkR", "summary": "This paper describes a test battery to test for the emergence of cognitive maps and planning abilities in LLM. The tests are based on existing cogsci experiments converted into text prompts. For example, to test for a planning ability, the prompt first describes an apartment layout, and then asks the model to plan a path through several rooms to retrieve something. The authors show that LLM are poor at completing such tasks.", "questions": "1.\n\"\nWe evaluated the robustness of our findings by regenerating the results for each combination of factors\nand parameters multiple times and applying a statistical model of how each of these factors contribute\nto variance in LLM performance.\n\"\nWhat does this mean? Where is the exact experimental design? How many trials were simulated? Which conditions were evaluated?\n\n2. \nWhere is Experiment design for Experiment 1? Where are the graphs?\n\n3.\n\"198 For the graph community block model, example graphs are shown in Figure 2 ...\"\nWhat does this mean? Does Figure 2 show all graphs, or a subset of them? If this is a subset, then how many graphs of each kind were shown?\nWhere is the \"community block model\" described?", "limitations": "I can not make out what was done exactly, but am happy to read the next draft. The experiments and statistical analysis need to be described so that people could reproduce them.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "I think the adaptation of cogsci planning experiments to text prompts is a great idea. There needs to be more tests like this to map the scope of LLM capabilities.\n\nThe authors evaluate multiple prompts from the same generative model and compute statistics of LLM responses. This approach stands in contrast to many arguments for emergent intelligence in LLM, which derive from anecdotal examples. It is important to do actual experiments with LLM.", "weaknesses": "This is a presentation issue, but it affects my ability to understand and evaluate the paper. The writing is unedited. There are a lot of unnecessarily long sentences that can be condensed for pragmatics and readability. The grammar and punctuation are weird, like this might be an initial draft. I had to read the same content multiple times, and missed important details throughout the paper. \n\nFigures are odd, seem to be just thrown in together from unrelated bits. For example Figure 1 shows examples of graphs A-E. Are these all the graphs used in Experiment 1? If so, this is not stated. Weirdly, each graph is shown in a different pictorial style. I'm sorry, but this looks like someone just downloaded different types of graphs from google search, panel E has a random red outline around it.\n\nExperiment design is not given. I can not check the statistics without the experiment design. What are these degrees of freedom? It looks like multiple regression models were built? I think the intention is a good one, but it is not clear what was done", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses\n\n1.\tThank you for this thoughtful suggestion. We have edited longer sentences and improved grammar, punctuation, as well as the figures. \n2.\tYes, all graphs are evaluated in experiment 1, each with 3 instantiations with different spatial and non-spatial domains (18 total environments), 30 measurements of 15 tasks for openAI models and 3 for the remaining 5: a total of 8910 measurements. Domains include spatial ordered rooms (room 1), general spatial (flower field), and social (e.g., Bob is friends with Alice) to explain the structure. The reviewer is correct that we copied the figures of the larger graphs from one of the author’s previous papers. Thanks to the reviewer’s excellent suggestions, we've improved Figure 1 with better graphics.\n3.\tExperiment design: We thank the reviewer for this comment. We describe the experimental design in various portions of the paper but agree the reader would benefit from a concise summary. We elaborate on the design and the statistical approach taken below, and the reviewer can find all the prompts that were used in the prompt viewer link (https://cogeval.github.io/cogmaps/). We are happy to provide further information.\nOur experiment is designed to assess zero-shot LLM behavior on planning tasks. The experimental factors and levels are as follows:\n1.\tLarge Language Model (LLM):\n•\tOpenAI: GPT-4, GPT-3.5-turbo-175B, davinci-003-175B. Measurements per Factor Combination (MFC): 30 \n•\tGoogle: Bard. MFC: 3\n•\tHuggingFace: BigScience Bloom-176B. MFC: 3\n•\tCohere: Cohere-52.4B. MFC: 3\n•\tAnthropic: Claude-1-52B. MFC: 3\n•\tOthers: Pythia-20B, LLaMA-13B, Alpaca-7B. MFC: 3\n2.\tGraph structure of the environment: A, B, C, D, E, F\n3.\tItem Domain: spatial (numbered rooms), spatial (spaces), social ties\n4.\tConditions:\n•\tValue-based planning & traversal (what’s the optimal path?) \n•\tReward revaluation\n•\tTransition reevaluation\n•\tShortcut (with & without teleportation)\n•\tDetour (with & without teleportation)\n•\tTemperature: 0, 0.5, 1.0\nIt’s essential to note that this is an unbalanced design since the number of measurements differs across LLMs (30 for each of the OpenAI models and 3 per non-openAI LLMs).\nStatistical Analysis: We used logistic regression to model the probability of success as a function of experimental factors. The outcome variable is a binary measure of whether the LLM successfully answered the question or not, given a combination of factors.\nGiven the repeated measures, logistic regression effectively accounts for the effect of each factor and their interactions on the probability of success. However, only one logistic regression was run to ensure a comprehensive and unbiased evaluation.\nWe hope this clarifies the experimental design and the statistical approach. Please see PDF for new table.\n\nQuestions\n\n1.\tWe hope our responses above address this concern & are happy to elaborate\n\n2.\tExperimental design: All experiments were designed to test planning ability using different tasks in environments with different underlying structures and domains. All designs begin with explaining an environment (e.g., the rooms and connections in a castle, the social ties among people represented with names) with a Markov Decision Process (MDP) or graph, and rewards. Then a task follows. This task could be “traversal”: prompting the LLM to list the path from a given node to another; the “value path” condition: asking for the optimal path to highest rewards; “reward revaluation”: after responding to value path, the LLM is presented with a second prompt, in which a small local change in the magnitude of rewards is announced and optimal path is probed, to test robustness to changes in rewards; “transition revaluation”: after responding to value path, the LLM is presented with a second prompt with a small local change in the structure of the edges and the optimal path question; “detour”: after responding to value path, the LLM is presented with a second prompt, in which a specific previous path is blocked and the LLM needs to reroute and find a detour using the structure of the environment; “shortcut”: after responding to value path, the LLM is presented with a change in the structure that should reveal a shortcut, so the LLM is prompted for the optimal reward path again to test whether it can identify it.\n3.\t“Where is the \"community block model\" described?”  The block model is described in the submitted Figure 2.  This is most easily seen looking to the figure to the far right of this image left.  Each block (or community block) contains 5 vertices.  In the image to the right, each separate color here represents its own “community” (there are three of them), where nodes have a specified likelihood of being connected to other nodes in the same block (probability of intra-connection).  In this example image on the right, the likelihood of connection within a block is set to 100%, creating a fully connected clique (all vertices are connected to one another within the block.  \nThis has now additionally been further clarified in the supplementary material as follows: “To systematically evaluate GPT-4's planning or graph traversal failure modes, we created a three-block community graph structures where each block contains five vertices. Using this approach, we vary the connection density within each community block and ask GPT-4 to perform reasoning tasks over each permutation of the graph structure as block density is varied. \nExample graphs are shown in Figure 2 with the community graphs starting as simple line graphs on the left - representing the sparsest level of connectivity. We then create a new edge within each block for each iteration of the experiment until each community block forms a clique structure as seen on the right of Figure 2”.  For the experiment, we simply vary the connection probability within a block and observe how the LLM performance varies as we move from a largely disconnected block (Figure 2, left) to a fully connected block (Figure 2, right). "}]

Author rebuttal:
Dear reviewers, 

We are deeply grateful for your careful and detailed review of our work as well as your constructive questions and suggestions. Given the space constraints we have tried to address your questions as best as we could and have provided further material in the *attached PDF*. We hope that we have addressed all your comments and are more than happy to engage in further discussion. We hope to be able to show you how these comments have changed the paper overall in the camera-ready version.

Thank you,

The authors

Meta-review: This paper presents CogEval, a cognitive science-inspired protocol for evaluating language models. All reviewers unanimously found the experiments thorough across a variety of dimensions, and appreciated the statistical rigor of the analyses. The discussion has been very productive. Thanks to the reviewers’ excellent comments and deep engagement, authors have improved their framing and added more nuance to the interpretation of the results. They have also made attempts at clarifying details regarding the experimental setup and statistical analysis, as well as presentation of the figures/tables.

However, the main remaining concern is the following: The proposed tasks can be solved with cognitive maps, and the LMs are not able to solve them perfectly. However, equating this with “no emergent planning” is an incorrect interpretation. Several reviewers argue that it is an oversimplification and a stronger conclusion than the observations suggest. While the authors’ proposed edits address this concern partially, they have not provided further contextualization to other recommended baselines that best resemble a “chance” control setup and might provide a fairer testbed for comparison.

In short, the experimental setup is laudable, and the paper has resulted in engaging discussions. Facilitating such discussions more broadly could be beneficial for the community. In any case, I strongly recommend authors consider the insightful suggestions by the reviewer regarding alternative control setups to better detect "imperfect, yet better than chance" planning for future work, and highlight that as a limitation of the current draft.
Decision: Accept (poster)