SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Brief descriptions of the task conditions applied to varying graphs and domains
Condition	Description	Group
valuePath	The optimal solution is to find the optimal policy, or shortest path, which yields the highest reward
1stepPath	The optimal solution is a 1-hop policy, i.e., goal is adjacent to the starting state	Traversal
2stepPath	The optimal solution is a 2-step policy
3stepPath	The optimal solution is a 3-step policy
nstepPath	The optimal solution is an n-step policy, where max n is the diameter of the graph (longest shortest path)
rewardReval	Upon a local change in the reward structure, the goal has changed and the optimal solution requires finding a new path	RewReval
policyReval	Upon a local change in the reward structure, the optimal solution requires finding a new policy
transReval	Upon a local change in the transition structure, the goal is the same but the optimal solution requires finding a new policy	TransReval
transRevalStochastic	Upon a local change in the transition structure, the goal is the same but the optimal solution requires finding a new policy in a stochastic environment
nonteleShortcut	Upon a change in the graph structure, the optimal solution requires finding a shortcut
nonteleShortcutCoT	Upon a change in the graph structure, the optimal solution requires finding a shortcut, an additional CoT prompt is given	Shortcut
teleShortcut	Upon a local change in the transition structure, the optimal solution requires finding a shortcut using a teleportation portal
teleShortcutCoT	Upon a local change in the graph or transition structure, the optimal solution requires finding a shortcut using a teleportation portal, an additional CoT prompt is given
nonteleDetour	Upon a change in the graph structure, the optimal solution requires finding a detour	Detour
teleDetour	Upon a local change in the transition structure, the optimal solution requires finding a detour using a teleportation step
Figure 2: Results for planning experiments in 8 LLMs. (left) Mean and standard error of performance on all tasks for each of the different graphs (see Figure 1 for graph details) across different LLMs studied in this work. (right) Mean performance compared across per main task category (see Table 3 for details).
differently, but performance varied as a result of varying graphs, domains, and conditions. Conversely, the temperature showed a non-significant chi-squared statistic  $(\chi^2(11) = 1.28, p = .53)$  and the interaction between the LLM and temperature was also non-significant  $(\chi^2(11) = 10.69, p = .71)$ . Noteworthy, the interactions among graph-domain, graph-condition, domain-condition, and graph-domain-condition were all significant (all  $p's < .001$ ). The interactions among graph-domain  $(\chi^2(11) = 334.41, p < .001)$ , graph-condition  $(\chi^2(50) = 1651.33, p < .001)$ , domain-condition  $(\chi^2(39) = 310.53, p < .001)$ , and graph-domain-condition  $(\chi^2(108) = 1133.16, p < .001)$  were all significant. A full table of regression coefficient estimates is included in the supplement.
In summary, while the 'temperature' and the interaction of 'LLM' and 'temperature' do not show significant effects on planning tasks, all other factors and their interactions significantly contribute to the variations in the dependent variable. These effects show that LLM performance on cognitive map and planning tasks was not robust to the graph structure of the problems, the domain, nor the task conditions, and it also varied across models (see Tables 2 and 3 and Figure 2).
3.2 Failure modes
We note three main failure modes when the task had an underlying graph with a dense community structure. Notably, when we probe the LLMs to list connected rooms or items as (state, actions, state) tuples, they do well (e.g., (room1, opendoor, room3) is a tuple for graph A in Figure 1). However, when asked to do any tasks with a community graph structure using this tuple knowledge, LLMs display the following failure modes; (1) hallucinate edges that do not exist, or (2) produce longer trajectories instead of shortest paths, or (3) produce trajectories that fall in loops. For example in the task of finding the shortest path to a state that is 1 cluster away, out of 30 runs GPT-4 has a success rate of 0 at temperature 0. Even with changing the temperature to 0.5 or
Table 2: Step-wise contribution of adding each factor to the logistic regression fit of LLM model performance (number of successes out of max possible successes in dialog).
term	Chi-squared Stat (Deviance)	df	p value
1	LLM	2357.87	7	<0.001
2	graph	3431.53	5	<0.001
3	domain	458.74	2	<0.001
4	temperature	1.28	2	0.53
5	condition	2080.04	4	<0.001
6	LLM and temperature	10.69	14	0.71
7	graph and domain	334.41	10	<0.001
8	graph and condition	1651.33	20	<0.001
9	domain and condition	310.53	8	<0.001
10	graph, domain, condition	1133.16	44	<0.001
Table 3: Mean and standard errors for planning performance across all task conditions in all 10 LLMs. ARI scores closer to zero represent poor performance by the LLM and ARI scores reaching 1.0 represent performance matching Leiden.
Condition	gpt-4-32k	gpt-35	davinci-003	claude-v1	pythia-20b	cohere	llama-13b	alpaca-7b	bard
1stepPath	0.99, 0.08	0.76, 0.32	0.52, 0.45	0.57, 0.37	0.64, 0.41	0.27, 0.42	0.23, 0.38	0.27, 0.41	0.05, 0.10
2stepPath	0.82, 0.35	0.73, 0.38	0.16, 0.25	0.61, 0.41	0.67, 0.42	0.29, 0.44	0.22, 0.37	0.35, 0.47	0.25, 0.50
3stepPath	0.55, 0.38	0.37, 0.37	0.58, 0.43	0.27, 0.31	0.35, 0.49	0.04, 0.11	0.04, 0.07	0.06, 0.20	0.11, 0.10
nonteleDetour	0.55, 0.39	0.51, 0.35	0.55, 0.43	0.50, 0.41	0.51, 0.37	0.21, 0.35	0.19, 0.33	0.26, 0.38	0.29, 0.48
nonteleShortcut	0.56, 0.40	0.52, 0.39	0.49, 0.40	0.62, 0.43	0.40, 0.36	0.16, 0.27	0.11, 0.18	0.20, 0.30	0.29, 0.48
nonteleShortcutCoT	1.00, 0.00	1.00, 0.00	0.09, 0.07	0.58, 0.38	0.36, 0.38	0.37, 0.49	0.17, 0.29	0.37, 0.37	-
nstepPath	0.47, 0.38	0.31, 0.34	0.17, 0.27	0.33, 0.37	0.27, 0.42	0.05, 0.11	0.06, 0.08	0.12, 0.32	0.00, 0.00
policyReval	0.21, 0.18	0.18, 0.23	0.13, 0.04	0.28, 0.30	0.00, 0.00	0.00, 0.00	0.04, 0.07	0.05, 0.22	0.00, 0.00
rewardReval	0.67, 0.40	0.57, 0.36	0.34, 0.25	0.48, 0.35	0.60, 0.45	0.31, 0.44	0.28, 0.43	0.33, 0.44	0.14, 0.14
teleDetour	0.47, 0.35	0.34, 0.30	0.53, 0.44	0.37, 0.33	0.44, 0.41	0.21, 0.35	0.23, 0.37	0.23, 0.38	0.29, 0.48
teleShortcut	0.54, 0.39	0.35, 0.33	0.44, 0.41	0.45, 0.39	0.27, 0.33	0.16, 0.27	0.16, 0.22	0.12, 0.24	0.29, 0.48
teleShortcutCoT	0.50, 0.00	0.50, 0.00	0.04, 0.01	0.50, 0.50	0.39, 0.36	0.19, 0.40	0.83, 0.29	0.35, 0.36	-
transReval	0.60, 0.42	0.59, 0.40	0.49, 0.38	0.55, 0.36	0.47, 0.42	0.19, 0.28	0.22, 0.33	0.27, 0.37	0.08, 0.17
transRevalStochastic	0.73, 0.36	0.52, 0.36	0.91, 0.24	0.78, 0.34	0.36, 0.32	0.00, 0.00	0.11, 0.19	0.22, 0.39	-
valuePath	0.58, 0.41	0.66, 0.40	0.66, 0.39	0.44, 0.41	0.49, 0.46	0.31, 0.40	0.27, 0.39	0.33, 0.45	0.29, 0.48
1 and repeating the same 30 runs its success rate can not exceed  $10\%$ . Please refer to Figure 3 for examples of above failure modes.
4 Discussion and future directions
This paper makes two main contributions. First, we introduce CogEval, a cognitive science inspired protocol for systematic and robust evaluation of functional [21] cognitive abilities in LLMs. Second, we follow the CogEval protocol to evaluate multiple LLMs' native or emergent ability to extract cognitive maps for sequential planning, navigation, or graph inference. All tasks and prompts are based on non-linguistic human cognitive science experiments that we adapted into text prompts for the first time. We test for robustness of the findings by varying task conditions, graph structure, domains (spatial, social), and LLM temperature. Our systematic evaluation reveals that while LLMs display apparent competence on some tasks in simpler graphs, they do not have out-of-the-box zero-shot emergent cognitive map comprehension or planning competence for other graphs.
Methodological contribution. CogEval, our proposed cognitive-science inspired protocol [12] for systematic evaluation of LLMs, affords the following contributions. (1) We avoid the reuse of contaminated standardized benchmarks by creating novel prompts based on non-text-based experiments that are known to evaluate cognitive maps and planning in humans, animals, and RL. (2) We use multiple tasks to probe the cognitive constructs (cognitive maps and planning) and repeat each interaction multiple times and across different temperatures. (3) We use statistical analysis to evaluate the robustness and reliability of each effect, with three main factors of graph structure, item domain (spatial vs. social), and task condition (e.g., value-based decision making, shortcut, detour, see Table 1). (4) We employ chain of thought and instruction prompts to evaluate the limits of out-of-the-box cognitive abilities of LLMs (Supplementary Experiment 2), and (5) analyze and categorize different types of failure modes. Please note that CogEval is not a benchmark nor limited to evaluating cognitive maps and planning, it is a general protocol for evaluating any cognitive capacity in LLMs. As an example, in this paper we have applied it to the domain of cognitive maps and planning.
No evidence for understanding cognitive maps or planning. Our systematic and incremental evaluations reveal limited to no cognitive map capacities in the current generation of LLMs - including
GPT-4, temp=0, graph D, n-step
GPT-35, temp=1, graph D, 1-step
GPT-4, temp=0, graph D, n-step
Figure 3: Examples of three failure modes. (left) Edge hallucination. (middle) Failure at finding a 1-step policy within the same cluster. (right) Failure at multi-hop path by both falling in a loop and hallucinating edges. In each example the blue box is the task prompt, the grey box shows the model response, and the green arrows demonstrate the correct response on the graph.
GPT-4. Specifically, we find that LLMs only show apparent competence on simple sequential inference tasks where route memorization can help, and given LLMs have received all possible trajectories in the text prompt. We also observe that the sparsity of graph structure drove performance. However, when 1-step and multi-step traversal and planning require understanding the underlying relational structure of the environment graph, LLMs including GPT-4 fail by hallucinations, suboptimally long routes, or falling in loops.
How did LLMs solve the simpler tasks? Without access to full architectures or training sets, we can only form hypotheses based on our behavioral observations. We observe that LLMs do better in problems where the entire trajectories are explicitly available in the text prompts, and they only need to retrieve and piece together partial changes. However, planning behavior in more complex graphs is far worse, and this is not just due to graph size: performance was worse for graph B (7-node tree) than C (12 node, parallel lines). Performance on the 15-node graph with 3 dense clusters was worse than the 16-node (4-cluster) graph that has better cross-cluster connectivity and more paths among clusters.
These observations suggest that LLMs may fail at planning problems where they need to use the transition structure to unroll the trajectories and find the correct path, which is closer to the notion of planning in model-based RL and in cognitive science. Capturing the underlying structure and using it to unroll trajectories are quintessential to cognitive maps and planning ability. Thus, the apparent competence in simpler tasks may be due to using cached or memorized routes rather than understanding the cognitive map, planning, or inference ability.
LLMs may do better in smaller and simpler graphs because the prompt already expands all the possible paths or trajectories. When there is a change in the rewards or transition structure, LLMs only need to change one step in an already laid out path. However, in more complex graphs only the one-step connections are laid out in the prompt, but not all paths or trajectories between any given two nodes. We observed that failures significantly increase in tasks with these larger graphs with community structures, even when an LLM can list the pairwise tuples of connected states (see failure modes, Figure 3).
Interpreting the results. The experiments in the paper are not meant to be interpreted as a benchmark for planning. They probe the same construct in different ways, evaluating the ability to use information about (state, action, state) tuples, e.g., (room1, openleftdoor, room3), to piece together policies in response to task prompts. A reader may wonder why we claim that LLMs do not display emergent planning in spite of non-zero performance for some tasks in our experiments (Figure 2. We interpret the findings as such due to failure modes and various inconsistencies in success cases (Figure 3). For instance, a common failure mode is generating sequences with hallucinated (state, actions, state) tuples that do not exist. For GPT-4 this constitutes over  $20\%$  of 4-step plans. Another common failure mode is that they fall into loops when prompted to find the shortest path between two states (Figure 3, left and right). LLMs sometimes even fail to identify 1-step paths or suggest multi-hop trajectories for traversing to an adjacent state (Figure 3, middle).
These observations seem counter-intuitive given some LLMs can generate a list of tuples when asked, but fail to use the tuples to make (even 1-step) valid plans. This shows that, while LLMs appear to solve planning problems when given simple routes that can be explicitly memorized, they cannot generalize from route memory solutions (trajectories directly in the prompt) to using the tuples to adaptively generate branching plans. Together, these inconsistent observations are in line with the hypothesis that LLMs do not understand cognitive maps and therefore cannot consistently plan. Elsewhere propose black box architectures that improve planning performance [54]. However, our findings point to a lack of out-of-the-box zero-shot emergent planning ability.
Limitations. First, we lack knowledge of LLMs like GPT-4's architecture or training. To address this limitation, we did not use existing text-based benchmarks and instead generated novel prompts not in their training sets. Second, in the human experiments that influenced our prompts participants learn gradually, experiencing states one-by-one, only tested after they showed signs of learning, similar to a model-based RL agent having the transition structure and using it for inference and planning. To address this difference, we present the environment's structure in linguistic format. The LLM had to extract the cognitive map, identify the goal location based on instructions, and infer the policy towards the goal. Third, we do not have a human baseline for the language-based tasks, so a comparison with human behavior on the exact language prompts remains the topic of future studies.
Implication for applications. LLMs are expected to be applied in fields like gaming, planning, and social reasoning, with tasks that require understanding the inherent relational structure of the problem from the input for flexible reasoning and planning. However, here we show various failure modes in the understanding of the underlying cognitive maps or planning abilities, including hallucination and falling in loops. Even when provided instructions and Chain of Thought (CoT) prompts like breadth-first search (BFS), we observe that GPT-4 struggles to process multi-hop paths it has not experienced (Supplementary Experiment 2). These findings suggest caution in the application of LLMs in problems that involve planning or complex structures. Below we discuss future directions that may mitigate these challenges for problems with simpler structures.
LLMs as programmable machines rather than emergent intelligence? While some regard LLMs as agents with emergent intelligence comparable to humans and animals, our findings are more consistent with the view that LLMs are programmable machines where natural language is their programming language [20]. Thus, here we evaluated planning in LLMs in a functionalist and multiple-realizability sense rather than making any assumptions of them being "human-like" [27].
Future directions. A future direction is to analyze embedding representations and attention in LLMs, and test hypotheses about representations underlying success and failure in planning. This mirrors how neuroscience analyzes neural data to understand representations in model-based and predictive planning and decision-making [25, 7]. A recent study suggests a promising future for this direction [57]. Another interesting direction is to study the limits of LLMs' transitive inference using pairwise associations [37, 34], given we observed that while some LLMs could list pairwise tuples or recognize the goal, they still struggled with planning. A further direction is to study whether the use of schemas, i.e., overused, generalized cognitive maps such as "airport" [50, 22, 11, 51], can improve performance on real-world scenarios by evoking helpful structures, given LLMs have shown promise with analogical reasoning tasks [53]. Finally, some have suggested ways to improve planning by augmenting LLMs with algorithms that enable executive control, an interesting direction that can contribute to the future of augmenting both larger and especially smaller language models (see [54]).
LLMs need a hippocampus and prefrontal cortex. The hippocampus and prefrontal cortex in the brain extract the relational structure or cognitive maps from sequential data to flexibly plan at multiple scales [7, 26]. Their functions can inspire memory, planning, and executive control augmentations to LLMs in order to mitigate failure modes such as hallucinating edges. Ongoing research shows that indeed prefrontal cortex-inspired solutions can improve planning performance in an LLM-based architecture with multiple calls to GPT-4 [54], pointing at a promising future direction.
Summary. We introduced CogEval, a cognitive science inspired protocol for systematic and robust evaluation of LLMs. We applied CogEval to evaluate planning performance across 8 LLMs and found poor performance.
5 Acknowledgement
We are extremely grateful to Peter Lee, Michael Frank, John Krakauer, Joshua Tenenbaum, Paul Bennett, Alison Gopnik, and Melanie Mitchell for early feedback on methods and results. We would also like to acknowledge Derek Worthen and Richard Ciapala for engineering support.
Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Mlkiowski, Piyush Patil, Pouya Pezeshkporr, Priti Oli, Qiao zhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahul Habacker, Ramon Risco Delgado, Raphael Milliere, Rhythm Garg, Richard Barnes, Rif A Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Ryan Yang, Sahib Singh, Saif M Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R Bowman, Samuel S Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad Steven T Piantadosi, Stuart M Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra Tal Linzen Tal Schuster Tao Li Tao Yu Tariq Ali Tatsu Hashimoto Te-Lin Wu Theo Desbordes Theodore Rothschild Thomas Phan Tianle Wang Tiberius Nkyili Timo Schick Timofei Kornev Timothy Telleen-Lawton Titus Tunduny Tobias Gerstenberg Trenton Chang Trishala Neeraj Tushar Khot Tyler Shultz Uri Shaham Vedant Misra Vera Demberg Victoria Nyamai Vikas Raunak Vinay Ramasesh Vinay Uday Prabhu Vishakh Padmakumar,Vivek Srikumar William Fedus William Saunders William Zhang Wout Vossen Xiang Ren Xiaoyu TongXinran Zhao Xinyi Wu Xudong Shen Yadollah Yaghoobzadeh Yair Lakretz Yangqiu Song,Yasaman Bahri,Yejin Choi,Yichi Yang,Yiding HaoYifu ChenYonatan Belinkov Yu HouYufang Hou Yuntao Bai,Zachary Seid Zhuoye Zhao Zijian WangZijie J WangZirui Wang,and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. June 2022.
[41] K L Stachenfeld, M Botvinick, and others. Design principles of the hippocampal cognitive map. Adv. Neural Inf. Process. Syst., 2014.
[42] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, November 2018.
[43] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following LLaMA model, 2023.
[44] Rita Morais Tavares, Avi Mendelsohn, Yael Grossman, Christian Hamilton Williams, Matthew Shapiro, Yaacov Trope, and Daniela Schiller. A map for social navigation in the human brain. Neuron, 87(1):231-243, July 2015.
[45] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, HengTze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
[46] E C Tolman. Cognitive maps in rats and men. Psychol. Rev., 55(4):189-208, July 1948.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[48] Tomer Ullman. Large language models fail on trivial alterations to Theory-of-Mind tasks. February 2023.
[49] Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). June 2022.
[50] Marlieke T R van Kesteren, Sarah F Beul, Atsuko Takashima, Richard N Henson, Dirk J Ruiter, and Guillén Fernández. Differential roles for medial prefrontal and medial temporal cortices in schema-dependent encoding: from congruent to incongruent. Neuropsychologia, 51(12):2352-2359, October 2013.
[51] Marlieke Tina Renee van Kesteren and Martijn Meeter. How to optimize knowledge construction in the brain. NPJ Sci Learn, 5:5, May 2020.
[52] Yi Wan, Ali Rahimi-Kalahroudi, Janarthanan Rajendran, Ida Momennejad, Sarath Chandar, and Harm van Seijen. Towards evaluating adaptivity of Model-Based reinforcement learning methods. April 2022.
[53] Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. December 2022.
[54] Taylor Webb, Shanka Subhra Mondal, Chi Wang, Brian Krabach, and Ida Momennejad. A prefrontal cortex-inspired architecture for planning in large language models, 2023.
[55] James C R Whittington, Joseph Warren, and Timothy E J Behrens. Relating transformers to models and neural representations of the hippocampal formation. December 2021.
[56] Eunice Yiu, Eliza Kosoy, and Alison Gopnik. Imitation versus innovation: What children can do that large language and language-and-vision models cannot (yet)? May 2023.
[57] Safoora Yousefi, Leo Betthauer, Hosein Hasanbeig, Akanksha Saran, Raphaël Millière, and Ida Momennejad. In-context learning in large language models: A neuroscience-inspired analysis of representations, 2023.