SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluating Cognitive Maps and planning in Large Language Models with CogEval
Ida Momennejad*
Microsoft Research New York, NY idamo
Hosein Hasanbeig*
Microsoft Research New York, NY hosein.hasanbeig
Felipe Vieira Frujeri*
Microsoft Redmond, WA felipe.frujeri
Hiteshi Sharma
Microsoft Redmond, WA hitshei. sharma
Robert Osazuwa Ness
Microsoft Research Redmond, WA robertness
Nebojsa Jojic
Microsoft Research Redmond, WA jojic
Microsoft Research Redmond, WA hpalangi
Jonathan Larson
Microsoft Research Redmond, WA jolarso
@microsoft.com
Recently an influx of studies claims emergent cognitive abilities in large language models (LLMs). Yet, most rely on anecdotes, overlook contamination of training sets, or lack systematic Evaluation involving multiple tasks, control conditions, multiple iterations, and statistical robustness tests. Here we make two major contributions. First, we propose CogEval, a cognitive science-inspired protocol for the systematic evaluation of cognitive capacities in LLMs. The CogEval protocol can be followed for the evaluation of various abilities. Second, here we follow CogEval to systematically evaluate cognitive maps and planning ability across eight LLMs (OpenAI GPT-4, GPT-3.5-turbo-175B, davinci-003-175B, Google Bard, Cohore-xlarge-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B). We base our task prompts on human experiments, which offer both established construct validity for evaluating planning, and are absent from LLM training sets. We find that, while LLMs show apparent competence in a few planning tasks with simpler structures, systematic evaluation reveals striking failure modes in planning tasks, including hallucinations of invalid trajectories and falling in loops. These findings do not support the idea of emergent out-of-the-box planning ability in LLMs. This could be because LLMs do not understand the latent relational structures underlying planning problems, known as cognitive maps, and fail at unrolling goal-directed trajectories based on the underlying structure. Implications for application and future directions are discussed.
Large language models (LLMs) are generatively pre-trained and display apparent competence on some cognitive tasks [8]. This has led to a recent surge in studies claiming LLMs have emergent human-level cognitive abilities, which may encourage applications that interact with LLMs in a zero-shot or few-shot manner with expectations of human-level cognition. However, most claims of competence are based on anecdotes rather than systematic evaluation. In response, we make two contributions. First, we propose CogEval, a Cognitive Science-Inspired [12, 5, 39] protocol for Measurement and Evaluation of cognitive abilities in LLMs (Figure 1, top), such as planning, theory
of mind, causal inference, or other abilities. Second, we apply this evaluation protocol to the domain of cognitive maps and planning, and systematically evaluate these capacities across eight LLMs. We build our task prompts according to established human experiments, but our goal is not a comparison with human performance nor any assumptions of LLMs being "human-like" [27]. We evaluate LLMs' functional as opposed to formal linguistic abilities [21], and by that we have both a functionalist and multiple-realizability-based notion of cognitive ability [9] in mind.
We investigated whether LLMs (OpenAI GPT-4, GPT-3.5-175B, and davinci-003-175B, Google Bard, Cohere-52.4B, Anthropic Claude-1-52B, LLaMA-13B, and Alpaca-7B) understand the latent structure of planning problems (cognitive maps). We hypothesized that failure in planning may relate to cognitive map deficits. To address these questions, we followed the CogEval protocol (Figure 1). First, we operationalized the latent ability (cognitive map and planning) in terms of multiple tasks with variations in three factors: (a) the latent structure of the tasks' environment (different Markov decision processes (MDPs) or graphs), (b) the domain (spatial vs. social ties vs. object relations), and (c) multiple planning tasks for each latent graph structure (c.f. Section 2 for detail). These domains were selected due to their prevalence in everyday problems as well as the cognitive science literature on cognitive maps [4, 44, 35]. We then generated repeated measurements across small and large LLMs (c.f. Section 2 for choice of LLMs) and conducted statistical analysis to compare the results. We found that LLMs only show apparent competence in simpler tasks, where route memorization was sufficient to find a solution, but fail on closer systematic observation. Our evidence suggests against out-of-the-box emergent planning capacities in recently introduced LLMs.
What is a cognitive map? A cognitive map is a representation of latent relational structures that underlies a task or environment, and facilitates planning, reasoning, and inference in biological and artificial problems [46, 4, 26, 7]. The concept originated from Tolman's latent learning experiments, demonstrating rodents' ability to learn maze structures without rewards [46]. This challenged the dominant behaviorist view that learning only occurs with reinforcement; and paved the way for a cognitive revolution. Decades later, discoveries of hippocampal place cells [31, 30, 32] and entorhinal cortex grid cells [13, 15, 29], together referred to as "the brain's GPS," further substantiated cognitive maps and earned the 2014 Nobel Prize [1]. Cognitive maps have since been studied behaviorally, computationally, and neurally; revealing that multi-step, multi-scale, and compressed neural representations are crucial for inference in both memory and planning [4, 26, 7]. Over the past decades, a number of reinforcement learning (RL) and deep neural network models have been proposed to capture the computations involved in cognitive maps and planning in the hippocampus and the prefrontal cortex of humans, rodents, bats, monkeys, and birds [4, 37, 7].
Why would LLMs plan with a cognitive map? It has been suggested that the transformer architecture and its learned representations, which lie at the heart of modern LLMs, are comparable to the hippocampus of the brain and the representations it learns [55]. Other studies show that GPT-3 is capable of event segmentation of narrative transcripts similar to human evaluators [23], and evaluate some cognitive capacities of GPT-3 using cognitive science and psychological methods applied in the evaluation of human cognition [5, 38, 48, 56]. Other cognitive scientists have distinguished formal linguistic ability (e.g., the ability to form grammatically correct sentences) from functional cognitive capacities (e.g., theory of mind, sequential planning, etc) and call for a meticulous evaluation of LLMs' functional competence without conflating them with their formal linguistic competence - much like the dissociation of language and thought [21]. Taken together, these studies raise the hypothesis that LLMs would be able to extract and use cognitive maps from text, and second, that LLMs' failure in capturing cognitive maps could underlie failure modes in planning.
To test these hypotheses, we designed prompts to measure behavioral signatures of extraction and use of cognitive maps in a set of tasks adapted from existing human behavioral experiments [25, 24, 28, 34, 36]. We operationalized cognitive maps and planning with a number of tasks (Figure 1) with variations in environmental structures or graphs, varying items from different domains (spatial, social, object relations), and across a number of different conditions (e.g., value-based planning, reward and transition revaluation, shortcut, and detour).
Notably, the corresponding human experiments that inspired our prompts were never in linguistic form, and this is the first adaptation of them to prompts to the best of our knowledge. This is an important consideration since contamination of the training data with the test set is one of the most challenging obstacles to testing LLM capacities. To prevent any possible contamination, we avoided BIG-bench [40], which has been flagged by OpenAI for contamination [2], and a planning benchmark
Latent structures of environments (with varying domains)
Example task prompt Graph:A, Domain: spatial Cond:Value-based planning
Imagine a building with six rooms. From the lobby you have two choices, you can go to room 1 or room 2. You enter room 1, at the other end of room 1 there's a door that leads to room 3, and room 3 leads to room 5. There's a chest in room 5. You open it and there's 10 dollars, but you do not take any money. Then you exit and start over. This time in the lobby you choose room 2, which has a door to room 4, and room 4 has a door that leads to room 6. You find a chest with 50 dollars in room 6, but you do not take any money. You return to the lobby. You will only be able to choose one path that leads to the most money. Which room from the lobby will lead to the path where one can make the most money?
Figure 1: The CogEval protocol, task structure, and example task prompt. (top) In the CogEval protocol, a latent ability can be evaluated by first, being operationalized as tasks, and second, be measured multiple times and with variations and controls. We followed this protocol to evaluate cognitive map and planning. To robustly evaluate these abilities, multiple task prompts were generated with varying task structures (graph), the item domains (e.g., spatial or social), and task conditions (e.g., value-based path, detour). LLM responses were generated 30 times per task prompt and temperature for the three OpenAI models studied in this work and once per task and temperature for other LLMs. The results were compared across task configurations, LLMs, and temperatures using statistical analysis. (middle) The prompts' underlying task structures were six graphs based on human experiments. A: simple line graph from [25]. B: simple tree graphs based on [24]. C: graph A with double depth and stochastic transitions. D, E, and F represent community graphs from [36], [28], and [34] respectively. (bottom) An example prompt for graph A. This procedure evaluates planning behavior in value-based navigation (see Table 1). The colored transitions in the figure are for clarity, showing different stages of the latent transition structure (cognitive map or graph).
for GPT-3 [49] as both pre-date GPT-4 and raise data contamination issues. Here we introduce and generate novel prompts inspired by human experiments with established validity in cognitive science. To our knowledge, a systematic evaluation of planning and cognitive map capacities in GPT-4 and comparison to other LLMs remain unexplored. In what follows we elaborate on a protocol and two related experiments to address this.
The CogEval protocol. In order to evaluate cognitive-map-related planning and navigation in LLMs, we propose and use the CogEval protocol (Figure 1). Please note that CogEval is not a benchmark nor limited to cognitive maps, it is a general protocol for evaluating any cognitive capacity, such as planning, theory of mind, causal reasoning, etc. As an example, here we have applied it to the domain of cognitive maps and planning.
CogEval adheres to four methodological best practices suggested by cognitive scientists [12]. First, the latent construct or ability: here we evaluate cognitive maps, which are representations that capture a model of the task structure, and adaptive planning, which requires an internal representation of task structures (similar to model-based RL [42] or task-oriented model-free RL [16-19]). Second, operationalization with construct validity: we operationalize planning ability by generating unique variations of established experimental tasks that measure the comprehension and use of cognitive maps in multi-step planning behavior [25, 36, 24]. Third, multiple tasks and multiple response generations: we generated many tasks for each condition of interest varying graph structure, and domain (spatial with ordered states such as room numbers, spatial with unordered states, social ties, object relations). Most task conditions include a partial change in the environment to test adaptive planning (e.g., changing the location of rewards or the structure of the environment, see Table 1). Collectively, these tasks allow us to robustly measure the latent construct: cognitive map and planning ability. Fourth, including multiple task conditions allows us to control for multiple factors when making inference about the ability.
Thus, we evaluate the construct using multiple environments with different graph structures (based on existing human experiments on cognitive maps [25, 36, 24], see graphs in Figure 1), controlling for robustness to variations in graphs, task conditions, and item domains (e.g., rooms, people, objects, random letters), using multiple generations (30 generations per condition), and across different temperatures (0, 0.5, and 1).
LLMs evaluated. We compared the following LLMs: GPT-4-\* [2], GPT-3.5-turbo-175B [33], text-Davinci-3-175B [6] (Azure OpenAI API), Bard-\* [45], Anthropic Claude-1-52B [3], LLaMA13B [47], Cohore-52.4B [10], Alpaca-7B [43] (nat.dev API), where \* means the number of parameters is undisclosed.
Experiments. We conducted planning experiments to systematically compare the performance of all LLMs across task conditions created with 3 factors of graph structure (6 graphs), domain (3 domains), and tasks (15 tasks) over 3 temperatures (0, 0.5, 1).
2.1 A cognitive science inspired evaluation of cognitive maps and planning capacity in LLMs
We designed our experiment prioritizing robustness and control conditions. Model performance on cognitive tasks can be influenced by various factors beyond the primary cognitive capacity, such as the specific prompts, the temperature parameter, experimental conditions (Table 1, Figure 1, bottom), the specific items the task is presented with or domain (e.g., spatial connections vs. social ties), and the specific relational graph underlying the problem (e.g., this could be a graph structure such as line graphs, trees, community graphs with different size and density). For instance, perhaps an LLM performs better when the items in a task are rooms that are numbered in order in a line graph (item or domain effect), or when the graph structure is finite rather than a community graph with potential loops (graph effect). Thus, we implemented measures to mitigate such effects, like potential performance variations due to task item selection or its underlying graph structure. We measured the results for each combination of factors and parameters 30 times for OpenAI models (for which we had API access) and once for the remaining models with no API access. We compared the results across 10 LLMs.
Why vary temperature? Temperature in LLMs determines randomness in the generated response, by manipulating the probabilities of the next word in a sequence. Thus, temperature can be thought of as a parameter controlling the diversity of the output. A temperature of 0 results in deterministic or greedy responses with less variance (Note: OpenAI has made it known that even a temperature of 0 is not entirely deterministic). With higher temperatures, especially closer to 1, the LLM creates more diverse and varied text upon repetition, akin to exploration. While a higher temperature may be helpful for tasks that require varied responses or creativity, it could go either way for planning: on the
one hand, precision in planning trajectories may seem more in line with a deterministic temperature, and on the other, a higher temperature leads to exploration, which may improve behavior by getting out of local minima. Repeating the experiments with varying temperature can help address its possible effect in either direction.
Statistical analysis. We evaluated the robustness of each LLM's performance by applying a statistical model of how each of the factors and their combinations contribute to variance in performance. Specifically, we fit a logistic regression analysis with domain, condition, and graph types as categorical regressors, and included second and third-order interaction terms between these three terms. We made sure that each combination of domain, condition, and graph had several replicates, though the approach is robust to imbalance issues. We included model version and temperature as separate independent variables that account for technical variation distinct from our conditions of interest. See supplement for full details on analysis and results.
2.1.1 Task prompts
Navigating cognitive maps requires adaptive multi-step planning using compressed representations of the environment, not mere memorization of all routes. Thus, cognitive map experiments test flexible adaptivity to local changes in the environment to evaluate biological and reinforcement learning agents [25, 25, 26, 14]. Latent learning experiments found that rodents who explored a maze with no reward could quickly find the shortest route to a newly introduced reward, i.e., find an optimal policy in RL context. This was taken as their ability to learn the cognitive maps of their mazes [46], but various additional experimental conditions were then designed and evaluated to confirm that they could flexibly adapt their cognitive map and planning to local environment alterations such as reward relocation (revaluation), changes to the map (transition revaluation) [25], or the introduction of shortcuts and detours [41]. Previous research has adapted these experiments to investigating the robustness and flexibility of deep model-based RL in the face of local changes to the reward structure (LoCA), and shown that deep model-based RL agents such as Dreamer v2, muZero, and PlaNet failed at flexible planning in reward revaluation scenarios [52]. Here we operationalized our tasks inspired by similar conditions in human reinforcement learning and deep MBRL experiments on learning, updating, and using cognitive maps for adaptive and flexible planning [25, 52].
Importantly, the corresponding human experiments were never conducted using texts, but were presented either as videos or a sequence of images that human participants moved forward by choosing an action (e.g. pressing left, right, up, or down). We believe this mitigates the risks of contamination. Moreover, when possible, we set the date earlier than our pilot studies to avoid potential contamination due to our experiments in the past month. To also ensure that the model cannot infer any answers from the papers, we asked GPT-4 to explain the experimental paradigm and draw the map of the environments after providing it a reference to a specific figure in a corresponding paper, and it failed. Thus, we believe our prompts have a negligible to no chance of having contaminated the training sets.
Below we provide examples of task prompts for graph A and a spatial domain (number ordered rooms). All prompts are available in the supplementary material and on https://github.com/cogeval/cogmaps.
I. Value-based or goal-driven planning. Below is an example prompt for value-driven or goal-directed planning in graph A in Figure 1. Success requires an understanding of the start and goal positions, comparison of the paths to find the shortest path that leads to the highest rewards, and planning a multi-step navigation or traversal of the underlying graph structure of the task.
Imagine a world with six rooms. From the lobby you have two choices, room 1 and room 2. You enter room 1, at the end there's a door that leads to room 3, and room 3 leads to room 5. There's a chest in room 5. You open it and there's 10 dollars. Then you exit and start over. This time in the lobby you choose room 2, then enter room 4, which leads to room 6. There's a chest with 50 dollars. You return to the lobby. Which room will you choose to make the most money?
II. Transition Revaluation, after prompt I. This condition occurs when the structure of the environment (e.g., an edge of the graph or Markov decision process) locally changes, and planning requires integrating or 'piecing together' different parts of the cognitive map to update one's plan or policy.
Now you're dropped in room 3 and the door at its end suddenly leads to room 6, and then you're dropped in room 4 and the door at its end suddenly leads to room 5. you return to the lobby. Which room will lead to more rewards?
III. Reward Revaluation, after prompt I. A common local change in any environment is when the location of rewards or goals change, without any changes to the map or structure of the states (or the cognitive map). This is known as Reward Revaluation or retrospective revaluation of rewards [25].
Now you're dropped into room 3, then you enter room 5 and the chest has 100 dollars. Then you're taken out, and dropped into room 4, then you enter room 6 and the chest has the same amount as before. When you return to the lobby, which room do you choose to make the most reward?
V. Shortcut prompts with and without teleportation, after prompt I. Tolman's experiments on cognitive maps [46] included a condition evaluating the animal's ability to discover shortcuts. Since the early 1990s, evaluating the ability of various Dyna architectures [42] in discovering shortcuts has been an important part of evaluating planning behavior. Below are two different shortcut prompts.
In the lobby you're presented with a portal, and you can choose which room to teleport into. Which room do you choose to maximize rewards?
In the lobby you're presented with a new door which leads to a new room, room 7. Room 7's door leads directly to room 6. Remember that you will only be able to choose one path that leads to the most money. Which room from the lobby will lead to the path where one can make the most money?
V. Detour prompts with and without Teleportation, after prompt I.
You enter the lobby and this time you encounter a new room, room 7. Room 7's door leads to room 8, and room 8 leads to room 9. From room 9 you can teleport anywhere. You return to the lobby, and choose the room that leads to the most reward, but the door to the next room is blocked. You go back to the lobby. Which room do you choose to reach the most rewards?
You enter the lobby and this time you encounter a new room, room 7. Room 7's door leads to room 8, and room 8 leads to room 6. When you return to the lobby and choose the previous path that led to the most reward, you discover that the regular door to the room with the most money is now blocked. You go back to the lobby. You will only be able to choose one path that leads to the most money. Which room from the lobby will lead to the path where one can make the most money?
3.1 Repeated measures comparison of planning across LLMs
We evaluated out-of-the-box emergent or native ability of different LLMs on the cognitive map tasks. Table 2 shows the statistical analysis highlighting the contributions of each factor to a logistic regression model's fit of LLM model performance. The magnitude of chi-square test statistics indicate contribution to overall model fit. Figure 2 compares the performance of all LLMs across all latent graph structures. Table 3 shows mean and standard error for planning performance across tasks and LLMs.
The results in Table 2 indicate that the LLM  $(\chi^2(11) = 2357.87, p < .001)$ , graph  $(\chi^2(11) = 3431.53, p < .001)$ , condition  $(\chi^2(11) = 2080.04, p < .001)$ , and domain  $(\chi^2(11) = 304.06, p < .001)$  each yielded significant chi-squared statistics. This means that not only did different LLMs performed