Evaluating Spatial Understanding of Large Language Models
Authors: Yutaro Yamada
Yihan Bao
Andrew Kyle Lampinen
Jungo Kasai
Ilker Yildirim
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) show remarkable capabilities across a variety of tasks. Despite the models only seeing text in training, several recent studies suggest that LLM representations implicitly capture aspects of the underlying grounded concepts. Here, we explore LLM representations of a particularly salient kind of grounded knowledge --- spatial relationships. We design natural-language navigation tasks and evaluate the ability of LLMs, in particular GPT-3.5-turbo, GPT-4, and Llama2 series models, to represent and reason about spatial structures. We also compare these abilities to human performance on the same tasks. These tasks reveal substantial variability in LLM performance across different spatial structures, including square, hexagonal, and triangular grids, rings, and trees. In extensive error analysis, we find that LLMs' mistakes reflect both spatial and non-spatial factors. These findings suggest that LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains.

Published in Transactions on Machine Learning Research (02/2024)
Evaluating Spatial Understanding of Large Language Models
Yutaro Yamada
Yihan Bao
*
Andrew K. Lampinen
Jungo Kasai
Ilker Yildirim
Yale University
Toyota Technological Institute at Chicago
{yutaro. yamada, yihan. bah, ilker. (ataeyyyyldrimmmylleeee.ee
Reviewed on OpenReview: httppp/r/poopueeeee..o/////ooooa
Abstract
Large language models (LLMs) show remarkable capabilities across a variety of tasks. De- spite the models only socing text training, several recent studics suggest that LLM repro- sentations implicitly capture aspects of the underlying grounded concepts. Here, explore LLM representations of particularly salient kind of grounded knowledge spatial rela- tionships. We design natural-language navigation tasks and evaluate the ability of LLMs, in particular GPT-3.5-turbo, GPT-4, and Llama2 series models, to represent and reason about spatial structures. These tasks reveal substantial variability in LLM performance across different spatial structures, including square, hexagonal, and triangular grids, rings, and trees. In extensive error analysis, we find that LLMs' mistakes reflect both spatial and non-spatial factors. These findings suggest that LLMs appear to capture certain aspects of spatial structure implicitly, but room for improvement remains
1 Introduction
Large language models (LLMs) show remarkable capabilities in language, and also hints implicitly learning about the grounded concepts beyond language. For example, language models can develop semantically- organized internal representations for basic concepts like color and direction (Abdou et al. 2021 Patel & Pavlick 2 which can allow grounding the models with only a few examples. Furthermore Li et al.
(2021) demonstrate that internal representations of language models can dynamically track the states of entities and their relations during discourse. Human language use manifests the semantics of the world from which originates, and thereby might allow LLMs to implicitly learn something about the entities and processes that exist the physical world.
Natural intelligences extract and use such knowledge of the physical world often referred to as world models. A particularly salient example is the ability of humans and animals to create and manipulate mental maps, which serves as fundamental prerequisite for flexibly navigating and interacting with their environments. Cognitive maps [Tommnn 1948) were suggested as metaphor for mental representations that enable adaptable behavior such as planning routes or finding shortcuts. The quest to uncover how the brain represents such maps has led to significant discoveries about the neural mechanisms underlying such maps, such as place cells (eefee Dostrovsky 1971), grid cells (Hafting et al. 2005), and boundary cells (Lever et al. 2009). While navigation generally involves active, grounded experience, some studies suggest that humans use similar representational structures for abstract knowledge well (e.g. Whittington el. al.) 2020 Constantineseu et al.) 2016). Furthermore, cognitive and neural evidence suggests that humans and animals can learn spatial structure solely from sequences of observations (Whittington et al. 2022 Garvert et al. 2017). This raises an intriguing possibility that LLMs might also be capable of inferring sophisticated spatial relations from their sequential, text-based inputs.
Equal contribution.
1 Our code and data are available http:///tttthh.ccmmmmmg///ptsttt http:////////ggggo///////aaaaaaa
yyansda/Ssatiallvallli

---

Published in Transactions on Machine Learning Research (02/2024)
In this paper, we examine the spatial understanding capabilities of LLMs in particular OpenAT's GPT-3.5- turbo, GPT-4 models as well as Llama2-7B, Llama2-13R, Llama2-70B, CodeT. models (Touvron et al. 2023). We designed broad set of navigation tasks rendered in natural language such that successfully solving these tasks requires accurately representing the underlying spatial relations. These relations include grids with square; hexagonal, and triangular topologies, in addition to rings and trees. Our study reveals that LLMs exhibit varying performance when the underlying spatial structures differ ($3.1). We also observe that presenting the global map upfront actually makes the task more challenging compared to providing local navigational instructions only ($3.2). Moreover, we investigate the effect of the spatial patterns (e.g., random order, row-major) by which global map is expressed on the performance of LLMs ($3.3). We also provide evidence that LLMs spontaneously utilize object information as landmarks for constructing spatial maps ($,5),, much like humans and animals. Finally, detailed error analyses ($4) confirm that in spat structures that LLMs perform well, their mistakes manifest the underlying spatial topology, as well as non-spatial factors. These crror distributions suggest that GPT-4, which often substantially outperforms GPT-3.5, seems implicitly grasp certain elements of spatial structure, but there is still room for improvement.
We believe gaining insights into the spatial comprehension abilities of LLMs is valuable in euhancing our undorstanding of how these models acquire and grasp grounded concepts.
2 Spatial Understanding Task
How evaluate the text-only models' understanding of spatial information? With human participants, we could have them explore the environment and then ask them to draw map. However, for text-in, lext-out models like LLMs, formalizing the task of spatial reasoning is difficult because these models lack the capability to directly interact with the physical world or visually draw the entire map. However, some studies in human cognition (Garvert et al., 2017) have presented participants with sequential data that is sampled from an underlying spatial structure. These studies suggest. that humans implicitly acquire knowledge and learn representations that mirror the spatial structure that is latent in the data. This motivates the hypothesis that presenting sequential transitions might be enough for LLMs to achieve spatial understanding.
For example, if model comprehends square map's structure, it should be able to answer the following question: "You start at a spot where you find an apple. You move up and find a banana. Then you move right and find an orange. Next, you move down and find grape. Now, you move left. What do you find?" Answering this question correctly demonstrates an understanding of loop closure, which is a fundamental aspect of this spatial structure. That is, we have a square grid and an initial location, we can allow the model to take a random walk until it reaches a location it has already visited before. At each newly visited location, we inform the model about the objects it perceives, and then we ask the model which object it would have seen just before reaching the already visited location. By generating such questions synthetically, we can systematically evaluate the spatial understanding of LLMs. For each question, we randomly seleet the object names from the ImageNet-1k labels to fill every location of the spat ial grid to create the underlying map.
2.1 Models and evaluation metrics
We test. GPT-3.5 (gpt-3.5-turbo-0300), GPT-4 (gpt-4-0314), Llama2-7B, Llama2-13B, Llama2-70B, and CodeLlama-34B. The decoding parameters we use are: frequency penalty = 0.0, presence penalty 0.0, lemperature 1.0, top p 1.0. All Llama models are Llama-Chat models and the CodeLlama model is the Instruct variant. The context window sizes for these models are 4,096 tokens except for GPT-4, which has 8, 5,192 tokens. We focus on zero-shot experiments, where we used the following system prompt: "You are given a task to solve. Make sure to output an answer after 'Answer:' without any explanation." "
To ensure consistent evaluation, we utilize the following protocol: We first check if the generated text contains the keyword " "Answer:". If present, we consider the subsoquent text as the model's prediction. In situations where there are multiple ground-truth answers, we store the answers as a set using "," as the separator. We consider the prediction correct only when the generated set of answers matched the ground-truth set exactly.
2

---

Published in Transactions on Machine Learning Research (02/2024)
Figure 1: The spatial structures we examine for the underlying maps include squares, triangles, hexagons and rings. Additionally, we analyze tree structure to explore its relational nature.
Question: been given 2 by square grid. Starting from a vertex, you move along the adges the grid. Initially, you positioned at the bottom left comer of grid, where you
given equilateral trinngular map consisting of where first row las one tile and second row three tiles. Starting from will along the
Question: "You
these tiles, Initially, you posiliooeed the botiom left coreer of the map, where you find box lurtle. You move right by one slep. where you find hand plane. You up-right by step, where you find guacamole. You move down-tight by one step, where you find a table lamp. You move left by one step. What will you find?" Answer: "hand plane"
turlle move right by one step, where you a table lamp. You move up by one step, where you find an American black bear. You move left by step, where you hand plane, move down by one step, find?"
you
Answer: "box turtle"
(a) Square
(b) Triangle
"You have been given pointy-topped regular hexagonal tile consisting Startin vertex, you move along edges the tile. Initially, you positional at the corner of the where you (ind an pop. You move down- right hy step,
Question:
Question: "You have been given circular grid consisting of connected dets, Starting from you will move along the edges of the circular grid. Initially, you positioned the dot that's located at the top of the grid, where you find palace You
you
Hoxer. You move down by one step, where you where you find comburation lock. You
around the ring step clockwise direction, where you find gong. You move around the ring by steps in a clockwise direction, where you find shopping basket. You around the ring steps clockwise direction, What will you find!"
poke bonnet. You move up-left by one step, where you a spotlight You move up by one step, you find a gibbon. You move up-right by one step. What will you find!"
down-lell
Answer: "ice pop"
Answer: "gong""
(c) Hexagon
(d) Ring
Figure 2: Example question and its answer for square, triangle, hexagon and ring structure.
3

---

Published in Transactions on Machine Learning Research (02/2024)
3 Results
3.1 Do different spatial structure features affect model performance?
In Section 2 we provide an example that utilizes a square grid to assess the understanding of spatial structures in LLMs. In the example, exploit the concept of loop closure within the square grid for this purpose. Since loop closure also exists in other spatial structures, we also included triangles, hexagons and rings to explore the spatial understanding ability of LLMs less common 2D structures. Example prompt is in Figure 2 In this section, we are interested in analyzing how different factors contributes to the "difficulty" of a problem instance (via its prediction accuracy). There are two sources of difficulties: extrinsic difficulty, like the number of navigation steps ? in the prompts, which is independent of the specific structures; and intrinsic difficulty, which includes high-level structure features, like the structure type, and low-level structure features, like the number of edges and vertices.
With these goals, run logistic regression analysis to examine what factors influence the prediction accuracy. Specifically, using the square lype as the reference level for graph types, analyze the prediction outcomes via the following model:
Prediction correctness ~ intereopt + (graph type == hexagon) + 1 (graph type = triangle) + (graph type = ring) + number of edges + number of navigation steps.
Note that we only pick the number of edges as our low-level graph feature because the correlation between the number of edges and the number of vertices is very high (corrclation cocfficient was 0.998). We chose logistic regression as our model because the Prediction correctness is binary variable (for every prompt, the prediction is either correct (Prediction correctness = 1) or wrong (Prediction correctness = =]). We collect a total of 6,100 prediction results of different prompts using GPT-4 varying the structure type, the number of edges, and the number of navigation steps (i.e. hexagon: 1400 samples, ring: 1500 samples, square: 1800 samples, and triangle: 1400 samples). The summarized results are presented in Table
Estimate
Std. Error
z value
Pr(> [z]
(Intercept)
3.448
0.142
24.273
<2--16
(***)
type hexagon
-2.327
0.091
-25.595
<2e-16
(***
type is triangle
-1.820
0.082
-22.199
<2--1
(***)
type is ring
-2.117
0.103
2006116
<2e-16
(***)
number of edges
-0.002
0.002
-1.062
0.288
number of navigation steps -0.345
0.018
-18.924
<2e-16 (***)
Table 1: Logistic regression results. We see that the number of edges (an example of lower-level features) is not a significant predictor variable for prediction correctness (p value = 0.288). However, the higher-level graph structure (e.g. square or hexagon) is a significant. predictor of correctness (all p values < 26-16).
We observe that the prediction accuracy is not significantly influenced by lower-level structure features (the number of edges). However, it does depend on the structure type. Furthermore, the accuracy is heavily influenced by the number of navigation steps (p value < 26-16). This finding aligns with our intuition, as longer navigation poses greater challenges in tracking objects in a spatial map.
In the above, we model the difficulty of navigation tasks a function of both extrinsic and intrinsic difficulty attributes. Now we evaluate the difficulty of the task as function only of the graph structure, separately from the extrinsic difficulty. To achieve this, we condition the extrinsic difficulty by fixing the number of navigation steps (to be 8) in order to get measure of the "intrinsic difficulty" of each structure. We use a 3 by 3 square grid, size 2 hexagonal grid, size 3 triangular grid, and size 12 ring grid as the underlying maps. The prompts are generated in a similar manner as Figure We test on several LLMs. The results is in Figure
Navigation step refers movements asked be performed in the prompt. In the example prompts in Figure the number of navigation step is for square, for triangle, for hexagon, and 3 for ring.
4

---

Published in Transactions on Machine Learning Research (02/2024)
Model
GPT-4
4ame-2-708-cha--r
0.5
CodeLlame-SOb-insbuue
Accuracy
0.4
0.2
0.1
0.0
Ring
Hoxagan
Triangie
Structure
Figure 3: We compare the accuracy of the models across the different spatial structures. The random guessing accuracy is 1/8 since the predictions from random guessing are uniformly selected from the nodes encountered by the models, which corresponds to the local path with 8 navigation steps. GPT-4 have higher prediction accuracy than random guessing in square, ring and triangle structures, but worse in hexagon. ChatGPT exhibits lower prediction accuracy than random guessing across all of these structures. Llama2- 70B and Code shows a similar pattern to GPT-4. The error bars indicate 1.96 times standard error across 5 runs.
When comparing GP-T-35-tuuro and GPT-4, we find that GPTT-3.-turbo performs poorly across all spatial structures, while GPT-4 shows higher variation in performance. GPT-4 excels on the Square structure, ranks second best on the Ring and Triangle structures, and performs the worst on the Hexagon structure. Llama2-701 and CodeLlama-34B, while generally performing worse than GPT-4, exhibit a similar pattern of performance to GPT-4. We omit Llama2-7B and 13B from our discussion because they achieve zero (or very close to zero) accuracy across all structures. This indicates that tackling zero-shot spatial reasoning tasks may necessitate larger models.
The relative ease of the square grid in comparison to other grid types could be attributed to factors such as the prevalence of tabular data and city grid navigation within the model's training data. In addition, coding problems related maze exploration often involve navigating a two-dimensional square grid, while triangular and hexagonal grids are less commonly encountered. Thus, is conceivable that such exposure during pre-training makes GPT-4 possess enhanced understanding of 2D square grids. Analogously, for humans, individuals who grow up in cities with more grid-like structure may exhibit greater difficulty in navigating through less organized environments, such as older European cities, and vice versa (Coutrot et al. 2022). Additionally, we perform an experiment using the rhombus grid, which was achieved by rotating the square grid 45 degrees. Under the same experimental condition, we find that GPT-4 maintains an accuracy of 0.66, which is slightly lower than the original square-grid accuracy of 0.71 but still significantly higher than other structures. This outcome provides further confirmation that the specific grid structure with two axes contributes to its strong performance.
3.2 Is building local map more difficult than building full map and retrieving path?
In the previous section, LLMs were tasked with constructing local map gradually as they received new information, step at a time, which we refer to as the "local" setting. Alternatively, we can provide LLMs with the complete map from the start and instruct them to begin exploration from randomly selected initial location for a specific number of steps, which we refer to as the "global" setting. On one hand, local exploration may be deemed easier as it requires relaining less information. On the other hand, presenting the global map upfront could potentially aid in more accurate map navigation. To address this question, we compare the local and global settings in our spatial understanding task. For this comparison, we use square and ring structures since there are widely accepted methods of specifying global coordinates, making it easier to specify paths from randomly selected initial position. In particular, we provide the global map
5

---

Published in Transactions on Machine Learning Research (02/2024)
information in the following manner: For the square structure, we list the object names row by row. As for the ring structure, we list the object names starting from the top and proceed clockwise. We then have the model follow a fixed number of navigation instructions, just like the local setting. Example prompts for the square and ring structure under the global sctting are in Figure
Question: "You have been given a 3 by 3 square grid. In the 1st row, from left right, we have tiger, rifle, and English Setter. In the 2nd row, from left to right, we have soup bowl, mongoose, and ski. the 3rd row, from left to right, we have stopwatch, bow, and wolf spider. You start at soup bowl, then you go down by one step, then you go right by one step, then you go right by one step, and then you go up by one step. What will you find?" Answer: "ski."
"You have been given a circular path consisting of 12 connected dots. At the start, you positioned on the dot that is located at the top of the path, where you find a mop. Moving a clockwise direction from the mop, the clements the path home theater, a limousine, vacuum cleaner, a Bedlington Terrier, a fireboat, a red panda, projector, seashore, Redbone Coonhound, Scottish Terrier, and a laptop computer. Starting from the laptop computer, you move around the ring by steps a clockwisc dircction. What will you find?" Answer: "laptop computer."
Question:
(a) Square
(b) Ring
Figure 4: Example question and its answer for square and ring structure under the global setting.
The results in Figure 5 show that the global setting is slightly harder than the local setting for both 3 by square and size-12 ring structures (all under 8 navigation steps), except when the performance already low for the ring structure for Llama2 models, which shows a less clear pattern as such.
The results presented in Figure show that, in general, the global setting is more challenging than the local setting for both the square and the ring structures. However, this trend becomes less evident when considering the already low performance of Llama2 models on the ring structure, leading to a less clear pattern in this case.
Model
GPT.3
GPT4
06
Usama 70t chac-hhtthf
Coditlama- 34b-tnatuuc-lt
0.5
0.2
Square Local
Ring Local
Square - Global
Ring Global
Figure 5: Performance evaluated on GPT-4, Llama2-70B, and CodeLlama-34B. For both square and ring structures, observe that the prediction accuracy of GPT-4 using the local map is higher compared to the global map. TJama2-70B and CodeLLama-334 show similar pattern for the square, while the pattern is less clear for the ring. The error bars indicate 1.96 times standard error across 5 runs.
3.3
The order of presenting the map impacts spatial understanding
In the previous section, our approach to providing complete map information upfront the model has involved specific method. We describe the items in the map row by row, indicating their positions from left to right. For example, in the first row, have item A, item B, and item C. In the second row, we

---

Published in Transactions on Machine Learning Research (02/2024)
have item D, item E, and item F from left to right, and so on. However, there exist multiple ways to convey the same information. Here, we explore different approaches to feeding data into the model. In addition to the aforementioned method, we examine two alternative techniques: random and snake order. The random approach involves placing items in the map at random positions using the global coordinate system. On the other hand, the snake order method follows a specific pattern. In the first row, items are fed from left to right as before. However, when transitioning to the second row, we introduce the instruction "you move down by one step" to indicate the change in row. In the second row; items are then fed from right to left. By investigating these alternative data feeding methods, aim to understand their implications and assess their impact on the model's performance. Example prompts using random and snake order are shown in Figure B
Question: "You have been given 3 by 3 square grid with various
"You have been given 3 by 3 square grid. Starting from a vertex, you will move ulong the edges of the grid. Initially, you are positioned at the battom-left comer
Question:
items located at different indices: a restaurant is index (1, 2), half-tmick index (2, 2), a sleeping bag
index (2, 3),
grid, where you will find toilet paper, then you go right, where you will find stethoscope, then you go right, where you will find a screw. Then you go up, where you will find a modemn, then you go left, where you will find a pretzel, then you go left, where you will find an Otterhound. Then you go up, where you go right, where you will find tile roof, then you go right, where you will find a slip-on shoe. Now you have all the information the map. You start at the position where the pretzel is located, then you go right by one step, then you go down by one step, and then you go up by one step. What will you find?'
Scottish Terrier al index (2. 1), wok is at index (1. 3). balance
beam is index (3, 3), military uniform is at index (3, 1),
marimba at index (1, 1), and a radio telescope at index (3, 2). You start at the position where the half-track located, then you go left by one step, then you go up by one step. then you go right by one step. then you go right one step. then you down by one step, then you go down by one step, then you go left by one step, and then you go up by one step. What will you find?! Answer: "balf-track."
find an Asian elephant, then you
Answer: "modem.." "
(a) Random order
(b)
Snake order
Figure 6: Example question and its answer for by 3 square grid using random and snake order.
The results are shown in Table 2 Although the GPT-4's accuracy degrades for Random and Snake, is noteworthy that Random is better than Snake. We note that we omit the results for Llama-2 models because even Llama2-70B's performance was already very low (e.g. row-by-row's accuracy is 0.04 for Llama2-70F whereas GPT-4 achieves 0.55.)
Row-by-row Random
Snake
Snake Coord
GPT-4 Acc 0.572 (0.019) 0.495 (0.007) 0.440 (0.057) 0.525 (0.049)
The order of presenting the map impacts spatial understanding accuracy. Snake+ refers to the setting where we append the global coordinates of the location after each step. The mean and the standard deviation (shown in parentheses) are calculated over 5 different runs.
Table 2:
What could potentially explain these phenomena? It is reasonable to speculate that different methods of inputting data could influence how LLMs internally represent spatial relations. For instance, when utilizing the row-by-row approach, the LLM can register these items in key-value dictionary, where the row id serves as the key and a list of objects represents the corresponding value. The 'random' approach also enables the LLM to store a key-value dictionary where the key denotes the location address and the value denotes a single item, which can be leveraged for navigation purposes later on. On the other hand, the 'snake' order approach necessitates the LLM to simultaneously handle the storage of object item information and spatial relational understanding. This added complexity potentially complicates the task.
To investigate whether or not such key-value data structure plays role, we perform additional ex- periment where we add the global coordinate information to the snake order approach. We see that this approach indeed increases the accuracy from 0.40 to 0.56, corroborating our hypothesis.
3.4 Relational structure: Tree
In addition to spatial structures, relational structures can also be represented using connected graphs. In this section, we examine tree structure. Unlike spatial structures, the hierarchical structure of a tree is
7

---

Published in Transactions on Machine Learning Research (02/2024)
more naturally presented in global setting, where all objects are provided at the beginning, and relational questions can be asked subsequently.
To ensure comparability with the square and ring structures, we set the number of nodes in each tree to 9. For comparison, we also included a 3 by 3 square grid and a 9-node ring this experiment. The exploration steps are set. to for all structures. For the tree structure, we utilize the same ImageNet object labels, but focus on relational questions that involve 4 steps, such as "What is the cousin of A?", "What is the preat-great-grandparee of A?", and "What is/are the tat--gaaatggrnchhlde illustrative example question and its corresponding answer can be found in Figure
of A?". An
Question: "You have given structure with nodes. The node great white shark. The great white shark has 2 children: a garler snake und Gila monster. The gurter snake has children: a jigsaw puzzle and a moped. The jigraw puzzle has child: Tibetan Terrier The Tibetan Terrirr has a child: an umbrella. The umbrella has no children. The Gila monster has 2 children: a has no children. The horse-drawn vehicle has no children. What the cousin of the mopod?"
children. The moped
stocking and horse-drawn
Christmas stocking
Answer:
stocking. horse-drawn
Figure 7: Example prompt for tree structure.
We observe that for GPT-4, while the tree structure performs worse than the square structure, it outperforms the ring structure. On the other hand, for GPT-3.5-turbo, the tree structure exhibits better performance compared to the square structure. We observe that just like GPT-3.5-turbo, Tree performs better than Square for all Llama-2 models. The only exception is GPT-4; this further demonstrates that GPT-4's exception ability to comprehend the square structure. We also note that Ring harder than Square for all Llama models and GPT-4. Further investigation into how relational structure, spatial structure, and model size impact performance would be intriguing topic for future rescarch.
The evaluation results are depicted in Figure g
Model
0.5
CPT-3.5-uuto
GPT-4
0.5
2-70b-chat-tf
Code Stt-mmmm------inuuttttt
0.4
Acorssy
0.2
0.1
Square
Ring
Struclure
Figure 8: We evaluate the prediction accuracy of the models on 9-node tree, 3 by 3 square, and a 9-node ring structure with 4 exploration steps in the global setting. Comparing the performance of GPT-4 and random guessing, GPT-4 outperforms random guessing with higher prediction accuracy, with the order of accuracy being square > tree > ring. GPT-3.5-turbo also performs better than random guessing on the tree structure, but worse on the square and ring structures, with the order of accuracy being tree >> ring > square. Just like GPT-3.5-turbo, Tree performs better than Square for all Llama-2 models. The error bars indicate 1.96 times standard error across 5 runs.
3.5 Grid size inference from sequences navigational instructions
In the preceding sections, we examined the capability of LLMs understand the spatial and relational structure of a map. In this section, our focus shifts investigating whether LLMs can infer the global size of map based solely on a sequence of local navigational actions. Simultaneously inferring the extent. of
8

---

Published in Transactions on Machine Learning Research (02/2024)
map while navigating is an important component of navigation cognitive task. Inferring the extent of map is a non-trivial task which requires integrating information about the history of actions and reasoning about what it implies spatially. Indeed, humans often have a sense of the size of the places they visit, not only how to navigate from point A to point B. Specifically, we provide navigational instructions that guide the exploration of all locations within a reclangle. As before, we also provide what item the agent finds each step. Then we ask LLMs about the height and width of the rectangle. This task necessitates LLMs maintain the entire path in order to accurately deduce the overall dimensions of the rectangle.
Table 3 iillustraes the accuracy comparison of GPTT4 for different size configurations of the same area (e.g., 2 by 6, 3 by 4 for an area of 12). We prepare 200 samples for each area. We observe general trend where accuracy decreases as the length of the sides increases and as the area size of the rectangular grid increases. We omit the results for -.T-,,5-tuubb, Llama-2 2- 2-70B and CodeLlama- 34B because these models were not able to infer the size of rectangle.
3x4 or 4x3
6x2 4x6 or 6x4 3x8 or
2x12 or 12x2
GPT-4 Acc 0.612 (0.209) 0.103 (0.058)
0.162 (0.071).
0.016 (0.017) 0.005 (0.007)
Table 3: Grid size inference performance of GPT-4. The mean and the standard deviation of prediction accuracy (shown parentheses) are calculated over 10 different runs.
Additionally, we evaluale the same setup but exclude object item information during navigation. In this case, GPT-4 relies solely on directional information (e.g., "you go up by one step, then you go down by one step, ctc."). Shown in Figure are the mean accuracy and standard deviation across 10 different runs, grouped by the shape of the rectangles. We see variations in accuracy across based on the shape of the rectangles.
1,0
Prompt
Direction Item
Direction
Acarsay (mean)
0.4
0.2
0.0
2x3 3x2
3x4
4x3
2x6
6x2 Structure
4x6
6x4
3x3
4x4 5x5
Figure 9: For grid size inference, we sec variations in accuracy based on the shape of the rectangles, specifically whether they were 'tall' or 'fat'. (e.g. "4x3" is consistently better than "3x4" for prompts that combines both directional and object item information at each step, while "3x4" consistently better than '4x3" for the approach that only uses directional information.) The error bars indicate 1.96 times standard error across all runs.
4 Error analysis
In this section, perform detailed analysis of the errors produced by GPT-4, to assess whether it is modelling the correct topology. In our error analysis, we focus on GPT-4 because of its relatively strong performance, which reveals intriguing error patterns.
To study the extent to which LLM understands the topology of a given spatial structure, we examine what types of mistakes it makes. In our spatial understanding task, when LLM makes mistake, in the overwhelming majority of cases, it provides the name of an object at different location (rather than naming object that did not appear at all in the prompt). Therefore, we measure the distance between the
9

---

Published in Transactions on Machine Learning Research (02/2024)
correct location and the location of the predicted object with respect to the underlying topology. If the LLM represents the spatial structure of the map, the distribution of these distances will tend to cluster at small values, which we call spatial-topology bias. That is, if the LLM represents spatial structure, we should expect more mistakes for objects topologically close to the correct location, and fewer mistakes for objects larther away. A natural choice for measuring distances in grids is the shortest distance between two vertices in the grid.
A
E
G
H
An example of two distance metrics in 3 x 3 square grid in the local setting. If the path in the local setting starts at node and follows through B, C, F, I, H, E, and finally ends at B, and the LLM predicts F instead of B, then the temporal distance between B and F is 4, while the spatial distance is 2.
Figure 10:
LLMs may also show non-topological biases their predictions, an instance of which is the temporal bias an inclination predict objects that are observed in the textual vicinity of the ground truth item in the given prompl. To investigate the presence of this bias, we measure the temporal distance the number of objects between the first occurrence of the ground truth in the prompt and the predicted item (including the predicted item). An illustrative example of the temporal and spatial distances in the local sctting is shown in Figure 10
In the following, we examine the error distributions of GPT-4 in square, triangular, and hexagonal grids. For each experiment, we collect 1000 predictions from GPT-4, and analyze them along with their corre- sponding prompts and ground-truth correct answers. Out of the 1000 prompts, we only consider the subset of predictions in which the model was wrong. We compare GPT-4's errors with the random distributions of spatial and temporal distances with respect to uniform baseline. To do this, we randomly pick one of the prompts the model is tested with. Next, we record the location of the ground truth object based on the prompt. Then, for the global setting prompts, we at random select a location from the entire grid; for the local setting prompts, we at random select one of the visited nodes along the path. We then calculate both the spatial or temporal distances between this randomly selected location and the ground truth location. We repeat this procedure for 100,000 times to generate the error distribution for the uniform baselines.
4.1 Comparing error distributions of square, triangular, and hexagonal grids
In our analysis, we used a by 3 square grid, a triangular grid with a size of 3, and a hexagonal grid with size of 2. We chose these grid configurations ensure a fair comparison across different grid structures, allowing for prompts conducting 8 navigation steps in each grid. The precise shape of the size-3 triangular grid is shown in Figure 15 in Appendix, and the shape of size-2 hexagonal grid is shown in Figure 1 The results for the local setting are shown in Figure
For the square grid, GPT-4 tends to make more errors at spatial distances of 1 relative to random baseline, indicating a spatial-topology bias (Fig. 1h, left). However, temporal distance also shows a stronger peak at the value of (relative to the uniform bascline; Fig. 11p, right) indicating that having two items more closely located in the prompt is also an effective predictor of GPT-4's errors. In Appendix Figure 19 we also generated plot for the conditional distribution of TD when SD=1 to further validate the temporal bias.
In the case of the hexagonal grid (Fig. 11p), we see a lack of spatial-topology bias, with the distribution of distances peaking at 2 (instead of 1). We also do not see a temporal bias in GPT-4's behavior, again with a distribution peaked at the temporal distance of 2. This indicates the presence of some other source of non-spatial bias besides the temporal bias. Closer inspection revealed that whenever GPT-4 makes an error in hexagona grids, these errors are often due to the model predicting the very first object on the path, which often ends up having spatial and temporal distances of 2 from the ground truth correct answer.
10

---

Published in Transactions on Machine Learning Research (02/2024)
37
67
60
n6 3.00
67
07
Zoh
63
62
0.1
Spannl Distance
Temporal Distatoe
Spatal Distance
lemponal Distance
Spatial Distace
Temporal Distanne
(5) Square
(b) Hexagon
(c) Triangle
The spatial and temporal distance (SD, TD, respectively) histograms for square, hexagonal, and triangular grids under the local setting. Blue histograms show random basclincs. Orange histograms show the observed distribution of errors as the spatial (left) and temporal (right) distances ween the ground truth and GPT-4's predicted locations. (a) In grids with square topology, GPT-4 makes more errors both when SD is 1 and TD is 1, compared to the uniform baseline, meaning that both spatial and temporal biases contribute to GPT-4's errors. (b) In grids with hexagonal topology, we do not observe spatial nor temporal bias. (c) The simultaneous lack of spatial bias and the presence of temporal bias indicate that GPT-4 was not able to accurately construct the triangular grid.
Figure 11:
For the triangular grid (Fig. I1c). the distribution of SD from GPT-4 and random guessing is almost the same, suggesting that there is almost no spatial bias. However, there is a spike when TD = 1. We find that among all the instances where TD equals 1, the proportion of predicting the starting position 0.416.
Hence, it appears that the temporal bias accounts for more than half of the bias observed in the triangular grid. In terms of the starting position bias of the square grid, see Appendix Section
0.6
0.6
0.5
0.5
Frequency
0.4
Frequency
0.4
0.3
0.2
0.1
0.0
0.0
4
2
3
8
Spatial Distance
Temporal Distance
The spatial and temporal error histograms for the by 3 square grid under the global setting. We see spike when the temporal distance (TD) is and 6, indicating an effect of spatial distance these two TD values correspond to the spatial distance of and 2.
Figure 12:
we present analyses for the global structure 3 When dealing with a square grid, we provide GPT-4 with the complete map upfront by listing each object row by row. For instance, in prompts for the square grid structure the global setting, such as "In the first row, we have item A, B, and C. In the second row, we have item D, E, and F, the temporal distance between and would be 3. If GPT-4 represents the square grid would decrease steadily. However, we observe spikes at 3 and 6 in the temporal distance, which correspond to spatial distances of 1 and 2, respectively. This finding suggests that GPT-4 makes more errors when objects are closer to the ground truth in terms of spatial structure, rather than temporal distance. It supports the notion that GPT-4 actually models some aspect the two-dimensional structure. Furthermore, we investigate whether there are any discrepancics in error distributions when the distance is calculated using either row-major order or column-major order. Figure demonstrates that there is an almost symmetrical distribution between and columns. This provides additional evidence that GPT-4 recovers the structure of the two-dimensional array.
In Figure 12
mieedimesioonl array, expect that the frequency of temporal distance
We omit the ring structure because each movement can involve many steps around the ring (e.g. "move by 3 steps clockwise") which effectively introduces edges between graph nodes, and thus renders analysis based spatial distance less meaningfin full
11

---

Published in Transactions on Machine Learning Research (02/2024)
0.3
Frequency
Frequency
0.1
7 8 Distance based row-major order
2
3
2
3 4
7
8
Distance based column-major order
Figure 13: The row-wise and column-wise error histograms for the by square grid with navigation steps under the global setting. We see that both row and column-wise histograms are almost identical, which suggests GPT-4 does not have bias for column
5 Comparison to human baseline
Finally, we have conducted human experiments to assess the average human baseline performance. These experiments focused on local navigation tasks using the structures shown in Figure 3 which include a 3 by square grid, size hexagonal grid, a size triangular grid, and size 12 ring grid. For cach of these structures, of 80 candidate prompts. Then, for each participant in the experiment, we randomly chose 10 prompts from this pool of candidate prompts. Participants were asked to provide textual answers to these questions. Additionally, we included 4 attention check questions. These questions were intentionally designed to be easy so that we can assess whether participants were providing meaningful answers, and were used as an exclusion criterion in our analysis. These attention check questions were drawn from the set of questions associated with very simple structures, such 2 by 2 square grid, size 1 hexagonal grid, size triangular grid, and a size 5 ring grid. These attention check questions were distributed randomly throughout the survey.
randomly selected 20 prompts from the previously used dataset, resulting in total
We established a criterion to measure participants' engagement in the experiment: if participant made more than one mistake in the attention check problems, we would exclude their response from the analysis. Specifically, each participant had 30 mimutes to solve 14 problems, which consisted of 10 regular problems to evaluate human performance and 4 attention check problems. The participants were not informed whether the given problem was regular task or an attention check.
In this experiment, we received completed responses from total of 23 participants. Details about the participant recruitment process and the selection criteria are discussed in Appendix Section F Among them, 5 participants did not meet the attention check criterion mentioned earlier. Their responses were excluded, leaving us with the responses of the remaining participants for analysis. In total, we collected 180 question- answer pairs, distributed as follows: 48 pairs for the 3 by 3 square grid, for the size 12 ring grid, 48 for the size 2 hexagon grid, and 43 for the size 3 triangle grid. We converted all the responses to lowercasc and removed all English articles (a, an, the) from both the human responses and the ground truth. A response considered "correct" only if it matched the ground truth exactly. The results be found in Table Similar analysis under different criterion of participants' engagement measurement is conducted with
results in Section
Square
Ring
Hexagon
Triangle
Aggregated
Human
0.90
0.78
0.41
0.58
0.67
GPT-4
0.69
0.22
0.05
0.20
0.29
Table 4: Human baseline performance compared against GPT-4. The values of the GPT-4 accuracy are taken from Figure 3
The aggregated accuracy across all the structures 0.67, and the accuracy for cach structure 0.90 for by square grid, 0.78 for size 12 ring grid, 0.41 for size 2 hexagon and 0.58 for size 3 triangle grid respectively. For GPT-4, the accuracies for each structure from Figure 3 Although human responses
12

---

Published in Transactions on Machine Learning Research (02/2024)
are not perfect, they still outperform GPT-4 by a significant margin. It is also interesting to note that, like GPT-4, non-expert humans struggle with non-square grid shapes. The differences in human performance across different structures may be attributable to familiarity with such structures in every-day life and how well these structures can be described in natural language. For example, in contrast to the square and ring structures, hexagonal and Iriangular grids are much less frequently encountered everyday life. Moreover, these structures are intrinsically more difficult describe in natural language.
6 Related Work
Some research suggests that language models have the ability to acquire implicit world models (Abdou et al. 2021 Patel & Pavlick 2022 Li al. 2021). Spatial understanding particularly intriguing because it might seem counterinluitive that a language model, which lacks visual or sensorimotor input, can comprehend spatial structures.
Patel & Pavlick (2022) provide evidence that GPT-3 is capable of grounding spatial and cardinal direction terms in a lext-based grid world. They present contextual examples of cardinal directions (e-g,, north, east, northeast) and evaluate whether the model can generalize to a different subset (e.g,, south, west, southwest). Our work expands upon these findings by assessing spatial understanding that necessitates the accurato construction and retention of representations of spatial structure in more challenging tasks.
Previous studies have employed text-based navigation tasks to evaluate language models. Bubeck et al.) (023) evaluate GPT-4 across varions domains, including mathematics, coding, vision, medicine, law, and psychology. In one task involving embodied interaction, they create a simple map and prompt GPT-4 to explore it interactively using actions such as left, right, up, and down. A human provides feedback during the exploration. They demonstrate that GPT-4 successfully tracks all the locations and visualizes them using a generated program. However, their study only examines single instance of square-grid map and does not thoroughly investigate the extent of GPT-4's spatial understanding. Another similar task is present in (Whittington et al., 2020) but the goal is to test structural generalization. [BGG-bench collaboration (0023)
has task where the agent is required to determine whether it would return to its original starting position based on a set of navigation instructions. However, this task only involves providing "yes/no" answers. In the NLP community, more complex spatial reasoning tasks have been introduced, including those involving multi-hop reasoning (Shi et al. 2022) and tasks that involve understanding of textual descriptions for intricate visual scenes (Mirzaee et al. 2021). ). In contrast, our study investigates various spatial structures, such as rings, trees, hexagons, and triangles, while also requiring the language model remember and track object names.
Moreover, LLMs increasingly underlie embodied agents, and exhibit impressive spatial reasoning capability (Singh et al. 2023 Liang et al. 2023 Huang et al. 2022 Rajvanshi et al. 2023). Our work explores the basic spatial reasoning skills of these models via controlled experiments using synthetic grid data. This approach allows us to identify both their strengths and weaknesses.
A concurrent work (Momennejad et al. 2023) also evaluates LLMs in terms of cognitive mapping and planning abilities. While they examine wide array of tasks, their focus is on planning (e.g. generating path between given vertices), and their graph structures are limited to trees, linear paths, and social graphs. In contrast, our work is centered on path integration for sensory prediction and perform systematic evaluation of basic topological spatial structures, coupled with in-depth error analysis with experiments on human participants.
7 Conclusion
In this work, we investigated whether LLMs could build representations of spatial structure implicitly from their sequential inputs. We found that LLMs are able to answer questions about spatial relationships, and tend to make errors that reflect spatial proximity. However, the details of their performance depends on the task and structure- square grids are casier than other structures, and local presentation casier than global. Overall, results suggest. that LLMs implicitly learn to represent aspects of spatial structure,
13

---

Published in Transactions on Machine Learning Research (02/2024)
though their performance is far from perfect. These findings contribute to the growing literature the aspects of world knowledge that LLMs implicitly acquire from their language-only training.
Limitations
In our investigation into the spatial understanding of LLM, our focus lies on the zero-shol setting, We have conducted a preliminary study to examine the impact of chain-of-thought style prompting, and we observe an increase in performance for GPT-3.5. However, beyond the 3-shot sctting, the performance improvement reached plateau. Details of these studies can found Fig 14 Appendix, and we encourage future research to explore the effects of additional variations of chain-of of-though prompting on LLM's spatial understanding. We note that the lack of complete detail about the training of GPT-4 makes understanding the origins of its strong spatial performance somewhat challenging. Investigating how smaller models fine-tuned on spatial tasks exhibit spatial underslanding would be an intriguing subject for further study.
Broader Impact Statement
Before the human experiment starts, all participants are provided with an online form with information about the experiment, including the purpose, brief description, the procodure, confidentiality, and voluntary participation. The participants are informed that their responses will be confidential and they are free to decline or end participation at any time. Tn the "agreement to participate" section, we need participants confirm that by clicking the "start" button, they acknowledge that they have read the above information, and agroe to participate in the study. The experiment only starts after they click the "start" button.
The participants were recruited from online crowdsourcing platform (Prolific.comm) it is platform for researchers to post behavioral studies and recruit participants. We set the only participants' criteria to be "fluent in English" as all our prompts and questions are written in English. Our study is evenly distributed all available participants. The demographic data of the 23 participants that took part in our study is as follows: age range (min: 21, max: 62, median: 36, mean: 41.9), gender (Male: 12, Female: 10, No data: 1), employment status (Full-time: 9, Not-in-paid-workk 5, part-time: 3, unemployed: 2, No data: 4). Although seek mitigate the bias by setting as few criteria possible, there is still bias toward English-speaker with computers and internet connection, as well as some characteristies of the WEIRD (Western, Educated, Industrial, Rich, Democracies) population.
As large language models continue to advance and find applications in real-world scenarios, becomes increasingly crucial to assess the risks and unintended consequences associated with such LLMM-ased appli- cations. Although our study on the spatial understanding LLMs may not directly address the reduction of harm and bias in t these models, we believe it is important to comprehend their inner workings. We hope that our work contributes to the ongoing effort of understanding and exploring the mechanisms at play within LLMs.
References
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Paylick, and Anders Søgaard. Can Language Models Encode Perceptual Structure Without Groudiing A Case Study in Color. In Proceedings of the 25th Conference on Compulational Natural Language Learning, pp. 109 132, Online, November 2021. Association for Computational Linguistics. doi:
BIG-bench collaboration. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions Machine Learning Research, May 2023. ISSN 2835-8856.
Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4, April 2023.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Alexandra O. Constantineseu, Jill X. O'Reilly, and Timothy E. J. Behrens. Organizing conceptual knowledge in humans with a gridlike code. Science, 352(6292): 14644)4))4441488, June 2016. doi: 10.1126 0e22///scieneeaaal000
A. Coutrot, E. Manley, S. Goodroe, C. Gahnstrom, G. Filomena, D. Yesiltepe, R. C. Dalton, J. M. Wiener, C. Hölscher, M. Hornberger, and H. J. Spiers. Entropy of city street networks linked to future spatial navi- gation ability, Nature, 604(7904):104 110, April 2022. ISSN 1476-4687. doi:
Mona M Garvert, Raymond J Dolan, and Timothy EJ Behrens. A map of abstract relational knowledge in the human hippocampal entorhinal cortex. eLife, 6:e17086, April 2017. ISSN 2050-084X. doi: 10.7554/
eLife.17086.
Torkel Hafting, Mariannne Fyhn, Sturla Molden, May-Britt Moser, and Edvard I. Moser. Microstructure of a spatial map in the entorhinal cortex. Nature, 436(7052):8001-83 806, August 2005. ISSN 1476-4687. doi: 10.1038/nature03777
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, Pierre Sermanct, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Conference Robot Learning, 2022. doi: 10. 10.48550/ ARXIV 2207.05607.
Colin Lever, Stephen Burton, Ali Jeewajee, John O'Keele, and Neil Burgess. Boundary Vector Cells in the Subiculum of the Hippocampal Formation. Journal of Neuroscience, 29(31):9771-99) 9777, August 2009,
ISSN 0270-6474, 1529-2401. doi: 10.1523/
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit Representations of Meaning in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joinl Conference on Nalural Language Processing (Volume 1: Long Papers); Pp. 1813 1827, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011 acl-long.143 143.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code Policies: Language Model Programs for Embodied Control. 2023 IEEE International Conference on Robolics and Aulomation (ICRA), pp. 9493 9500, May 2023. doi: 10.1109/ICRA48891.
10160901
Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning. In Proceedings of the 2021 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3582-4598, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 maacl-main.36i
Ida Momennejad, Hosein Hasanbeig, Felipe Vieira, Hiteshi Sharma, Robert Osazuwa Ness, Nebojsa Jojic, Hamid Palangi, and Jonathan Larson. Evaluating Cognitive Maps and Planning in Large Language Models with CogEval, September 2023.
J. O'Keefe and J. Dostrovsky. The hippocampus spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Reseurch, 34(1):171 175, November 1971. ISSN 0006-8993. doi: 10.1016/ 0006- (006-8993(71)90338.88
Roma Patel and Ellie Pavlick. Mapping Language Models to Grounded Conceptual Spaces. In International Conference Learning Representations, January 2022.
Abhinav Rajvanshi, Karan Sikka, Xiao Lin, Bhoram Lee, Han-Pang Chiu, and Alvaro Velasquez. SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments, September 2023.
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11321-1111 June 2022. ISSN 2374-3468. doi: 10.1609/naai. v36100....3100.113333
15

---

Published in Transactions on Machine Learning Research (02/2024)
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Program generation for situated robot task planning using large language models. Autonomous Robots, 47(8))999 1012, December 2023. ISSN 0929-5593,
1573-7527. doi: 10.1007/1105144033-00.3333
Edward C. Tolman. Cognitive maps in rats and men. Psychological Review, 55:189 208, 1948. ISSN 1939-
1471. doi: 10.1037/h0061622
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami,, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Fric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyan Zarov, Yuchen Zhang, Angela Fan, Mclanic Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023.
James C. R. Whittington, Timothy H. Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy E. J. Behrens. The Tolman-Eichenber Machine: Unifying Space and Relational Memory through Generalization the Hippocampal Formation. Cell, 183(5):1249 1263.e23, November 2020. ISSN 0092-8674. doi: 10.1016/j.cell http:/jeell.2222.1101 10.0.1..024.
James C. R. Whittington, David McCaffary, Jacob J. W. Bakermans, and Timothy E. J. Behrens. How to build a cognitive map. Nature Neuroscience, 25510))1257-1277, October 2022. ISSN 1546-1726. doi: 10.1035//411533-02--000
A Additional prompt examples
We provide additional examples of prompts we used with bigger size for different structures in Figure
B Temporal distance in the global setting
For instance, in prompts for the 3 by 3 square grid structure in the global setting, such as "In the first row, we have item A, B, and C. In the second row, we have item D, E, and F, ...." the temporal distance between A and D would be 3, while the spatial distance betweon A and D is
C Starting position bias of the square grid
In the 3 by 3 square grid, the ground truth is only present at the 1st, 3rd, 5th, and 7th nodes of the path. To qualify as a starting position bias when Temporal Distance (TD) equals 1, the ground truth would need to occur at the 2nd node of the path. Consequently, there no contribution from the starting position bias when TD = 1.
Chain-of-thoughe prompting
the case of in-context learning experiments, we used the following system prompt: "You are given a task to solve. Make sure output final answer after "Answer:'"" We include in-context examples in the user prompt in the following format: "Question:)p [question] \n Explanation: (n [explanation] \n Answer:)n
[answer]." An example prompt is given in Figure 16
16

---

Published in Transactions on Machine Learning Research (02/2024)
Question: have been given 3 by squze goid. ling from vertox, you will move aloug the edass of the gnid. Iniially, you are posiitined as the top right comner the grid, wierr you find a breesiplate. You move dowa by use stop. whene you find a hamminghind You move donn by one step, whene you find an casiere damorddeack ratharake.. Yos more lift by one stap. where you find a biskeball You move up by use sap, where you fina i pulled rickllun. You move jeft by ous step, where you dad a Black and Tai Coothoound You move up by sey, where you find a beeakwater. You move right by step, where find oscumber You move right by one step. What will you find!"
Question:
"You bess Iniisily, you sea positional tup the colge
one step, you find gryser move right by stop, where you
gnd, where you rind
move
right jellyfish move muchine, move dawn by une step. whon you etep, where you find megalihs You move up by one step. What will you findt"" Anewer: "ellyfish"
up stop,
you
impais
step, you
find box tuttle move
one step, where you an espresso move by ane
Answer "bresspplat""
(a) 4 by 4 square grid with 8 navigation steps
Question: "You have been given to equiltent trinngsla tile map consitiing of rows, where the first row has ore like, the secend me has three nllss and so ot, so thee the da now how 1ht-1 nile Souring from verses, you will mose allng the olye of thos likes Intially, you ane poottoncal at the top urmer of the map, where you find an ear. You move down left by one step, where you dind z sundial, You move dowzz-lett by one step, where you find a flagpole. You move downingghtt by one step, where you find a West Hightnd White femner You move right by
Quection: "You have been given an equiatent trinngzlar tile map cousiting of rows, where the firs row has one nle, the second now has thrre tite, and so on, so that the th has 2fi rike. Starting from a vortes you will move along the cégees of those tlle Iniisilly you see prsdiional at the botton right corner the map, where you find meesuring cup. You move left by one step. witere find a benoculaas, You move left one step, where you find mony dal teleghonn. You move luft by one step, where you find a space hester You move up-ight by whone find a belo You mover right one whre find on Africn bush clepitant You mave You luli one slep, where you find English Satter. You mave direcnleii find
where find Sinset Spanid You move up-right by slzn, where find booker monkey. You mave left by where you fin You sp-lett step, What you find?* Answers "oor"
where
where find a lelle.
find Biaact Huund. You mave ap-ightt by
What Anower Toolo rie"
(b) 3ize-3 triangular grid with 8 navigation steps
Question: "You have been given ponnt--oppec regular besagonal the mop consistig 2f rowa, where the firtt row has oue the and the secord now his two
Quretion: "You have hore giver a pointy-teppel negslor hetagorall rile map consisting of rows, where the first cow one tle and the secoad now has two
tiles. Starting from a vestes, you will mave alona die odges of these tiles. Initaals: you are positooeed at the botton left comer of the map, where you find dniling rg. You move by one step, where you find corcusel. You move right by one stp, where you find loupe You move fiid a gown. You mave right ouz stop, where you find púnner You move down right by one stop, where you End a black footed feneet You move down by siss, where you find a station wagon. You move dowr, left by one step, where find towel, Amswer: "Toupe"
tiles Staring from vente, you will move zlong the edges of these tiles. lnmally. pualiione at the conner the mup, where you find bucket. You
move dowes left one stop, where you find gown. You more down by one step, where you find macke. You move down--ight one where find arrpiihous vehicle, You You down-left left step, where find libmrry move Tishirt. You noged You up- right by step, What you find?"
one step, where you
down by
where find CD
whhre you find
up-let left step. What
find?"
"nacke""
(c) Size-2 hexagonal grid with 8 navigation steps
Question: "You hove been given circular grid consutiing of 12 connected dors. Staming from vertes, you will move slong the edges of the circular grit. Inmialyy you am positined you find a zilk can. You move arcund the ring a steps is counnes clockwise direction, where you find mustroo.. You move around the ring by 9 sops in cledoo-- dinecton, where you find spoilight. You move sround the ring by I sup a counen-clocke cocoooooiiin soloiie dinection, whene you find a shoppin hackri You move around the ring by 3 stps in commireciaciiiin dinettion where you find safery piu. You move around the ring by steps in counter-
Question: "You have bees given circular gnd consisting
connected dots.
Stating fhom a veree, you will move along the edges of cinolar gnd Inmally, you positoneed on the doe thar's located the top the grd,, whire you find sleat paids. You move cround the ring by 9 staps a clockwise direction, where you find a pickup track. You move around the ring by 6 steps as counter
doe that's lncated the top the grid, where
clockwase direction, where you find car wheel You move around the ring by step in clocloriis di rochions whem you find valture. You move around the ring by 3 strps in a clockoiie dimection, where you find is quill. You mose sroundi the ring 5 steps couneer dl clockwise director, where you find dountaii pen. You move acound the ring tind a snock You move sround the ring steps in conter-clollosee dieveriorn where find military uniforn You secund the rng strps in eleckaiic dinetiion. What will you find "vulture"
clockvise direction, where you find poiee botnet. You move sround the ring to sigs in cledowsee dinectio,, where you find Standard Schnauner You 6
steps counter clociwiie directionn where you
move sround the shovel You move aourd the ring by expe in clockriie diratiinn. What will fud?""
countmeledoowrs dinection, where find
Answer: "sateyy
(d) 12-node ring with 8 navigation steps
Figure 14: Example question and its answer for square, triangle, hexagon and ring structures under local setting.
17

---

Published in Transactions on Machine Learning Research (02/2024)
size-3 triangular grid
Figure 15: Size-3 triangular grid.
"You are at the top left corner of 2 by 2 grid, where you find box turtle. You move right by one step, where you find table lamp. You move down one step, where find American black bear. You move where you find hand plane. You move
Question:
left by one Whut do you find?"
one step.
CoT: "We can describe our movements in the 2 by 2 grid starting from the
the top left corner follows:'n- Move right from (1.1) to (2.1) Move down from (2, to (2,2)in- Move left from (2,2) to (1,2) Move from (1,2) to (I,1) result, we reach the coordinate (1,1) where we find the box turtle. Therefore, the unswer is box turtle." Answer: "box turtle".
Figure 16: An example of Chain-of-thougli style prompting for the 2 by 2 square grid.
0.4
0.3
Accuracy
0.2
0.0
0-shot
0-shot
1-shot 3-shot 5-shot
GPT-4
GPT-3.5 GPT-3,5 GPT-3,5 GPT-3.5
Figure 17: When chain-of-thought style prompting is employed, the accuracy of GPT-3.55 shows improvement as we increase the number of examples provided in the prompt. However, it reaches a plateau when the number of examples ranges between 3 and The setting here is Ring of size the number of navigation steps is 3.
18

---

Published in Transactions on Machine Learning Research (02/2024)
1-shat CoT
3-shot CoT
1.00
0.75
Accuracy
0.50
0.25
0.00 2 by
by 3
4 by 4
5 by
Figure 18: The accuracy for GPT-3.5 vs. the size of square structures in the global setting with navigation steps being Although 3-shot CoT improves over 1-shot CoT, the performance platcaus, as we increase the size of the square structure.
E Conditional distribution of temporal Distance
Condition on spatial distance = 1, we can see that the temporal distance has clear bias towards smaller values in Figure
19
2J7
Portos or uaan spmu Ontones
1.0
Wompt eaen Sparal Diraccs
0.7
Pioeppe for Spatu Untincc
28
28
0.8
Fraquattcy
05
04
02
D1
60
4 DesanoalDsaannee
0o
2
Distar
The temporal distance histogram conditioned on spatial distance (SD) for the square, SD = 2 for the hexagonal, and SD 1 for the triangular grid under the local setting. These spatial distance conditions are selected because of its maximum frequency in Fig 11
Figure 19:
F Human experiments
Recruitment. Process We recruited participants online through an online crowdsourcing platform (Prolific.com), it is a platform for researchers to post behavioral studies and recruit participants. The way it works is that first create web interface for our study and upload it to Prolific with estimated finish time, hourly payment and participants criteria, then the platform will send out the study invitation to participants who match the criteria. Participants take part in the study online and remotely using their own laptop.
Impact the exclusion criteria In our experiment, we set the participants' criteria to be fluent in English as answering the prompts required the understanding of English, and the study be evenly distributed all available participants. The demographic data of the 23 participants that took part in our study is follows: age range (min: 21, max: 62, median: 36, mean: 41.9), gender (Male: 12, Female: 10, 5, part-time:3, unemployed: 2, No data: 4), student status (Yes: 3, No: 17, No data: 3). In the experiment, we seek to mitigate the bias by setting as few criteria as possible, but it is inevitably skewed to people with computers and internet connection, and some characteristics the WEIRD (Western, Educated, Industrial, Rich, Democracies) population, which is unfortunately still a common practice across much of cognitive science and psychology research.
No data: 1), employment status (Full-time: 9, Not-in-paid-wont
Different criterion for measuring participants' engagement As one of the reviewer suggests, given the difficulty of non-square grid questions, conducted similar analysis under the criterion that only
19

---

Published in Transactions on Machine Learning Research (02/2024)
the participants that gets the square attention check wrong will be excluded. Among the 23 participants, only one participant failed the attention check for the rectangular grid, and this participant got 3 out of 4 attention checks wrong. If we exclude this participant' response and calculate the prediction accuracy for each structure among the rest of the participants, we have 0.91 for square, 0.76 for ring, 0.36 for hexagon and 0.48 for triangle. The overall accuracy is 0.62. The accuracy are generally lower (except for the square) compared to using our old criterion where we exclude participants that get more than attention checks wrong. However, the performance is still higher than that of GPT-4.
G Variation on prompts
Hexagonal and triangular grid In this variation, we gave a more detailed description of the hexagonal and triangular grid. Example of prompts are shown in Figure 20 A similar analysis was conducted on the
same dataset as Figure g Given that the performance of the other models other than GPT-4 are below random guessing, we only include the result from GPT-4 here. The prediction accuracy for triangle is 0.238 (with std 0.016) and for hexagon is 0.088 (with std- = 0.029), both are still within the margin of error the results (for triangle is 0.204 with std 0.015 and for hexagon is 0.046 with std 0.018) from the original prompting.
Question: "In hexagonal tiling (or hexagonal tessellation), equilateral hexagonal tiles are arrunged to 2-dimensional space with no overlaps and no gaps. "Pointy-topped refers to the orientation of the hexagon, indicating that it has pointed acute angle the top. Since equilateral hexagons have equal sides and 120 degree internal angles, each vertex has directions go along the edges. Now, consider the graph given by the vertices of two rows of Pointy-topped hexagonal tiling. The top row contains one hexagonal tile, while the bottom row has two. Starting from a vertex, you will move between vertices along the edges of these tiles. Initially, you are positioned at the bottom left comerr of the map, where you find drilling tig. You move up by one step, where you find carousel. You move up-right by step, where you find loupe. You move up by step, where you find gown. You move up-right by one step, where you find printer. You move down-right by one step, where you find a black-footed ferret. You move down by one step, where you find a station wagon. You move down-left by one step, where you find bath towel. You move up-left by one step. What will you find?" Answer: "loupe."
Question: "In a triangular tillng (or triangular tessellation), equilateral triangulur tiles are arrunged to fill 2-dimensional space with no overlaps and gaps, Since equilateral triangles have equal sides and 60 degree internal angles, each vertex has directions go along the edges: leftright, down leftright, and leftright. Now, consider the graph given by the vertices of two rows of triangulur tiling. The top row contains one triangular tile, the bottom row has three. Hence, there are three rows of vertices: the first row has one vertex, the second has two, and the third has three. Starting from vertex, you will move between vertices along the edges of these tiles. Initially, you are positioned at the bottom left comer of the map, where you find box turtle. You move right one step, where you find a hand plane. You move up-right by one step, where you find a guacamole. You move down-right by one step, where you find a table lamp. You move left by one step. What will you find?"
Answer: "hand plune."
(a) Hexagon
(b) Triangle
Figure 20: Example question and its answer for size-2 triangular grid and size-2 hexagonal grid with more detailed description.
Different tree traversals In this variation, we used breadth-fistt traversal method (instead of depth- first traversal in the main text) to generate the prompt, which gives layer-wise description of the tree structure. An example of the prompt are shown in Figure 21 The prediction accuracy under both breadth-
first traversal and depth-frrs first traversal method is in Table 5 error) across different LLMs.
We can see they are similar (within margin of
CPT-3.5
GPT-4
1lama-2-70b-chan
CodeLlama-34h, I4b-Insttuctt-hf
depth-first traversal (Original)
0.250 (0077)
0.506 (std=0.017)
0.350 (st5=0.025)
0.264 (std=0.029)
breadth-first traversal
0.262 (std-0.033 0.032)
0.356 (std 0.020)
0.316 -0.00)
0.264 (std-0.025)
Table 5: Prediction accuracy under depth-first traversal vs. breadth-first traversal for tree structure. The mean accuracy and standard deviation are calculated over 5 runs on the same dataset.
Prompting CodeLlama to write code
For CodeLlama-34it we performed an additional experiment using Program-of-Thoiie (PoT) prompting, where we prompt it to write python code. An example is shown in Figure 23
20

---

Published in Transactions on Machine Learning Research (02/2024)
"You have been given tree structure with nodes. The root node is great while shark. The great white shark has 2 children: garter snake and a Gila munster. The garter snake has 2 children: jigsaw puzzle und moped. The Gila monster has children: a Christmas stocking and a horse-druwn vehicle. The jigsaw puzzle has child: Tibetan Terrier. The moped has child: umbrella. The Christmas stocking has children. The horse-drawn vehicle has children, The Tibetan Terrier has children. The umbrella has children. What is the cousin of the muped?'
Question:
Answer: "Christmas stocking, horse-drawn vehicle."
(a) Tree (Breadth-Firrt Traversal)
Figure 21: Example question and its answer for the tree structure under breadth-first traversal.
0.8
Promot Direction Itm Direction
0.7
Acourity (0000
0.5
0.5
0.4
0.3
2x3
3x4a4x3
2xtw5y2
4xta
3x3
4x4
8x6
Blucham
Figure 22: the pattern of "Direction Item' 'irecttion' can be observed when we group by the area size, The mean and the standard deviations calculated across 10 different runs.
def solution():
square_grid [[Nne for range(n)] for range(n)]
We use the coordinate systen that is consistent with
OpenCV's conventions.
Initial position is botton right corner of the grid so
position (2, [2 "yellow lady's slipper"
you nove by one step, the new coordinate is (2, 1)
square_prid[2] "swinning cap"
you nove left by one step, so the new coordinate is (2, 8)
square_grid(2) [0] "bell pepper"
you nove by one step, so the new coorindate is (1, 8)
square [aaare_rrid[1[{)) "poke bonnet"
step, the coordinate is (0,
square_grid(0)(0) "leaf beetle"
you right
coordinate is (8,
(11 = "Lion"
you right by step, [quare_grid[0] "Chow Chov"
coordinate (8,
you nove down by one step, square_grid[3] "plate"
new coordinate is (1,
you nove left by one step, so the new coordinate is (1, 1)
[q "tow truck"
you nove down by one step, the new coordinate is (2, 1)
square_grid(22
return answer
Figure 23: An example of code prompting for CodeLlama.
21

---

Published in Transactions on Machine Learning Research (02/2024)
We tested CodeLlama-34b and CodeLlama-133 (which is more tailored for autocompletion) for the square grid of size 3:
CodelJlama-133 13b (PoT)
Codel.lama-34b (PoT)
CodelJama--3h--Immmnee (No PoT)
Accuracy
0.25
0.20
0.25
Table 6: Prediction accuracy of CodeLlama models with code prompting.
As we can see in Table there wasn't any improvement in accuracy. Upon examining the outputs, we noticed that the generated code often suggest incorrect navigation, resulting in wrong answers. This implies that while using code prompts works well for simple arithmetic, it struggles with spatial navigation tasks. Additionally, while this method might be suitable for simple grids like squares, il's not yet clear how to effectively apply it to more complex grid patterns, such as those involving hexagons and triangles.
22


Reviews:
[{"review_id": "9HhOaNLwn6", "reviewer": "Reviewer_w7Kr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper provides a fairly comprehensive comparison amongst the factors it considers. The authors succeed at demonstrating that, for the models they work with, the model is likely to benefit in navigating in a square room and that some ways of representing global representations seem to hurt performance compared to the local representations they consider.\n\nHowever, there are quite a few methodological quirks that make this work hard to replicate and hard to see the support of the work from the evidence that they provide. \n\nFirstly, and most importantly, the authors do not indicate the checkpoint of the models they are using, along with their decoding parameters for their models. For models like GPT4 and GPT3.5, it is best practice to use a model like the 1106 version so that experiments can be replicated. Additionally, it is usually highly relevant to discuss what the temperature is for these evaluative tasks, as otherwise it is difficult to confirm that the results are deterministic and also to aid in replication and further research.\n\nSecondly, it would benefit the authors to be clearer about their design choices, and what certain terms mean. In section 3.1, the authors say to \"keep the number of steps to be 8\", but it is unclear from context what the \"steps\" mean here. Later on, the authors also use the term \"exploration steps\", which I am unable to find a clear definition of. Additionally, it's not clear why the authors did not use zero-shot chain of thought prompting, or any other existing strategy that is not few-shot, and if they had considered other variations of the prompt they propose. Given that even prompt formatting can influence performance, I'd like some clarification on what other designs were considered and why the authors ended up on the set that they did. In general, the paper should spend more time discussing the precise generation approach to allow for replication and wider contribution to the community.\n\nFurthermore, the authors claim that the number of steps was statistically significant, but earlier state that they made the number of steps a constant. I do not understand how the number of steps can causally \"contribute\" to the error when it remains at a constant, and am unsure if the methodology is poorly explained or if the evaluation is poorly explained.\n\nLastly, I generally had troubles understanding why the authors included several models besides GPT-3.5 and GPT-4. Given that there is minimal analysis of any model besides GPT-3.5 and GPT-4, it's confusing as to what the benefit of this analysis really is, and what I should take away from this analysis other than LLaMA generally failing at these tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a wide variety of simple navigation tasks to understand the ability of existing LLMs to move across several structures. They focus on geometrically shaped systems, asking the model to tell what object is at a particular location after making a traversal of the spatial map. They very the topology of the \"rooms\" in the map, the way the \"rooms\" are laid out in the map, the way the \"rooms\" are laid out in the prompt, and consider the inclusion of a global map. Through all of this inference, the authors suggest that current LLMs do better in some kinds of rooms, benefit from the utilization of additional non-navigational information to reason about the path, and that LLMs generally seem to do better with local map representations as opposed to global map representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- \"using the rhombus grid, which was achieved by rotating the square grid 90 degrees\" - did you mean 45 degrees? Rhombuses are normally not axis-aligned, and so I'd imagine this is a typo ( not critical for acceptance, but seems like an easy fix ).\n - The authors must include more detailed information about the nature of their decoding process and what precise model checkpoints they used. Given the sheer number of papers that evaluate an LLM's capability to perform some task, this is critical to ensure that the paper is taken seriously and that others can extend the work accordingly.\n- The authors must also give more information about the nature of what \"steps\" mean at the various parts of their work, and give examples of the \"random and snake\" prompt orderings, even if it's in the appendix. I had issues understanding what precisely was going on in these sections.\n- The authors should explain their particular choice of \"steps\" for their analysis, and why they included a \"steps\" parameter in their equation when they seem to incidate that this is a constant. Given that they focus on this parameter for their subsequent analysis, clarifying this is critical for acceptance as otherwise their claims do not seem to stand up to their analysis.\n- Besides the above critical changes, I'd like some better grounding to strengthen why these tasks would be important for our understanding of how LLM's reason spatially. Some connection to existing work like SayNav would be useful, as it would allow us to better appreciate the synthetic task and the more fine-grained analysis it offers.", "broader_impact_concerns": "I'm not entirely sure if the existing broader impact statement adequately discusses the potential social impact of the authors' work. Given that the authors did perform experiments with a human baseline, I'd like some information about the disclosure process and the steps the authors took to mitigate selection bias. I'm not entirely sure how their existing statement is focused in on the precise work that they have done.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GA3nIT66hj", "reviewer": "Reviewer_zsuW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- A novel task setup that converts the evaluation of spatial understanding into measuring navigation accuracy.\n- An important empirical finding that particular patterns in the training data have made LLMs more familiar with squares than other topologies.\n\n**Weaknesses**\n- The task setup is interesting but may be conflated by other variables. \n\nFor example, models may have difficulties due to either the unnatural language prompt (e.g. Figure 5 prompt for tree) or the lack of scratchpad space to workout their solutions. \nSpecifically, the prompt for tree structure implicitly requires the model to translate a layer-wise traversal result to a tree structure. I'd suggest exploring other traversal ways, or tree representations. The lack of scratchpad space may introduce extra difficulty orthogonal to spatial understanding, as humans would also find it hard to construct a spatial topology and keep track of the object on each node solely by imagination. Besides, CodeLLaMA might be better at writing codes to construct the tree structure so program-of-thought (PoT) prompting is suggested to reveal a more comprehensive picture on this problem.\n- No decoding hyperparameters (e.g. temperature, max_tokens, number of runs) reported. It is a common practice to try different decoding techniques (e.g. top-p, top-k, majority vote).\n- Unclear motivation of the size-inference experiment (Section 3.5). I'm having a hard time understanding how this experiment connects to the previous navigation experiment. Is it a pre-requisite for successful navigation on a maze with a certain shape? What would success or failure on size-inference imply about navigation ability, or the more fundamental spatial understanding ability? What does the finding of better performance with item information than without suggest? Is it possible that item information has actually made size-inference easier by counting unique object names?\n- The human baseline is surprisingly low. I expected humans to find this task trivial but the reported human performance is unexpectedly low (e.g. 0.41 for Hexagon and 0.58 for Triangle). I suspect that human participants did not devote enough effort but I might also missed something. Are there particular areas where humans find this task hard?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to evaluate LLM's understanding of spatial structures, including square, hexagonal, triangular grids, rings and trees. The authors propose a suite of navigation tasks rendered in natural language, which critically require understanding of spatial structures to correctly reason about the underlying maze. Experiments show that only GPT4 and the largest LLaMA models (LLaMA2-70b & CodeLLaMA-34b) achieve non-trivial performance, with success rate biasing towards squares compared to other topologies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Crucial**\n- Report decoding hyperparameters. It is also highly recommended to try different decoding techniques or aggregating results from multiple decoding runs to report consistency.\n- Adjust the prompt to see if models were performing badly due to the difficulty in understanding the task vs. performing the task.\n- Allow the models to use scratchpads and allow the code model to use programs to aid their work.\n- Clarify motivation for the size-inference experiment\n- Re-evaluate human performance or explain the human errors when their performance is far from perfect (e.g. 0.41 for Hexagon and 0.58 for Triangle).", "broader_impact_concerns": "No ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QmIvXecHrP", "reviewer": "Reviewer_VpRX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The analysis is very comprehensive, considering many factors such as model size, presentation format, and topology.\n- The error analysis provides interesting insight into the models' underlying representations.\n- A comparison with human behavior is performed, revealing a similar trend of difficulty with respect to topology (though overall better performance in humans vs. LLMs).\n\n## Weaknesses:\n- I found the description of the non-rectangular grids to be somewhat confusing (e.g. I had to think quite a bit to visualize a 'pointy-topped regular hexagonal tile map'). I think this may explain the relative difficulty of the other types of grids. This is also consistent with the fact that human participants had higher error rates on these grids. It may be worth mentioning that errors on these grids may be related to the inherent difficulty of describing them with natural language (as opposed to visually).\n- Related to the previous point, there is a concern that the non-rectangular attention checks are not easy enough to be considered attention checks. How do the human results look when participants are only excluded for getting the rectangular attention check wrong?\n- Even though the human performance is better than GPT-4, there is a very strong correspondence in the relative performance across different topologies. This strongly suggests that some common factor may explain the relative difficulty of these different topologies in both human participants and LLMs. This should be emphasized in the text.\n- Can a more detailed description of the human participants and study procedure be provided? e.g., how were they recruited? What was the age range of the participants? how was the study performed (online? in-person?)? \n- Some measure of error (e.g. standard error) should be included for all results, including tables and figures (as error bars).\n- How were the models evaluated (e.g. through the OpenAI API, or the ChatGPT web interface?)? It would be helpful to provide more details about the model settings (e.g. temperature, top p), if this information is available.\n\n## Minor comments:\n- The citations should use \\citep unless the name is explicitly mentioned in the sentence.\n- In the introduction, the statement about humans using spatial representations for abstract tasks should cite the following paper:\nConstantinescu, A. O., O’Reilly, J. X., & Behrens, T. E. (2016). Organizing conceptual knowledge in humans with a gridlike code. Science, 352(6292), 1464-1468.\n- In figure 3, colors are used to indicate model, but in figure 4 colors are used to indicate the task. I would suggest to use a consistent scheme throughout the entire paper (e.g., colors to represent model for all figures). Similarly, I would suggest to keep the models in a consistent order.\n- In describing the relationship between the present work and Momennejad et al. (2023), it should be more clearly noted that these two works primarily investigate different (but related) tasks, e.g. path integration for sensory prediction in the present work, vs. planning (generating a path to move from point A to point B) in Momennejad et al. (2023).\n- In the example given for the chain-of-thought prompt, the answer is stated before the chain-of-thought. Just to clarify, is the model then allowed to generate the chain-of-thought before producing an answer?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates the spatial reasoning abilities of LLMs, exploring a number of factors, including topology (ring, grid, hexagon, etc), presentation format, and model size. The results indicate some degree of spatial reasoning, particularly in GPT-4, with stronger performance for rectangular grids vs. other topologies. A direct comparison is also performed with human participants.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have indicated some requested changes in the previous section.", "broader_impact_concerns": "There are no discernible ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are happy with the experiments produced and the responses/changes (including to the rhetoric of claims).
Decision: Accept (poster)