Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
Authors: Wenshan Wu
Shaoguang Mao
Yadong Zhang
Yan Xia
Li Dong
Lei Cui
Furu Wei
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Spatial Reasoning
LLM Reasoning
Primary area: natural_language_processing
TL;DR: Visualization-of-Thought (VoT) prompting improves spatial reasoning in LLMs by visualizing their reasoning traces, mimicing the human mind's eye process to imagine the unseen objects and actions.

Abstract: Large language models (LLMs) have exhibited impressive performance in language comprehension and various reasoning tasks. However, their abilities in spatial reasoning, a crucial aspect of human cognition, remain relatively unexplored. Human possess a remarkable ability to create mental images of unseen objects and actions through a process known as the Mind's Eye, enabling the imagination of the unseen world. Inspired by this cognitive capacity, we propose Visualization-of-Thought (VoT) prompting. VoT aims to elicit spatial reasoning of LLMs by visualizing their reasoning traces, thereby guiding subsequent reasoning steps. We employed VoT for multi-hop spatial reasoning tasks, including natural language navigation, visual navigation, and visual tiling in 2D grid worlds. Experimental results demonstrated that VoT significantly enhances the spatial reasoning abilities of LLMs. Notably, VoT outperformed existing multimodal large language models (MLLMs) in these tasks. While VoT works surprisingly well on LLMs, the ability to generate mental images to facilitate spatial reasoning resembles the mind's eye process, suggesting its potential viability in MLLMs. Please find the dataset and codes in our [project page](https://microsoft.github.io/visualization-of-thought).

Mind's Eye of LLMs: Visualization-of-Thouhh Elicits Spatial Reasoning in Large Language Models
Wenshan Wu' Shaoguang Mao*
Yadong Zhang
Yan Xia Li Dong" Lei Cui Furu Wei
Miicosooft Research *East China Normal University
Abstract
Large language models (LLMs) have exhibited impressive performance in language comprehension and various reasoning tasks. However, their abilities in spatial reasoning, crucial aspect of human cognition, remain relatively unexplored. Human possess a remarkable ability to create mental images of unseen objects and actions through process known the Mind's 's Eye, enabling the imagination of the unseen world. Inspired by this cognitive capacity, we propose Visualization- of-Thought (VoT) prompting. VoT aims to elicit spatial reasoning of LLMs by visualizing their reasoning traces, thereby guiding subsequent reasoning steps. We employed VoT for multi-hop spatial reasoning tasks, including natural language navigation, visual navigation, and visual tiling in 2D grid worlds. Experimental results demonstrated that VoT significantly enhances the spatial reasoning abilities of LLMs. Notably, VoT outperformed existing multimodal large language models (MLLMs) in these tasks. While VoT works surprisingly well on LLMs, the ability to generate mental images to facilitate spatial reasoning resembles the mind's 's eye process, suggesting its potential viability in MLLMs. Please find the dataset and codes in our project page.
Ueeind you is the moreet
Verbal
Frmm agin to the oereley.
Mestal Imagrs
lutt: tmsight the office
Images
Text
Mind's Eye
Mind's
Visual
Mind's Eye of Humans
Mind's Eye LLMs
Conventional Prompting
Input
Ontput
Chain-of-Thought
Input
Thought
Thought
Output
Visualizatton-ole watpiiittttooooo--Toooo Thought
Input
Thought
Thought
Output
Visualization
Visualization
Figure Humans can enhance their spatial awareness and inform decisions by creating mental images during the spatial reasoning process. Similarly, large language models (LLMs) can create internal mental images. We propose the VoT prompting elicit the "mind's eye" of LLMs for spatial reasoning by visualizing their thoughts at each intermediate step.
"Contribution during internship at Microsoft Research.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Recently, large language models (LLMs) [BCE-23, BMR+ TLI+ JSM+23] have achieved remarkable performance on various language-related tasks. However, despite their success in math reasoning [KGR+23] 233], common sense reasoning [LK and other reasoning tasks such as symbolic reasoning logic reasoning [KGR 23], their abilities in spatial reasoning still remain underexplored 2RID+21, YBL 233, M4].
Spatial reasoning is an essential function of human cognition, allowing us to interact with the envi- ronment. It facilitates tasks that require understanding and reasoning about the spatial relationships between objects and their motions. The spatial reasoning of language models largely relies on language to reason about spatial information, whereas human cognitive capabilities extend far beyond verbal reasoning. Humans can not only create task-relevant abstract representations from visual perception [BK18, KC22], but also imagine unseen scenes through their mind's eye. It remains a research topic called mental image [She78] in domains of neuroscience, philosophy of mind, and cognitive science. Building upon this cognitive function, humans facilitate spatial reasoning by mental image manipulation, such as navigation [TTo448] mental rotation [SM71], mental paper folding [SF72], and mental simulation [MK09]. Figure illustrates the human process involved in navigation task. Humans enhance their spatial awareness and inform their decisions by creating mental images of route, utilizing various sensory inputs such as navigation instructions or a map image. Subsequently, they simulate route planning through the mind's eye.
Inspired by this cognitive mechanism, we conjecture that LLMs possess the ability to create and manipulate mental images in the mind's eye for spatial reasoning. As illustrated in Figure 1, LLMs could potentially process and understand spatial information in various formats. They might be capable of visualizing internal states and manipulating these mental images through their mind's eye, thereby guiding subsequent reasoning steps to enhance spatial reasoning. Therefore, we propose the Visualization-of- Thouuhh (VoT) prompting to elicit this ability. This method leverage LLMs to visualize their reasoning steps and inform subsequent steps, implementing the concept of visuospatial sketchpad [Bad92]. VoT adopts zero-shot prompting instead of relying on few-shot demonstrations or ext-to--mage visualization with CLIP [RKH 22]. This choice stems from LLMs' ability to acquire various mental images from text-based visual art [SB 14, SMM21, Reg19].
To evaluate the effectiveness of VoT in spatial reasoning, we selected three tasks that require spatial awareness LLMs, including natural- natural-language navigation [YBL*23], visual navigation, and visual tiling. These tasks require understanding of space, direction, and geometric shape reasoning. To emulate human-like multisensory perception, we designed 2D grid worlds using special characters enriched input formats for the LLMs in visual navigation and visual tiling tasks. We compared different models (GPT-4, GPT-4V) and prompting techniques across these three tasks. The findings reveal that the VoT prompting proposed in this paper consistently induces LLMs to visualize their reasoning steps and inform subsequent steps. Consequently, this approach achieved significant performance improvements the corresponding tasks.
The main contributions of this paper include:
1. We shed light on LLMs' mental image for spatial reasoning from a cognitive perspective, conducting quantitative and qualitative analyses the mind's eye of LLMs and its limitations. We also explore cues about the origin of this generalized ability from code pre-training.
2. We develop two tasks of "visual navigation" " and "visual tiling". along with corresponding synthetic datasets, emulating various sensory inputs for LLMs. These tasks are structured to support varying levels of difficulty, offering a well-designed testbed for the research on spatial reasoning.
3. We propose Visualization-of oofllizttionno-Thouuhh (VoT) prompting to elicit the mind's eye of LLMs for spatial reasoning and provide empirical evaluations on three tasks. Experiment results prove the effectiveness of VoT prompting compared with other prompting methods and existing MLLMs. This ability to generate mental images to facilitate spatial reasoning resembles the mind's eye process, suggesting its potential viability in MLLMs.
2

---

2
Spatial Reasoning
Spatial reasoning refers to the ability to comprehend and reason about the spatial relationships among objects, their movements, and interactions with the environment. This skill is vital for wide range of real-world applications such as navigation, robotics, and autonomous driving. These fields necessitate action planning based on visual perception and concrete understanding of spatial dimensions.
Although several tasks and datasets [WBC+ 15, SZL22, MK22, LB 18, RAB+20] have been devel- oped to probe the spatial semantics embedded in text, existing research efforts often focus on how spatial terms are linguistically structured. Recently, significant achievements and impressive results have been achieved in these benchmarks by converting spatial terms to logical forms through LLMs and adopting logic programming [YIL23]. This implies that excelling in these tasks does not nec- essarily equate to a genuine understanding of spatial information by LLMs, nor does it provide an accurate measure of their spatial awareness.
Spatial awareness involves understanding spatial relationships, directions, distances, and geometric shapes, all of which are essential for action planning in the physical world. To evaluate the spatial awareness and spatial reasoning abilities of LLMs, we have selected tasks that test navigation and geometric reasoning skills, including natural language navigation, visual navigation and visual tiling.
2.1 Natural Language Navigation
Natural language navigation task [YBL+ was inspired by prior research on human cogni- tion [GDB 17] presenting participants with sequential transitions sampled from a graph structure.
In this context, a square map is defined by a sequence of random walk instructions and associated objects at each location, denoted W = {(41, 01), (22,02), o2)..... (I,,,,)) ). Given square map W, and sequence of navigation instructions = {ii..... ............ ) the task for the model is to identify the
associated object E W at the specified location which is determined by the navigation instructions, as detailed in Equation and exemplified in Appendix B.2.
o~ p(o E WIW = (1),,)), 02). (In, on)).
2.2 Visual Navigation
Visual navigation task presents synthetic 2D grid world to LLM, challenging it to navigate using visual cues. The model must generate navigation instructions to move in four directions (left, right, up, down) to reach the destination from the starting point while avoiding obstacles. This involves two sub-tasks: route planning and next step prediction, requiring multi-hop spatial reasoning, while the former is more complex. Task instructions are available in Figure 6 in appendix.
Formulation The model is presented with a grid map M consisting of l consecutive edges E = {e(so, $1),e(s;, 82),... , e((6-1, 8k) }, where the starting point and destination are so and 8 respectively, as shown Figure Route planning task is to generate a sequence of correct directions D {d( so, $1), d(s1, (2)) s2).... d(sx-1, sk)}, as defined in Equation 2. Given M and t navigation instructions Dr = {d(ss. $1), d(s:-1, 8t)). next step prediction task is to identify the correct direction ((s1,8541) of the next step, as defined in Equation 3.
D ~ p((d(50, 81), d(si, 82), ... d(sk-1,, bk)} M)
PzZ 'zzž izzi
2nd Pett
inzk iznú
'22
222) 222
Geeh 2222
122
'22
(1) 1p2
2ez teet
(a) k=2
(b) k=3
(c) k=4
(d) k=5
(c) k=
(f) k=7
Figure 2: Examples of navigation map under different settings of kt with emoji of house indicating the starting point, and emoji of office indicating the destination.
3

---

d ~ p(d(s), M, D(,octck)
Implementation The navigation map's underlying graph is semi-Eulerian, alternating between horizontal and vertical edges, with possible spatial configurations for a k-hop navigation map. For each map and set of k navigation instructions, question-and-ansee (QA) instances,i.e. "what is the next step?" are created. Further implementation details are in Appendix A.I.
2.3 Visual Tiling
Introduced by [Gol66], polyomino tiling is classic spatial reasoning challenge. We extend this concept to test the LLM's ability to comprehend, organize, and reason with shapes in confined area, thus enhancing the evaluation of spatial reasoning skills. As depicted in Figure 3, the task involves rectangle with unfilled cells and various polyomino pieces, like the I-tetromino made of four aligned squares. The model must select the appropriate polyomino variant, such as choosing the orientation for the I-tetromino, to solve the QA puzzle. Task instructions are provided in Figure 7 in appendix.
Formulation The model is presented with rectangle R masked with k unique polyominoes MP = {mpi,... ... ,mpk) 2 corresponding variants of each polyomino Vi<= = {v11, 1,2), ,. and a polyomino query q MP. Visual tiling task is to identify the correct variant of g, defined in Equation 4.
~~ p(y R, {mpi,".. mpp)), {ui1, "12 Uki, Uk2).9)
Implementation
The dataset comprises valid spatial arrangements generated through existing algorithms(ES03, GN07], with random masking of polyominoes create QA puzzles. Details are provided in Appendix A.2.
(a) Fit 2 pieces into masked rectangle
(b) pieces into masked rectangle
Figure 3: Example of visual tiling with masked polyomino pieces. Variants of those polyomino pieces including rotation and reflection are not shown this figure.
Visualzationnoo-Thouuh Prompting
Considering the way humans process spatial information during tasks like navigation, it's common to create mental images, such as maps, to enhance spatial awareness or simulating movements to inform decision-making Our objective is elicit the spatial awareness of LLMs and ground their reasoning by visualizing the consequence of their intermediate reasoning steps.
We introduce Visualization-of- -ottoooh (VoT) prompting: "Visualize the state after each rea- soning step." " This new paradigm for spatial reasoning aims to generate reasoning traces and visualizations in interleaved manner. Qualitative results of this approach presented in Figure 4.
We use pa denote pre-trained LM with parameters 0, I, y, denote language sequence, and to denote visualization sequence in text form. In a multi-hop spatial reasoning task with input æ, CoT prompting generates a series of intermediate steps 21, Zn. each step 2; ~ po xr, 21...i-1) is sampled sequentially, followed by the output prompting enhances this process by adding visuospatial sketchpad each intermediate step 2,, then the subsequent step 2i+1 is sampled conditioned on prior steps Zi... and visualizations Vi....
Po (3) 21..).. As shown in Figure 1, VoT
As defined in the Equation 5 and 6, it forms interleaved reasoning traces and visualizations. A qualitative comparison between outputs of VoT and CoT is provided in Figure 8a in appendix.
U/ pp(u: prompty oT, 21.rd U1.---1)
(5)

---

Visual Navigation
Visual Tiling
Natural Language Navigation
You have given by square grid. Iniiially you are at the bottom-lelt corner... find cassette player...go right.. woel, go right. up... moving van, go left... confectionery store, siammnng, right... left.. colohuss, go right.... mini Now you have all the information on the map. You where the cassette player located, you go right by une step, yo rightt.... gu up,
pad par pat pat
go
pog pMd
peg pad par pnd
Provided::
fital the provided polyominoes into the empty squars, what's the correct vadiation of
Starting from @ provide the steps navigate to
left... go left. up..go right...go down step. hat will you find?
Visualitze state hher each reasoning step.
Visuelize the state after reasening step.
Visuelize state ofter each reasering step.
dil
Amstyne
are poy
Moxe
Move dowen
Mirre left
Visualize
Move
right
hove right
di
C
(1)
Movedoom
Move left
Move down
1. J.Pocel
Mane
Placn
7. Move
8. Nove right 9. Hove down
Figure 4: Examples of VoT prompting in three tasks, where LLM generates 2D grids as text-form mental images. The generated reasoning traces and visualizations form an interleaved sequence to track the state over time. The 2D grids in the input and responses are composed of special characters. Full responses could be found in Appendix B.
2++1 ~ (o promptyor: I, 21.-1, U1...i)
This reasoning paradigm enables LLMs with visual state tracking. We introduce the concept of state, denoted as 8, = [zt (1--i-1 representing a partial solution at step i with the input, the sequence of intermediate steps 21... and the sequence of visualizations U1...i-1.
U/ (o prompty It, 21...; U1...i-)
(o promptyer,
As shown in Equation 7, visual state tracking is implemented by generating the visualization ;; as representation of the internal state 8; after each reasoning step 2; (e.g. i2 could be a grid of the navigation map marked with path or filled rectangle.. Grounded by the visual state tracking sequence, the subsequent state is derived by 8:+1 ~ Pg (8++1 promppyoT x, 8,, v.). This mechanism allows for the derivation of subsequent states, reflecting spatiotemporal causality and enhancing the spatial reasoning capabilities of LLMs in grounded context.
4 Experiment
4.1
Setup
For the visual tasks where counterpart image exists for each text input, conduct additional exper- iments with a multimodal model. Specifically, we adopt GPT-4 [OA+23] and GPT-4 Vision [Ope23] via Azure OpenAl API they're state of the LLM and multimodal model respectively. API settings are lemperature 0 as greedy decoding and top 1, with model versions of 106-preview and vision-preview. For all experiments we adopt zero-shot prompting.
Depending on whether the LLM is explicitly prompted to visualize intermediate steps, we experiment with three settings of GPT-4, including shoo CoT prompting(GPT-4 CoT)) GPT-4 w/o Viz where visualization is explicitly disabled during reasoning, and VoT prompting (GPT-4 VoT). Additional setting of GPT-4 Vision with counterpart image input is GPT-4V COT. Prompts are following:
5

---

GPT-4 CoT: Let's think step by step.
GPT-4 w/o Viz: Don't I use visualization. Let's think step by step.
GPT-4V CoT: Let's think step by step.
GPT-4 VoT: Visualize the state after each reasoning step.
Task instructions and examples could be found in Appendix B.
4.2 Dataset
Natural Language Navigation We generate 200 square maps of size 3x3 which is described by landmarks in snake order traversal, and set of navigation instructions.
Visual Navigation We generate 496 navigation maps and 2520 QA instances in total, covering various map sizes, up to 7x9 and 9x7. The data distribution is provided in Table 4 in appendix.
Visual Tiling We first generate multiple unique configurations to fill 5 x 4 rectangle with polyomino pieces including two I tetrominoes, two T tetrominoes and one L tetromino. Then we randomly masked two three pieces of different types and generate QA instance for each masked pieces. The total number of QA instances is 796, and we show dataset details in Table in appendix.
4.3 Metric
We extract the answer from model output by pattern matching. For tasks except for route planning, we calculate accuracy by Equation 8. We adopted sub-string matching" as Seorrect to determine correctness.
acc Σ
ground_truth))
(8)
For the route planning task which predicts sequence of navigation instructions, we reject any sequences exceeding 100 instructions, considering them be random guesses. We then normalize the navigation instructions by executing each navigation instruction. Those instructions which violate navigation rules will be ignored. The length of normalized instruction sequence is considered the temporal distance against the starting point. Given the ground- truth of k navigation instructions, the completing rate route planning is l/ki For the dataset of n maps, we report two metrics including:
1. Average completing rate: Σ. li/ki/n. Average completing rate among all instruction sequences, reflecting LLM's effectiveness of route planning.
2. Success rate: Σ, (t, == kn) This metric represents the proportion of instruction se- quences with t = k. i.e., reaching the destination.
4.4 Results
As illustrated in Table GPT-4 VoT significantly outperforms other settings in all tasks across all metrics. The significant gap when comparing GPT-4 VoT with GPT-4V CoT and GPT-4 w/o Viz demonstrates that effectiveness of visual state tracking, which allows LLMs visually interpret their actions within an grounded world. And in the natural language navigation task, GPT-4 VoT outperforms GPT-4 w/o Viz by 23.5%. In the visual tasks, the noticeable performance gap between GPT-4 CoT and GPT-4V CoT indicates that LLM grounded with 2D grid could possibly outperform MLLM challenging spatial reasoning tasks.
On the other hand, performance of GPT-4 VoT is still far from perfect in all tasks, especially in the most challenging route planning task. Despite these tasks are relatively easy for humans, performance of LLMs drops significantly as task difficulty increases. Details on performance trends across difficulty levels are provided in figure and table in appendix.
We use this term for simplicity. natural language navigation tasks, LLMs often output additional words in the extracted answer besides the expected object name. For example, "Answer: You will find ...". In this case, sub-string matching is adopted without affecting the correctness. Otherwise, exact matching is adopted for multiple choice questions in visual tasks.

---

Visual Navigation
Natural-Langunt Navigation
Settings
Visual Tiling
Route Planning
Next Step Prediction
Completing Rate
Succ Rate
GPT-4 CoT
37.02
9.48
48.61
54.15
54.00
GPT-4 w/o Viz
37.17
10.28
48.49
46.98
35.50
GPT-4V CoT
33,36
5.65
46.59
49.62
GPT-4 VoT
40.77
14.72
55.28
63.94
59.00
Table I: Performance of different GPT-4/4V settings in all tasks. Underline denotes statistical significance with p 0.05 when comparing GPT-4 VoT against all baselines using two-sample z-test, while < 0.16 is observed compared with GPT-4 CoT in natural language navigation task.
5 Analysis
As explained in section 3, one of the core aspects of VoT lies in enabling LLMs with visual state tracking. During the experiments, it was observed that GPT-4 CoT occasionally exhibited this reason- ing pattern across several tasks with exception route planning. Besides, incorrect visualizations of VoT commonly observed in model outputs. In this section, analysis of VoT primarily focuses on three questions: (1) Do visual state tracking behaviors differ among prompting methods? (2) How visualizations enhance final answers? (3) Can VoT benefit less powerful language models?
5.1 Do visual state tracking behaviors differ among prompting methods?
For each model output, we extract the sequence of visualizations sampled prior to generating the final answer and discard any visualizations generated thereafter. Then we compare the sequence length 4 with the number of reasoning steps 1,. We calculate Complete Tracking Σ, (4, == 1,)/n when visualization ey corresponds each state s,. Similarly, we calculate the Partial Tracking metric as Σx (Lo 0)/n when at least one visualization is present before the final answer is generated. Figure 5 shows the significant differences between these settings. the GPT-4 CoT setting, demonstrated noticeable tracking rate across almost all tasks except route planning. This observation implies that LLMs inherently exhibit the capability of visual state tracking when spatiotemporal simulation is integral to reasoning.
On the other hand, the visual state tracking behavior is sensitive to prompts varying degrees. As showcased in Figure 8 in appendix, after removing "reasoning" from the prompt of VoT. the visualiza- tions are sampled after GPT-4 generates the wrong answer. Consequently, explicitly prompting LLMs to visualize their reasoning traces with VoT markedly improves the visual tracking rate, thereby enhancing overall performance. The potential contribution of code pre-training this emergent capability is further explored in Appendix C.
D carm ver Tnprece car cor4 wis
D caree con E
100 20.2
100
60.2
MUS
57.4
10.2
18.1
1.2
0.S.
0
Route Next Step Visual PlaaningPredictton
Natural language
Route
Next Step Visual Natural langunge
Tiling
PlanningPrediction Tiling
Navigation
Navigation
(a) Complete tracking rate
(b) Partial tracking rate
Figure 5: tracking rate of different settings across all tasksk.
5.2 How visualizations enhance final answers?
Ideally, VoT is supposed to generate an accurate visualization vy at each step, so that subsequent step Zi+1 could be determined correctly. This relies the spatial visualization and spatial understanding capability of I.LMs. To evaluate these capabilities of LLMS in these tasks, we extract the final visualization from each model output under the setting GPTT4 VoT in visual navigation and polyomino

---

tiling task. Specifically, for visual navigation task, we extract the visualized map where LLM completed all navigation instructions. For polyomino tiling, we extract the rectangle filled with corresponding polyomino piece. The spatial visualization capability is measured by two criteria: (1) Compliance, indicating whether the manipulation of mental image satisfies requirements such as avoiding overlap and navigating around obstacles. (2) Accuracy, indicating whether the mental image aligns with the corresponding state. The spatial understanding capability is measured by the proportion of correct answers when the corresponding visualization is generated accurately.
As could be seen from Table 2, LLMs demonstrate promising potential in performing multi-hop visualization while adhering to spatial constraints, with compliance rates of approximately 51-52%. However, the relatively low accuracy of state visualization (around 246%26) indicates a need for significant improvements in this area. Despite this limitation, LLMs are able to make correct decisions in 65%-77% of the cases when accurate internal state visualizations are generated, which enhances groundedness and contributes to notable performance gains. Several case studies are provided in Appendix for interested readers.
Spatial Visualization
Spatial Understanding
Task
Compliance
Accuracy
Accuracy
Visual Navigation
51.14
26.48
65.16
Visual Tilling
52.01
24.25
77.20
Table 2: Spatial visualization/uuderrttanntt evaluation in visual navigation and visual tiling task.
On the other hand, VoT prompting might underperform in those tasks where LLMs can leverage logical reasoning without visualizing internal states. We conducted experiments in natural language navigation within a ring [YBI.+23) where navigation instructions are either clockwise or counter- clockwise movements. By normalizing each instruction to a signed number, GPT- converts this task to mathematical calculation of adding and modulus operation. For example, instructions of 15 steps clockwise and 3 steps counter-clockwin are normalized to (15 3) % 12. Results show that GPT-4 CoT outperforms GPT-4 VoT with 52.5% VS 49.5% among 200 test instances with ring size of 12.
5.3 Can VoT benefit less powerful language models?
To evaluate the efficacy of VoT on less powerful language models, we conducted experiments across various model families [BMR+20 OA- -23, TLI+ and model sizes, including GPT-3.5 turbo, ALAMA3-8B-Instruce and ALAMAA3-0BB-nnttuee
We access GPT-3.5 via Azure OpenAl API with model version 1 1106 preview and apply greedy decoding to all models.
As shown in Table 3, within the same model family, performance improves across all tasks with increases in model size. LLAMA3-70B VoT significantly outperforms the baseline across all tasks except for visual tiling, where it aligns closely with results observed in GPT-4. This consistency suggests that VoT offers a scaling advantage when applied to more advanced models, markedly enhancing performance in larger models. In contrast, less capable models tend to rely on random guessing, especially in spatial reasoning tasks. For instance, in the route planning task. GPT-3.5 CoT often resorts to speculative responses, random guessing in nearly half of the instances, which leads to exhaustion of output tokens. While GPT-3.5 VoT effectively minimizes random guesses, such occurrences become increasingly rare with GPT-4 CoT as the model size expands. On the other hand, the reliance on random guessing introduces unpredictability in performance trends for less powerful models. It suggests their limitations in sustaining reliable reasoning processes across different difficulty levels. Details on performance trends are provided in Appendix D.
Related Works
Spatial Reasoning over Text Spatial reasoning and spatial language understanding [KPM20] in NLP domain mainly focus on semantic representation [CBGG97, Batlo, HK11], spatial information extraction [RMK18, KVOM1 1], learning and reasoning [KM15, SLYA17, KFP19]. Recent advance- ments have further explored spatial reasoning within the context of large language models (LLMss). To improve multi-hop spatial reasoning skills of language models, several works [MFNK21, MK22] proposed to pretrain language models with synthetic datasets. An increasing number of dataset were then developed to covers various type of spatial relations in 2D visual scenes [WBC+ 15, SZL22],
8

---

Visual Navigation
Natural- Languuge Navigation
Settings
Visual Tiling
Route Planning
Next Step Prediction
Completing Rate
Succ Rate
GPT-3.5 CoT
16.10
2.62
17.42
44.10
8.50
GPT-3.5 VoT
19.02
1.61
13.10
47.99
9.00
LLAMA3-8B CoT
4.65
0
28.73
47.24
16.50
I.AAMA3-B VoT
4.97
0.2
26.75
46.73
15.50
LLAMA3-70B CoT
19.90
2.62
49.01
56.41
26.00
LLAMA3-70B VoT
30.24
5.85
54.09
56.03
32.50
Table 3: Performance of VoT in GPT-3.5 and LLAMA3 models. Underline denotes statistical significance with p 0.05 compared corresponding CoT baseline using two-sample z-test.
geometric patterns [9] and 3D spatial information [AMKK21, HZC+ [FML 22] investi- gated spatial reasoning capabilities of transformer-basee models the UI grounding setting. On the other hand, some works adopted in-context learning, leveraging LLMs for general purpose reasoning to convert spatial information to logic forms [YIL23], or general pattern machine for sequence transformation [MXF+23]. Recently, several works focused on evaluating spatial reasoning of LLMs as cognitive capability on navigation [Y] and planning tasks [MHV+24) among various spatial structures. While most existing works rely on linguistic semantics and verbal reasoning, and might not always necessitate spatial awareness, we propose to elicit mind's eye of LLMs in spatial reasoning tasks with various formats from cognitive perspective. The VoT prompting induces LLMs to create mental images for visualizing their internal states and inform subsequent reasoning step.
World Models of LLMs While there have been many theoretical debates about whether LLMs can effectively learn an internal world model from ungrounded form alone [BHT+20 20, MGSS21], [LeC22] advocated that world models should represent percepts and action plans at multiple levels of abstraction and multiple time scales, with the capability planning, predicting, and reasoning. [LWG+22] proposed to ground LLM in the physical world by reasoning over the experimental results predicted by external simulation. [HGM 23] further leveraged LLMs as world models to predict the subsequent states by action simulation, given predefined states and actions per task. On the other hand, an increasing number of studies focus on investigating internal representations of LLMs. [PP22, AKH*21 showed that by utilizing in-context learning, LLMs' learned representations can be mapped to grounded perceptual and conceptual structure in color and spatial domains. Moreover, [GT23] and [NLW23] discovered linear representations of space, time and game state in specifically trained LLMs, which are important for dynamic causal world models. Our work does not probe the internal representations of specialized LLMs, does it depend on external simulation engine or state definitions. We demonstrate LLMs' zero-shot capability of representing their precepts at an abstract level, predicting and tracking the internal states time generate action plans in multi-hop spatial reasoning tasks, which possibly mirrors the causal world model within LLMs.
7 Conclusion
This study introduces Visualization-of of ohoThhthh Prompting (VoT), inspired by the human cognitive function of visualizing and manipulating mental images through the mind's eye. We have demon- strated that VoT enables LLMS exhibit the mechanism of "the mind's eye", as evidenced by their performance in multi-hop spatial reasoning tasks and our comprehensive analysis of the reasoning traces. Remarkably, VoT enable LLMs to outperform state-of- the-art multimodal large language models (MLLMs) in the tested visual tasks. While VoT demonstrates impressive efficacy in LLMs, this emergent capability to create mental images to enhance spatial reasoning resembles the mind's eye process, suggesting its promise in MLLMs.
Building on the success of experiments with GPT-4, we plan investigate how VoT can futher elicit "the mind's eye" in MLLMs to enhance their spatial awareness. Additionally, our future efforts will focus on automatic data augmentation from real-world scenarios, aiming to identify effective methods for learning generalized internal representations of menial images. This will further improve the mind's eye of L.L.s, ultimately contributing to the advancement of their cognitive and reasoning abilities.

---

Limitations
This work only scratches the surface of spatial reasoning of LLMs. Both mental images and visual state tracking rely the emergent ability of advanced LLMs. Therefore, it might cause performance deterioration less advanced language models or more challenging tasks. Besides, due to the limited data exposure and a lack of explicit instruction tuning, visual state tracking of current LLMs are sensitive to prompts. For example, when explicitly prompted with "use ascii-art" art", the tracking rate will significantly increase thereby boosting performance, while removing "reasoning" from the VoT prompt will cause decrease tracking rate. Moreover, the mental images tested in our work are limited to 2D grid. To strength the mind's ' eye of LLMs, more diverse and complicated representation should be explored the future, such as complex geometric shapes and even 3D semantics shown in Figure 11 in appendix.
References
[AKH*21] Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? case study in color. arXiv preprint arXiv:2109.0m 0900 2021.
[AMKK21] Daich Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19107-19117,200
[Bad92] Alan Baddeley. Working memory. Science, 255(5044):556-559, 1992.
[Bat10] John A Bateman. Language and space: a two-level semantic approach based on princi- ples of ontological engineering. International Journal of Speech Technology, 13:29 48 2010.
[BCE+23] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, J. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Y. Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, H. Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv.org, 2023.
[BHT+20] Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language. arXiv preprint urXiv:2004.10155 2020.
[K18] Nicholas Baker and Philip J Kellman. Abstract shape representation in human visual perception. Journal Experimental Psychology: General, 147(9): 2018.
[BR 20] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. Henighan. Rewon Child, A. Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S. Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, I. Sutskever, and Dario Amodei. Language models few-shot learners. Neural Information Processing Systems, 2020.
[CBGG97] Anthony G Cohn, Brandon Bennett, John Gooday, and Nicholas M Gotts. Representing and reasoning with qualitative spatial relations about regions. In Spatial and temporal reasoning, pages 97-134. Springer, 1997.
[Cho19] François Chollet. On the measure of intelligence, 2019.
[ES03] Niklas Eén and Niklas Sörensson. An extensible sat-solver. In International conference on theory and applications of satisfiability testing, pages 502-518. Springer, 2003.
[FMI++22] R. Freedman, Joseph B. Mueller, Jack Ladwig, Steven Johnston, David McDonald, Wauck, Ruta Wheelock, and Hayley Borek. symbolic representation of human posture for interpretable learning and reasoning. arXiv.org, 2022.
10

---

[GDB17] Mona M Garvert Raymond J Dolan, and Timothy EJ Behrens. A map of abstract relational knowledge in the human hippocampal-entorie entorriia cortex. elife, 6:e17086. 2017.
[GNO7] Eugene Goldberg and Yakov Novikov. Berkmin: A fast and robust sat-solver. Discrete Applied Mathematics, 155(12):1549-1500 2007.
[Gol66] Solomon W Golomb, Tiling with polyominoes. Journal of Combinatorial Theory, 1(2):280-296, 1966.
[GT23] Wes Gurnee and Max Tegmark. Language models represent space and time, 2023.
[HGM+23] 23] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model, 2023.
[HKI1] Joana Hois and Oliver Kutz. Towards linguistically spatial logies. In Dagstuhl Seminar Proceedings. Schloss Degstuhl-Leibmiz-Zznntte fA1/4r 4r Informatik, 2011.
[HZC+23] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models, 2023.
[33 Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[KC22] Yuna Kwak and Clayton E Curtis. Unveiling the abstract format of mnemonic represen- tations. Neuron, 112111)1822--188, 2022.
[KFP19] Nikhil Krishnaswamy, Scott Friedman, and James Pustejovsky. Combining deep learning and qualitative spatial reasoning to learn complex structures from sparse examples with noise. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2911-2918, 2019.
[KGR+23] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023.
[KM15] Parisa Kordjamshidi and Marie-Francine Moens. Global machine learning for spatial ontology population. Journal of Web Semantics, 30:3-21, 2015.
[Knu00] Donald E Knuth. Dancing links. arXiv preprint cs/00/ 1047, 2000.
[KPM20] Parisa Kordjamshidi, J. Pustejovsky, and Marie-Francine Moens. Representation, learn- ing and reasoning on spatial language for downstream nlp tasks. Conference on Empirical Methods in Natural Language Processing, 2020.
[KVOM11] Parisa Kordjamshidi, Martijn Van Otterlo, and Marie-Frand Francine Moens. Spatial role la- beling: Towards extraction of spatial relations from natural language. ACM Transactions Speech and Language Processing (TSLP), 8(3):1-36, 2011.
[LB18] Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sete to-suuueceeee recurrent networks, 2018.
[LeC22] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-
27. Open Review, 62(1), 2022.
[LKH+22] Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadeh. A systematic investigation of commonsense knowledge in large language models, 2022.
[LWG+22] Ruibo Liu, Jason Wei, Shixiang Shane Gu, Ye- Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M. Dai. Mind's eye: Grounded language model reasoning through simulation, 2022.

---

[MFNK21]
Roshanak Mirzaee, Hossein Rajaby Faghihi. Qiang Ning, and Parisa Kordjmashidi. Spariqa: textual question answering benchmark for spatial reasoning, North American Chapter of the Association for Computational Linguistics, 2021.
[MGSS2I] William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A Smith. Provable limitations of acquiring meaning from ungrounded form: What will future language models under- stand? Transactions of the Association for Computational Linguisties, 9:1047-1060, 2021.
[MHV*24] Ida Momennejad, Hosein Hasanbeig, Felipe Vieira, Hiteshi Sharma, Robert Os- azuwa Ness, Nebojsa Jojic, Hamid Palangi, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. Arxiv: http://arxiioorr/aab233330..///1 1512///99 2024.
[MK09] Samuel T Moulton and Stephen M Kosslyn. Imagining predictions: mental imagery as mental emulation. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521):1273-122)
2009.
[MK22] Roshanak Mirzaee and Parisa Kordjamshidi. Transfer learning with synthetic corpora for spatial role labeling and reasoning. Conference on Empirical Methods in Natural Language Processing, 2022.
[MXF*23] Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gon- zalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines, 2023.
[NLW23] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.0094/ 2023.
[0A+23] OpenAI, ;, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna- Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cum- mings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni. Gabriel Goh, Rapha Gontijo-Lopes. Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross Shixiang Shane Gu. Yufei Guo. Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Fukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew. Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Ja- cob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeon- woo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo,
12

---

Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Pas- sos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filippe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto. Michael, Pokorny, Michelle Pokrass, Vitchyr Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae. Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schul- man, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl. Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
[Ope23] OpenAI. Gpt-ov(ision) system card. 2023.
[PP22] Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations, 2022.
[RAB*20] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. benchmark for systematic generalization in grounded language understanding. Advances in neural information processing systems, 33:19861-19877 2020.
[Reg19] John Regehr. Explaining code using ascii art, 2019.
[RF Julia Rozanova, Deborah Ferreira, Krishna Dubba, Weiwei Cheng, Dell Zhang, and Andre Freitas. Grounding natural language instructions: Can large language models capture spatial information?, 2021.
[RKH*21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[RMK18] Taher Rahgooy, Umar Manzoor, and Parisa Kordjamshidi. Visually guided spatial relation extraction from text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Papers), pages 788-794, 2018
[SB14] 14] Anthony J Smith and Joanna J Bryson. A logical approach to building dungeons: Answer set programming for hierarchical procedural content generation in roguelike games. In Proceedings of the 50th Anniversary Convention of the AISB, 2014.
[SF72] Roger N Shepard and Christine Feng. A chronometric study of mental paper folding. Cognitive psychology. 5(2):228-243 1972.
[She78] Roger N Shepard. The mental image. American psychologist, 33(2):125, 1978.
[SLYA17] Alane Suhr, Mike Lewis, James Ych, and Yoav Artzi. A corpus of natural language for visual reasoning. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217-223, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[SM71] Roger N. Shepard and Jacqueline Metzler, Mental rotation of three-dimensional objects. Science, 171:701- 703, 1971.
13

---

[SMM21] Harini Sampath. Alice Merrick, and Andrew Maevean. Accessibility of command line interfaces. In Proceedings of the 2021 CHI Conference on Human Factors Computing Systems, pages 1-10, 2021.
[SZL22] Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts. AAAI Conference on Artificial Intelligence, 2022.
[TLI+23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv.org, 2023.
[Tol48] Edward C Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
[WBC+15] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks, 2015.
[YBL+23) Yutaro Yamada, Yihan Bao, Andrew K. Lampinen, Jungo Kasai, and Ilker Yildirim. Evaluating spatial understanding of large language models, 2023.
[YIL23] Zhun Yang, Adam Ishay, and Joohyung Lee. Coupling large language models with logic programming for robust and general reasoning from text, 2023.
14

---

A Synthetic Data
A.1 Visual Navigation
As depicted in 1, given a specific k. the process of generating a 2D navigation map is composed of 3 steps, which are instruction generation, instruction simulation, map rendering. In instruction generation step, we enumerate all possible instruction sets navigating from the starting point to the destination (e.g
up, then move right). During this step, only the direction of each instruction is determined, while the moving distance is undetermined until next step. In instruction simulation step, simulation applied in the 2D coordinate system with origin (0, as the starting point. To guarantee an unique answer in each navigation map, the moving distance of each instruction is dynamically calculated to avoid overlapping. Each time when an overlapping is detected, the moving distance of previous instruction will be increased by unit recursively until overlapping is resolved. As the distance is determined, those corresponding points are added to the navigating path. After all instructions are completed, the final point is marked as the destination. In the map rendering step, the bounding box those points is adopted and normalized to 2D square grid. The starting point and destination are marked with dedicated squares, and cells along the path are marked by empty squares, while other untouched cells are marked by obstacle squares.
Algorithm 1: Navigation Map Generation
Input :k
Output : Collution = {81, 82, $2, 8n Crentual may = (1.,,.).... 1n).
Crisual_ _amppl_appppp = (U1, U2, } where n 2k+1
dirs & (uup, left left, down, right)
2 instruction_seld <- gen_innnn inniinnuuctiondirrt k)
for dtr_instruct cur posp path ficchhpponts,..tttt moves +- initialize() for direction in dir jrr_ssstuuctss do
instruction sets do settioosssstttteeeeeeeeeeeeeee
// origin as the starting point
// instruction simulation
if not validate validate_plan(cus pwwplclaaaappppe _os, direction, path_points) then gerreas_preeviou__meee wwweemmvvvveooooeee moves, path path_points) end cur eur_pos, potios + for__orraddcc__pp,,oo direction, path_ moves +- moves direction end 12
10
- moves
13
4, U2 < render wttp:aaappptttteuuuuueeeee htwprraettoounnnnnooooopaa.ooon
14
Csolution + Csalution Us:
15
Ctextual_mag + Ctextuallmaan map Uf;
16
Cuisnal map f- C Coisual_mon U Uj
17
end
Since the direction of each navigation instruction is alternating, there are 4 2 2 kinds of spatial configurations for k-hop navigation map. During the implementation, we simplify the recursive implementation with early quit when path overlapping could not be resolved within one iteration, the main consideration of which is the size the map. So the number of generated map is slightly lower than 26+1 as the navigating step le increases.
A.2
Visual Tiling
The data generation process comprises 3 stages, including configuration generation, question gen- eration and polyomino rendering. In the configuration generation stage, to generate valid spatial configurations of rectangle and the corresponding polyomino set, we convert tiling problem to existing formalized problems. One of the problems is an exact cover problem leveraging dancing link algorithm [Knu00], which could be described as: given matrix of Os and Is, find a set of rows containing exactly one in each column. The conversion is to construct matrix of Os and 1s, each row of which represents possible arrangement of placing a specific polyomino
rectangle. As illustrated in Equation 9, given & polyomino pieces, and rectangle of n units to be filled, the first
15

---

columns compose an one-hot vector indicating the corresponding polyomino, and the last n columns are marked with 0 or depending on whether the corresponding unit is filled by that polyomino. Then finding a set of polyomino arrangements in rectangle equals to find a set of rows containing exactly one each column. Another adaptable problem the boolean satisfiability problem (commonly known as SAT), for which efficient solvers exist [ES03, GN07J. A tiling problem can be converted to SAT by introducing a boolean variable for each possible arrangement of each piece, and then adding clauses comprising of those boolean variables that ensure at least one arrangement of each piece is achieved, while avoiding conflicts between arrangements of one piece or two different pieces.
Given the size of a rectangle and polyominoes to be fit, multiple corresponding solutions are generated by applying those algorithms. Then the question generation stage, we randomly mask several polyomino pieces in the rectangle, and generate a question answer(QA) pair for each masked polyomino. Finally the rectangle and each polyomino piece are rendered with emoji squares.
C
Ck
C
Cn+k
0
0
0
1
P1
0
0
1....0
0
Pk
0......1
0 ..... 1
A.3 Visual Data Rendering
After gathering the textual dataset 2D square grid, we generate the corresponding visual dataset by drawing text onto an image. Specifically adopt color emojis for fair comparison as they' hre'rre more visual friendly to a multimodal model.
A.4 Dataset Details
Data distribution among various difficulty levels for visual navigation tasks and visual tiling tasks are provided in Table 4 and 5. It provides flexible difficulty control across different tasks. For visual tiling task, the difficulty is controlled by the number of masked polyomino pieces. As the number increases, the more spatial arrangements LLMs need to consider. Regarding the visual navigation task, as illustrated figure2, we use the number of roads & to control difficulty, which corresponding to the size of the map.
K Step
Mask count
Task
Total
Total
2
4
5
6
7
2
3
Planning
128
248
Configuration
248
124
376
Next Step Prediction
96
256
640
1488
2520
QA Instance
307
Table Data distribution of visual navigation dataset with the total navigating step of & indicating difficulty level. The reason why the number of generated map is slightly lower than 26+1 for h > 5 is explained in Appendix A.1.
Table Details of visual tiling dataset. Some QA instances are discarded when multiple solutions exist and all answers are correct.
B Examples
For visual navigation and visual tiling tasks, the structured input template is comprised of task instruction, input parameters and prompt of specific setting.
B.1
Visual Tasks
Task instructions and responses of each visual task under setting GPT-4 VoT are provided following:
Route Planning Task instruction in Figure 6, response in Figure 12.
16

---

Next Step Prediction Task instruction in Figure 6, response in Figure 13.
Visual Tiling Task instruction in Figure 7, response in Figure 14.
Route Planning
Next Step Prediction
starting point, the office the dentization. mears the nod. be mosts ubelacle. There exicts one and only soe viable roste for esch map. Each sap you chouse a diretion and mnce to the end the outtinucus roa or destinaton. masp.
Novigstion peosided nap,
Navigation Tosk: provided may, oéfice the destination. mosts the rodd mosts the ubetacle There cxists one and only one vastle muse for tech map. Each sup you chouse dinetion more to and the continurus soad the destnation map.
starting point, @
and pad
pot pot ond
Starting steps nnvigate M0.
Starting raviigtte you made following movomente contauous
Move continuous
masoning stop.
Move of continuous
What' direction nest movement?
Left
Down
D. Right
Visuelite the sote ofter each missoingg ste.
Figure 6: Task Instruction of visual navigation.
Task: civen of polromizess comspunding vatatius each polyomins, Et the empty
Variation fitting into it bounding box:
Varatiu 3 fitting into bounding box.
retangle cutade rettanll.. The reflection.
veriaions allo: only inndutionn, quaenteed alwyys a
Target Tanget metangie smptt aquares
Vadiatioss for Tetrmnin L:
Viriation 3 fitting into bounding bot:
polsominees the squars conect vatiation Tetromino
Vartation 7 fitting into bounding box:
mesoning step.
Prwvied pilyorinooss
Netronino 1:
Tetronimo
Tetronino
Variation: I fitting into hounding bot:
Variations for Tetrmino
Varlabions for Tetramino
Vartabion 2 mitting into its bounding bot:
Figure 7: Task Instruction of visual tiling.
B.2 Natural Language Navigation
Prompt Example You have been given 3 by 3 square grid. Starting from vertex, you will move along the edges of the grid. Initially, you are positioned at the bottom-left corner of the grid, where you will find torch, then you go right, where you will find an infant bed, then you go right, where you will find an American dipper. Then you go up, where you will find a jay, then you go left, where you will find terrapin, then you go left, where you will find microwave oven. Then you go up.
17

---

Visualize the after each reasoning
Let's think step by step.
First lets look ot the larget reciongle and iómitity the equines
First, lets look at the englyy spaces larget redinngle We have 4c1 the botton stugle squors The space the bottonn so well use Vartaton 2 the Letromtno to fill that
pelyominoes:
can only fitm
Now, let's coasider the Vetromino I vatations We have firee staçle empty am not strigitt line, se Vanaton with prrbusor, will not fit, Hooweve; Vanaztnn 3 of the Tetromino I has coniguuation
Tetrommno
is striight tine
We have beo pelyominoes to into this rectangle: Tetromino I and letrozino' I. The Letromino
the bettom unly place with four cocsecutive suply squares, either buricontally or vertisllyy Let's place the Tetronnno I in the botoo
timee remaining emply squarss one squace protioding to the side, which matches the layout the emplyy squass.
T'heefrre, the comoor vonadion of Tetrsmine Th into the rensitingg empty squers
I'e final configumtion would look like this.
letromino ( perlectly remaining emjty aqures mutches the shape empty squane, Let's plaze Varielion 2 Tetromino T. Lato the larget reclangle:
sornet varietion Tetomizo Gilnto the empty square Vacistion
A Answen A.
(a) Common behaviors of VoT and CoT prompting in visual tiling task, with the overall track rate of 87.1 1% and 57.4% respectively.
Visualize the state after each reasoning step
Visualize the state after each reasoning
To nevigate from Wi to (.). we seed to fiod the path that avoids obstacles
( proided map. follow instruction:: 1. Move down the end of the continmoss med.
navigata
(90) follows rads [[ Hend' step- step non gatton:
Mova downn from and of continoous mad.
2. Move right to the end of the continucus road.
juok di
3. Move up to moch the destination Mil.
Here's visualiattonn
step
Starting point:
d
A forr Stop 1 [Move downte
Move left from the curent poeltion to the office T
2wt aud db
2ed A
pue pad
Step tight::
pnet put d
Aler the move:
pM pm
A fler Step (Move uph:
di
has now tecche
completing zavigation teck)
(b) Route planning task is sensitive to prompts. After deleting the word "reasoning" from the VoT prompt, final answer and state is derived without conditioning on state visualization in many cases. The left example provides correct solution with imperfect intermediate visualization, while the right example provides wrong solution.
Figure 8: Example cases of visual state tracking behaviors in different tasks. We provide side-by-side comparisons to show how visual state tracking is influenced by prompt, where bold texts on the top are prompts to compare. Case on the left exhibits visual state tracking in an interleaved manner, where next state conditioned on visualization of previous states. Case on the right disentangles verbal reasoning and state visualization, which leads to a drop in tracking rate and task performance.
18

---

where you will find a baseball player, then you go right, where you will find harvestman, then you go right, where you will find a neck brace. Now you have all the information on the map. You start at the position where the torch is located, then you go right by one step, then you go right by one step, then you go up by one step, then you go up by one step, then you go left by one step, then you go down by one step, and then you go down by one step. What will you find?
Response Example See Figure 15.
C
Visual State Tracking
As for where this emergent ability stems from, it might derive from tabular data, city grid naviga- tion, maze exploration related coding problems [YBL*23], These tasks involves understanding and manipulating objects in 2D square grid. Besides, we conjecture the exposure of ascii-art art com- ments [Reg19] during LLMs'' code pre-training possibly enhances this generalized ability, As a fact to support this conjecture, the visual tiling task is different from navigation tasks because it requires shape understanding and spatial manipulation ability. While tabular data and square grid navigation data boost row-wise column- wise attention, ascil-art supplements intricate spatial attention to understand and manipulate 2D shapes. Additionally, ascii-art in code comments is presented in various formats, of which is interleaved ascii diagrams, natural language and programming language. It require LLMs to generate the interleaved mental images and text sequence, thereby enhancing spatial visualization ability and spatiotemporal causality. Interestingly in the natural language navigation task, when GPT-4 is prompted with "use ascii-art to visualize", the complete tracking rate increases to 98.5% (+78.5%), boosting task performance to 62.5% (+3.5%).
C.1 Ascii-art in Code Comments
Ascii-art is commonly used in code comments represent data structure, diagram, geometry and so on, which could benefit LLMs' spatial understanding and visualization capability. Besides, it's also used to illustrate how an algorithm works or simulate an operation, where reasoning traces and corresponding visualization are presented in an interleaved manner. Below are several examples in open-source projects.
Spatial nqusality:Doubleenddee queue in Rust, Scrolling web pages and tree rotation present triplets of previous visual state, instruction, and updated state of instruction follow- ing.
Temporal Causality: Undo systems from emacs provides various temporal states of the undo system when undo operation happens in different timelines and corresponding visualizations in an interleaved manner. Each visualization reflects the temporal casuality of the system state.
This kind of interleaved sequence tracks the system state over time, thus reflecting spatiotemporal casuality.
D Performance Trends Across Levels
In this analysis, we examine performance trends across varying difficulty levels in the next-step prediction task for models utilizing either CoT or VoT methods. These trends are crucial for understanding the inherent unpredictability associated with random guessing. As k: increases from to in k-step navigation map, distinct performance patterns emerge among different models. as depicted in Figure 9. Larger language models such as GPT-4 and LLAMA3-70B demonstrate more predictable decrease in accuracy with increasing k. This trend indicates robust ability handle progressively challenging tasks, despite the overall decrease in performance. Detailed statistics are provided in table 6. In contrast, less powerful models like GPT-3.5 and LLAMA3-S8 exhibit an irregular performance trajectory. These models show variable accuracy, with significant fluctuations at higher difficulty levels, suggesting a reliance on random guessing, particularly under conditions of increased task difficulty. This behavior highlights their limitations in sustaining reliable reasoning processes through more complex scenarios. Furthermore, the VoT method seems to offer modest improvement in performance for the less powerful models, particularly in scenarios of lower difficulty.
19

---

This observation suggests that VoT might be advantageous for enhancing reliable reasoning in simpler spatial reasoning tasks, potentially compensating for the inherent weaknesses of smaller language models.
CoT Models Performance Trend
VoT Models Performance Trend
100
100
80
80
Accuracy (%)
Accuracy (%)
60
60
40
40
20
20
2
3
4
6
2
3
5
6
K-step Map
K-step Map
GPT-4
GPT-3.5
LLAMA3-70B
LLAMA3-8B
Figure Performance Trends of CoT and VoT Models Across difficulty levels in ceatsstep-reeicctonn task.
Model
K-step Map
Map Count
CoT Accuracy (%)
VoT Accuracy (%)
2
8
75.00
75.00
32
68.75
62.50
GPT-4
4
96
60.42
68.75
5
256
50.78
64.06
6
640
52.34
55.16
7
1488
45.30
52.69
62.50
62.50
3
32
68.75
65.63
LLama3-70B
96
60.42
62.50
5
256
56.25
57.42
640
48.59
54.84
7
1488
46.71
52.35
Table 6: CoT and VoT performance of advanced models in next-step-prediction task across various difficulty levels. While performance drops as difficulty level increases, VoT method generally maintains a higher accuracy compared to CoT, highlighting its robustness in more challenging scenarios.
E Case study
We consider visual state tracking similar to spatiotemporal simulation. During the simulation in those tasks, we discovered several interesting behaviors of L.M..
1. Diverse visualization formats for state tracking: Nearly 30 different symbols found in the navigation tasks to track the navigation progress, including marking the path, marking the current location. Among those diverse representations, LLM succeeded in some challenging cases where it used directional arrow emojis to indicate both the location and moving direction at each step. More examples could be found in Appendix E.
2. Inconsistency between language and visualization: This is commonly observed across all tasks. Due to the limited visualization capability, sometimes LLM generates accurate language instruction but inaccurate visualization. And in other cases, L.M generates wrong answers even the visualization generated correctly, which reflects its limitation of spatial understanding as discussed in previous section. More examples could be found in Appendix E.2.
20

---

3. Self-refine mechanism: We found several cases in visual tiling tasks where spatial hallucination happens due to the inconsistency inaccurate visualization. Subsequently, LLM refined its reasoning. resulting in an accurate visualization and the correction of the final answer. More examples could be found in Appendix F.3.
E.1 mental images for State Tracking
In the visual navigation task, LLM adopted various symbols and representations to track the state of navigation progress. As shown in Figure 10, there's re several tracking styles.
Mark the path: adopting identical symbol to mark current location or part of the path.
Mark path and direction: using directional arrows to mark current location and indicate the moving direction simultaneously, which more challenging than simply marking the path.
Mark path with temporal steps: using numbers to demonstrate both temporal steps and current location.
Remove road: turning roads into obstacles avoid turning back, instead of adopting additional symbols to mark the path.
E.2
Inconsistency between Language and Visualization
In the visual tiling task, two inconsistent steps are highlighted in Figure 16. One is the inconsistent visualization with the language instruction of "place Variation 6 of Tetromino L". Another is the wrong decision to chose "Variation 2 of Tetromino given the visualization of the valid state.
F.3 Self-refine Mechanism
We found visualization could enhance LLM's reasoning by self- self-grounding and refining subsequent reasoning steps in some cases. As shown Figure 17, despite successfully identifying variation of tetromino L incorrect option, GPT-4 excluded the correct option of variation 6 even it's placed accurately due to spatial hallucination (overlapping with yellow pieces), which led to a impossible solution. Then it detected the mistake and re-evaluate the placement of variation 6. Finally it placed the correct piece into the top left corner and validated the answer by filling the remaining space.
21

---

Use round pin
Use checklist
fit.
pat paat
pane and
pnd just
and
paz pnd
62
and
pag
Original map
1. Move down
2. Move right
Original map
Move down
just jost jast jaot
pnd pad part (aat
aae pag and pnd
Remove road avoid turning back
jook jace jant
pad pas punt
and
jast juot
pant pat
200 png
joot (1]
pM (U)
junt ine
Move down
Move right
5. Move down
Original map
Move right
Use arrows reflect direction
Use numbers for temporal steps
Original map
1. Move right
2. Move down
Original map
1. Move left
2. Move down
3. Move right
4. Move down
5. Move left
3. Move right
4. Move down
5. Move right
Figure 10: Mental images for state tracking in visual navigation tasks. We found diverse visualization formats in LLMS to track the navigation progress, including marking the path, marking the current location, marking directional steps, and marking temporal steps. More details are available in Appendix E.
22

---

The ACTI art you've provided reprosents three- dinesinit cube witl labeled vertices from 0 untadd the face vertices 2. und St, would flt. cuming
the fromt lase (wluch frott face with vertices 2, and and we unfold downwanls, the night lock sometting like this:
Astii culee
2.5
Now, uzfold the fouzt face (1, 6, dowzwards
15
3
Untisld the froo Bace
2, untaliedd dowawands, vertical lines connectao the become borizontal, represented bus-dinencioall fooe fase infilded. Keep that simplifted regemsenaation would connected edges without gage
Figure 11: Representation for 3D semantics and qualitative results of GPT-4 manipulating the mental image correctly.
Sterting from
Jets reriigate by
Stop leftito the-end the continsous mad.
Star Move down to mach the
Inibal
ins and sit
pos dwd pat
pat and
ane ooe A A.
pas
pnd PA
Pag and
A
Aed And png
kne dos
avs ave ang aus
pas ane put put
Strp 4 More down the end the continacus mad.
Step More right from to the and of the continusue road.
The stops to zaxigate from
amt:
pue and aud
joet dic
pas aoet ave
pne owe
ans and
3.
Teft
Down
and pwe
nag and ase aas
pot ane pet
5 T.ft
ane
Stap St More left to the end of the continucus mad.
Stop More dowva to the end of the continuous road.
(we) dic
We hove now mached the dostination,
past aone anp
A
dic
A.
and Det and pet 3
and pae ang pnd
pwtt ane put ooe
pont
pee and ane pne
Figure 12: Responses of Route Planning
23

---

Litss visualioe the state after each movament:
Moue loft the and of the cintimanes mod:
Inabial
pon and and 2A4 J
pae
A J.
ane and and pat
ane ane ave
anr pue put
Move right to the and of the contimaruss road
Now, the sest movamentoptions wee
dit
Up- This would lesd lack previous positon which visble mose the destination
Lafi This prestble there olstacle ee) immediatell the
Tais would load the destination Right
obstoele immediltbll right.
pant pee
direction
ang and and ang
Montng
(())
2. Move dowaa the and of the continuous road:
di
eet pwt
oue par
2at ant
(and pnt
pant pat
Figure 13: Response of Next Step Prediction
24

---

by examining empty spaces rectang provided
polyominoes and their variations. We need to the Tetromino I, Tetromino and Tetromino into the empty spaces without overtapping any existing polyominoes or going rectangle.
The target rectangle with 12 empty squares looks like this:
have two variations Tatromino J, which can either placed vertically or
horizontally. Given the configuration of the empty spaces, way place the
Tetromino is horizontally at the top row, as the vertical placement would not fit anywhere.
After placing Tetromino {(
J, the target rectangle looks like this:
Next, we fitin the bottom left corner, while Variation 7 would not fit anywhere without overlapping or going outside the rectangle.
Tetromino
placing Tetromino L
target rectangle
(a) Response
Visual Tiling Page
25

---

Now, we have two variations for Tetromino T We need to this into the remaining would
empty spaces. Variation of Tetromino T cannot spoces next to each other vertically, which we da not have. However, Variation 3 of Tetromino fits perfectly into the remaining space.
ampty
placing Tetromino rectantle looks
The correct variation of Tetrominn T that fits into the empty squares is:
B. 3
(b) Response of Visual Tiling Page 2
Figure 14: Responses of Visual Tiling
26

---

Let's visualize the grid with the items you ve described:
5 BIM
cIVi
ITINIC
Where:
T= Cassette player
W Wool
C Conch
V Muving
C Confectionery
pie
Siamang
Black-and-white colobus
M = Minivan
You start at the bottom-left corner where the cassette player is located Now, let's follow steps:
right by one stop Wool W):
5
civ.
[***]
2. Go right by one step (to Conch C):
clcvi
(a) Response Natural Language Navigation Page
27

---

Go one step Moving van V):
s I
becebeedde. I'v"
T [
4. Go left by one step (to Confectionery store C):
s ISIIIN terckeodteo P.P. ("c*
Go left by one step (to Pot pie P):
I
|*pe| civ
6. Go by one step (to Siamang- S):
[+50| B I
right step Black-and-white colobus B):
(b) Response of Natural Language Navigation Page
Figure 15: Responses of Natural Language Navigation
28

---

S Pe") CIV
CNNCI
Go down by one step (to Confectionery store - C):
I ISIMI
velv
After following all the steps, you will find yourself at the Confectionery store (C).
Answer: Confectionery
(c) Response of Natural Language Navigation Page 3
Figure 15: Responses of Natural Language Navigation
29

---

First, let's look the target rectangle and identiff the empty squares
We have two polyominoes fit into this rectengle: Tetromlino L Tetromino the variations provided fill the empty squares without overlapping existing polyorninees going outside the rectangle.
We
because unique shape protrusions. Looking the veriations of Tetromino L:
Varistion
Variation 6:
We that Variation of Tetromino fits perfactly into the top left corner of the target rectangle, filling three empty squares vertically and turning to one horizontally. Let's place Variation Tetromino the target rectantee:
(a) Inconsistency between language and visualization Page
30

---

empty squares remeininng straight line, which matches the shape Tetromino I. Let's look at the variations of Tetromino I:
Variation 1:
Variation 2:
Since remainiin squares horizontal Varistion Tetromino will
perfectly. Let's place Variation 2 of Tetromino into the target rectangle:
The correct variation Tetromino that fits into the target rectangle
(b) Inconsistency between language and visualization Page
Figure 16: Inconsistency between language and visualization
31

---

Let's start by visualizing the target rectangle with the emply squares polyominoes:
and the existing
We have polyominoes fit into this rectangle: Tetromino and Tetromino We need to determine which variation of Tetromina will fit without overlapping any existing polyaminoes or going outside the rectangle.
Let's consider the variations for Tetromino L:
Variation 1:
Variation 6:
We also have the variations for Tetromino but since it's straight line, it can only in one way remaining space placing
Let's try to Variation
Tetromino first:
(a) Example case Self-refine
Page
32

---

As we can sec, Variation Tetromino does not fit because overlaps with the existing
yallow polyomino.
Now let's tr Variation & of Tetromino L:
goes rectangle
Since nelther Varistion Veriation Tetromino into the terget rectengle without overlapping existing polyominoes going outside the rectangle, the correct
Neither
However,
there seems to be mistake because the task guarantees that there always
exists solution, Let's re- e-eevaluate the placement Tetromino
We can see thet the Tetromino L must int the top left cornei because that's the only place with enough space shape. try placing Variation Tetromino top left
(b) Example case of Self-refine Page 2
Figure 17: Example case of Self-refine
33

---

This time, Variation Tetromino fits perfectly the top left comeet Now, we can place remaining space:
The Tetromino I fits perfectly in the remaining space using Variation (vertical placement). Therefore, the correct variation of Tetromino that into the target rectangle
A.
The initial placed in the top left corner. Upon correcting the placement, find that Variation does indeed fit, and the Tetromino no can be placed afterward to complete the puzzle.
Variation Tetromino incorrect because
(c) Example case of Self-refine Page
Figure 17: Example case of Self-refine
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We confirm that the main claims made in the abstract and introduction aceu- rately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We confirm the limitations of the work has been fully discussed, which could find in the Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
35

---

Answer: [Yes]
Justification:
We theoretically formulate the research topic we investigate in the Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We confirm that this paper does fully disclose all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
36

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The data and code associated with this study is publicly available and the link is provided in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httpssccccpbccccccccpeccppcccclt httllii//iieeeee gww.ee//eeeeee CoodeSubmi ssi aesionooiccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We do include all the details of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We do conduct a significance test on the experiments, and make explanations in Table 1 and 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have clearly reported the version of the API used, and the calculation time can be obtained based on the API version.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: /et::://neursspee
Answer: [Yes]
Justification:
We confirm that our research conform with the NeurlPs Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on one of the fundamental reasoning capabilities of LLMs. The tasks developed include navigation and geometric shape reasoning, which is very unlikely to cause negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
38

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
All external assets, including models and algorithms utilized this paper, are duly cited, ensuring proper credit the original creators. Furthermore, we have meticulously adhered to the specified licensing agreements and terms of use associated with these assets.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
39

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper provides comprehensive documentation, including implementation details, prompt template and the distribution of the dataset. See Appendix A and B.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research do not have any potential risks to human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
40

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "f2BgHQUDLf", "reviewer": "Reviewer_MbkM", "summary": "The paper presents a novel approach called Visualization-of-Thought (VoT) prompting, aimed at enhancing the spatial reasoning abilities of Large Language Models (LLMs). Inspired by the human cognitive process known as the “Mind’s Eye,” the authors propose a method where LLMs visualize their reasoning steps to guide subsequent reasoning. The paper evaluates VoT across three spatial reasoning tasks: natural language navigation, visual navigation, and visual tiling. Experimental results demonstrate significant performance improvements with VoT over conventional prompting methods and existing multimodal large language models (MLLMs).", "questions": "See above Weakness part.", "limitations": "See above Weakness part.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Strengths\n\n1. **Innovative Direction**: The concept of eliciting spatial reasoning through visualization is novel and well-motivated. Drawing parallels with human cognition adds a strong cognitive science foundation to the work.\n\n2. **Comprehensive Evaluation and Improved Performance**: The paper evaluates VoT across multiple tasks and benchmarks, providing a thorough analysis of its effectiveness. The results show that VoT significantly enhances spatial reasoning abilities in LLMs, outperforming existing methods. Meawhile, this paper includes detailed qualitative and quantitative analyses, shedding light on the mechanisms behind VoT’s success and its limitations.", "weaknesses": "Weaknesses\n\n1. **Limited Task Diversity**: While the selected tasks cover various aspects of spatial reasoning, the evaluation could be broadened to include more complex and varied real-world scenarios. In the main paper, we even didnt see more commonly adopted benchmark performance comparison. Include more varied and complex spatial reasoning tasks to better evaluate the generalizability and robustness of VoT.\n\n2. **Generalizability**: The paper primarily focuses on GPT-4 and GPT-4V. It would be beneficial to see how VoT performs across a wider range of models, particularly less advanced ones. It would be great to see more evaluations of VoT on a wider range of models, including less advanced ones, to assess its applicability across different LLMs.\n\n3. **Prompt Sensitivity**: The effectiveness of VoT appears to be sensitive to the specific wording of prompts, which may limit its robustness in practical applications. Investigate ways to reduce the sensitivity of VoT to specific prompt wording, making it more robust for practical use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough evaluation and constructive comments of our work. We'd like to clarify:\n> **Weakness 1: Limited Task Diversity**\n\nWe fully appreciate the commonly adopted benchmark such as bAbI[1], StepGame[2], SpartQA[3], SPARTUN[4] etc., which lay solid foundations for spatial reasoning based on text understanding. As explained in section 2 (line 73-77), our main concern is that as the development of LLMs and their exceling in linguistic and logic reasoning, it's challenging to provide an accurate measure of their spatial awareness with these tasks. So we carefully select tasks to cover various aspects of spatial reasoning and offer flexible difficulty controls. These visual tasks with grid input avoid LLM's linguistic and logic \"short-cut\", and the dynamic nature of navigation tasks requires LLMs to track the changing state in the environment. Despite these tasks are relatively simple to humans, they are challenging to current stage LLMs.\n> **Weakness 2: Generalizability**\n\nWe evaluated VoT and other baselines in GPT-family models (GPT-4/4V/3.5 turbo) and LLama3 family (LLama3-8B-instruct/70B-instruct). The resuluts and corresponding analysis are provided in **section 5.3**. Although the overall performance improvement of VoT in less advanced models is not as significant as in advanced models (least significant for LLama3-8B), the improvement in easier levels are distinctive (**discussed in appendix D**). As difficulty level increases, less advanced models exhibit irregular performance trajectory for both CoT and VoT. This observation indicates their inherent weakness, i.e., reliance on random guessing for increased task difficulty.\n> **Weakness 3: Prompt Sensitivity**\n\nWe adopted task-agnostic 0-shot prompting for VoT as it's the simplest approach to mimic corresponding human cognition in LLMs, and assure a fair comparison with other baselines. We only scratched the surface of this ability in LLMs and hope this initiative could possibly inspire others. For practical usage, users can add task-aware specifications about what to visualize and how to visualize, to force LLMs appropriately generate visualizations to guide subsequent reasoning. More fundamentally, this limitation could be addressed by instruction tuning with visual state tracking data. Due to the space limitation, we discuss the possibility that VoT might benefit from code pre-training in **appendix C (line 600-627)**.\n\n\nWe hope that our clarifications address your concerns and provide a better understanding of our work.\n\n## References ##\n[1] Weston, Jason, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. \"Towards ai-complete question answering: A set of prerequisite toy tasks.\" arXiv preprint arXiv:1502.05698 (2015).\n\n[2] Shi, Zhengxiang, Qiang Zhang, and Aldo Lipani. \"Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 10, pp. 11321-11329. 2022.\n\n[3] Mirzaee, Roshanak, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjmashidi. \"Spartqa:: A textual question answering benchmark for spatial reasoning.\" arXiv preprint arXiv:2104.05832 (2021).\n\n[4] Mirzaee, Roshanak, and Parisa Kordjamshidi. \"Transfer learning with synthetic corpora for spatial role labeling and reasoning.\" arXiv preprint arXiv:2210.16952 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YGr8fKS4RB", "reviewer": "Reviewer_us5D", "summary": "This paper focuses on enhancing the spatial reasoning capabilities of Large Language Models (LLMs) by introducing Visualization-of-Thought (VoT) prompting. Inspired by the human cognitive ability to visualize unseen objects - a process known as the Mind’s Eye - VoT visualizes reasoning processes to guide LLMs in tasks like natural language navigation and visual navigation in 2D grid worlds.", "questions": "Please refer to weaknesses.", "limitations": "This paper provides the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of Visualization-of-Thought (VoT) prompting represents a new method that mimics human cognitive processes, specifically the ability to visualize mental images for better spatial reasoning.\n2. To investigate the effectiveness of the proposed VoT, this paper considers three spatial-related tasks, including natural language navigation, visual navigation, and visual tiling.", "weaknesses": "**Method\n\nSome descriptions are unclear.\n\n1. The method for obtaining the visuospatial sketchpad $v$ remains unclear. This paper describes $v$ as representing a visualization sequence in text form, but it does not specify how visual information (maybe images) is converted into text.\n\n2. If visual information is transformed into text form, how does GPT-4V use it? Is it used as a textual representation of the visual image as well?\n\n**Experiment\n\n1. This paper also introduces the concept of state $s$. What is the performance comparison when this state is included versus when it is excluded?\n\n2. In Table 1, for the visual navigation and visual tiling tasks, why does GPT-4V CoT perform the poorest, even yielding results lower than GPT-4 without visual information?\n\n3. It seems that Section 5.2 aims to discuss “how visualization enhance final answers”, yet the connection between the results in Table 2 and this discussion remains unclear. It would be beneficial to include more detailed descriptions or results related to this.\n\n4. In Section 5.3, Table 3 shows that Llama3-8B with VoT underperforms in most results compared to its counterpart with CoT. Does this suggest that the proposed VoT does not benefit less powerful language models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, which are invaluable in helping us refine and clarify our work. We are grateful for the opportunity to address the points raised and provide further clarification on the aspects that may have been misunderstood.\n> **Weakness 1: Unclear descriptions**\n\nThere might be some misunderstanding about the modality of *mental images* in LLMs during spatial reasoning. As explained in **line 42-44**, VoT leverages LLMs' potential ability of ascii-art for visualization, rather than depending on external visualization tools. We also provide explanations about ascii-art contained in code comments and some examples in appendix C. Regarding the input format, we specify the use of a 2D grid with special characters in **line 48**. Due to space limitations, detailed implementation information for the textual 2D grid dataset is presented in Appendix A, along with the rendering method used to generate the corresponding image input for GPT-4V.\n\n**In summary**, during spatial reasoning, LLMs generate ascii-art as text-form visualizations. The image input used by GPT-4V is produced by rendering the special characters from the textual 2D grid, rather than the reverse process.\n\n> **Weakness 2: Experiment**\n\n> **W2-1: Concept of state `s`**\n\nWe introduce the concept of state `s` in line 139 to explain the **visual state tracking** nature of VoT. We discuss the visual state tracking behaviors among different baselines in **section 5.1**. As could be seen in Figure 5, there is a notable disparity in state tracking rate between **GPT-4 VoT** and **GPT-4 w/o Viz** across all tasks. We hope this comparison could address your concern.\n\n> **W2-2: Poor performance of GPT-4V**\n\nYour observation is indeed insightful. We were also suprised by the poor performance of GPT-4V, particularly in visual navigation task (being the poorest). After conducting a qualitative analysis of the responses, we hypothesize that visual hallucination may be hindering GPT-4V's performance.\n\nTo further investigate, we conducted an additional experiment using 496 maps from the route planning task, focusing on a simple task: identifying the direction of the roads connected to the home icon. Despite the task's simplicity, where only one road begins at the home icon, both GPT-4 and GPT-4V occasionally provided multiple answers. Accuracy is measured by the ratio when the exact answer is outputed.\n\n| Setting | Accuracy | Recall | Avg answers output|\n| -------- | -------- | -------- | -------- |\n| GPT-4V | 2.02% | 66.13% | 2.32 |\n| GPT-4 | 34.07% | 76.81% | 1.50 |\n\nAs could be observed in the table, GPT-4V with image input performs poorest in this easiest task. It indicates that GPT-4V suffers from hallucination in visual navigation task.\n> **W2-3: Unanswered question of section 5.2**\n\nWe appreciate your observation regarding the clarity of Section 5.2. We did analyze the correlation between accurate visualization and final answers. We provided the metric explanation in **line 230-231**, which corresponds to the last column in table 2. And we concluded in **line 235** that accurate visualizations ensure a correct final answer with high probability (above 65%). Similarly, other metrics demonstrated in table 2 are also explained in the first paragraph of section 5.2. They measure the visualization capability of LLMs, e.g., to what degree LLMs can generate accurate visualizations.\n\nWe recognize that the connection between the table and the narrative may not have been as explicit as intended. If these explanations address your questions, we'd like to adjust highlighting of numbers and observations to achieve a better understanding of table 2.\n> **W2-4 Less powerful LMs**\n\nWe acknowledge that the overall performance of Llama3-8B is less impressive when using VoT, as noted in our discussion of limitations in **line 314-315**. We believe this can be attributed to the increased difficulty in handling complex tasks, leading to outcomes akin to random guessing in less powerful models. To further explore this issue, we conducted an analysis of performance trajectory in next-step-prediction task across various difficulty levels, as presented in **line 640-643 in appendix D**. However, as highlighted in **figure 9**, there's a modest improvement when the task complexity is lower (K ≤ 3). This suggests that VoT can potentially **benefit less powerful models in less complex scenarios**.\n\nWe look forward to your further feedback and thank you for your valuable contributions to improving our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kgMIkSOZuZ", "reviewer": "Reviewer_yVwi", "summary": "The paper proposes a new prompting method, \"Visualization of Thought\" (VoT) prompting, to enable LLMs to perform better on spatial reasoning tasks.  In VoT, the LLM is prompted with instructions on performing a multi-step spatial reasoning task, followed by the text \"Visualize the state after each reasoning step\".  The authors evaluate this prompting method on three types of spatial reasoning tasks, and show that it substantially improves performance on these tasks as compared with Chain-of-Thought prompting, as well as with other variations on prompting.", "questions": "-- I was confused by Figure 8b.  The left side shows the house moving -- is that correct?  Also, it doesn't do what the instructions say: e.g., \"move down to the end of the continuous road\" -- after the move the house only moved down one square, not two.  Is this considered a correct solution?    The right side shows a person moving.  Where did the notion of moving a person versus moving a house come from?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "VoT prompting is an interesting and (I think) novel approach to elicit better spatial reasoning in LLMs, and may be useful in multimodal systems as well (this is future work).  The evaluated tasks are chosen to capture important aspects of spatial reasoning abilities. The paper reports on systematic experiments that evaluate these abilities on the chosen tasks.", "weaknesses": "There are three categories of weaknesses which I believe can be improved on by the authors.\n\n1. Some aspects of the paper are unclear and need more explanation.  \n\n-- The paper says \"This method augments LLMs with a visuospatial sketchpad\".  It's unclear what is meant here by visuospatial sketchpad.  This sentence makes it sound like there is a new architectural feature added to the LLM, but my understanding from the rest of the paper is that VoT just asks the LLM to generate a series of \"visualizations\" in its output.  Please make it clearer in the description what exactly you mean.  \n\n-- Figure 4 gives examples of VoT prompting.  But it wasn't clear to me whether these prompts include actual images (e.g., the pictures of barriers and people in the navigation task, or the pictures of colored and empty boxes in the visual tiling task) or whether these images in the figures were actually translated into some kind of text encoding for the LLM.  Please make clear exactly what is in the prompt---images or text---and if text, what exactly is the text encoding of the pictures you are showing? \n\n-- Equation 5 defines v_i as \"visualizations\".  What exactly are these, in terms of LLMs?  Are they text or image outputs from the LLMs?  \n\n-- In several figures (e.g., Figure 4, 6, 7) prompts and responses are shown but it needs to be made very clear which part is the prompt and which part the response from the LLM.  These were not clear enough for me.  \n\n-- In Figure 8, again, what is part of the prompt and what is part of the response?  \n\n-- Figure 10: The caption only says \"mental images for state tracking in visual navigation tasks.\"  Please say more in the caption about what this figure is showing.  Was \"Use round pin\" part of the prompt, or is this just describing what the LLM did?  What are you trying to show in this figure?  In general, please make the figure captions more informative.  \n\n-- \"Specifically, we adopt GPT-4 and GPT-4 Vision\" -- there are several versions of each of these LLMs on the Azure platform -- please specify which ones you used for these experiments.  \n\n-- \"We adopted sub-string matching as f_correct to determine correctness.\"\nThis didn't really explain how you determined correctness.  Please give more details (enough for someone else to replicate) and explain why this sub-string matching method is reliable for assessing correctness. \n\n2. Scaling:  The tasks used were relatively simple: 3x3 maps for navigation, 5x4 rectangle for tiling.  Did you do any experiments with scaling up the size of these?  How does this method's spatial reasoning abilities scale as the size is increased?  \n\n3. Anthropomorphism:  The paper is filled with anthropomorphic language, which makes it difficult to understand.  For example: \n\n-- \"LLMs possess the ability to create and manipulate mental images in the mind's eye.\"   Using terms like \"mental images\" and \"the mind's eye\" assumes that the LLM is doing something analogous to the brain, which is not necessarily true!  \n\n-- \"visual state tracking is implemented by generating the visualization v_i as\na mental image of the internal state si after each reasoning step z\".   A mental image is something a human has.  Using that term for LLMs is unhelpful anthropomorphizing. \n\n-- The paper calls the intermediate outputs of LLMs \"thoughts\" (e.g., \"this process by adding a visuospatial sketchpad to each thought z_i\") -- again, an unhelpful and confusing anthropomorphism.  \n\n-- \"We demonstrate LLMs’ zero-shot capability of visualizing their precepts at abstract level\". Visualizing is a cognitive ability of brains.  Saying that LLMs \"visualize\" is an incorrect anthropomorphism.  \n\n-- \"\"this emergent capability to create mental images to enhance spatial reasoning resembles the mind’s eye process, suggesting its promise in MLLMs.\"  Same comments as above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your careful and constructive review of this paper. Your insights are invaluable, and we have carefully considered each of your comments.\n> **Weakness 1: Unclear details**\n\nWe'll improve clarity of *visuospatial sketchpad*, *caption of figure 4 and 10* in the camera-ready version. The data construction and evaluation code will be released after paper acceptance to facilitate replication. Below are explanations per question:\n1. We recognize that \"augment\" here is misleading. The concept \"visuospatial sketchpad\" is borrowed from the \"Working Memory Model\"[1]. We use it to refer to LLMs' ability of retaining visuospatial information within its context, rather than an external tool added to LLMs.\n2. All the input for LLMs are text only, the map/rectangle input in the tasks is 2D grid comprising of special characters (**line 48**). For example, 🏠, 🚧 and 🟦 are emoji icons in the input that LLMs can understand, and 🚶is generated by LLMs in the output. For GPT-4V, we use Pilmoji library to draw special characters into an image. We discussed this implementation in **appendix A**.\n3. Visualizations are also text-form (**line 130**).\n4. We apologize for any confusion caused by Figure 4. It's indended to provides the overview of three tasks without covering all details. It illustrates the input format, VoT prompt and the visualizations generated by LLMs. We'll add descriptions to the caption of Figure 4. Regarding comprehensive details about inputs and responses, they are listed in appendix B.1.\n5. Figure 8 provides side-by-side comparisons to show the how response is influenced by prompt, where bold texts on the top are prompts to compare.\n6. We regret the brevity of Figure 10's caption. Figure 10 is to illustrate diverse visualization formats in LLMs to track the navigation progress, including marking the path, marking the current location, and directional/temporal steps. We will include these descriptions in the figure caption and note that detailed descriptions are available in **Appendix E.1**.\n7. As specified in **section 4.1 (line 152)**, model version of GPT-4 is 1106-preview, and GPT-4V is vision-preview.\n8. In the natural language navigation task, LLMs often output additional words in the extracted answer besides the expected object name. For example, \"Answer: You will find ...\". We adopted the substring matching method as previous work did. For visual navigation and visual tiling task, we adopted exact matching. Since sub-string matching can encompass exact matching, we refer to f<sub>correct</sub> implementation using this term for simplicity.\n\n> **Weakness 2: Scaling**\n\nWe appreciate the insightful question regarding the scalability of the tasks. Our testbed indeed provide flexible difficulty control across different tasks. For visual tiling task, the difficulty is controlled by the number of empty squares. As the number increases, the more spatial arrangements LLMs need to consider. Regarding the visual navigation task, as illustrated in figure 2, we use the number of roads to control difficulty, which is directly correlated to map size. Data distribution is provided in table 4 and 5. \nSpecifically, in the next-step-prediction task, the difficulty is also controlled by the number of navigation instructions. For example, a 7 * 7 map with 6 instructions is significantly more complex than a 3 * 3 map with 1 instruction. The distribution of map size is as follows:\n\n| Height \\ Width | 3   | 5   | 7   | 9   |\n|----------------|-----|-----|-----|-----|\n| 3 | 24  | 60  | 116 | 48  |\n| 5 | 60  | 272 | 652 | 120 |\n| 7 | 116 | 652 | 184 | 24  |\n| 9 | 48  | 120 | 24  | N/A    |\n\nThe performance of LLMs across these varying difficulty levels is shown in Figure 9. We observe that for advanced LLMs, VoT is more robust to various difficulties compared to CoT. To clearly illustrate the performance gap between VoT and CoT in advanced LLMs across varying difficulty levels, we provide the following comparative table:\n\n| Model       | K-step Map | Map Count | CoT Accuracy (%) | VoT Accuracy (%) |\n|-------------|------------|-----------|------------------|------------------|\n| **GPT-4**   | 2 | 8 | 75.00 | 75.00 |\n| | 3 | 32 | 68.75 | 62.50 |\n|| 4 | 96 | 60.42 | **68.75** |\n|| 5 | 256 | 50.78 | **64.06** |\n|| 6 | 640 | 52.34 | **55.16** |\n|| 7 | 1488 | 45.30 | **52.69** |\n| **LLama3-70B** | 2 | 8 | 62.50 | 62.50 |\n|| 3 | 32 | 68.75 | 65.63 |\n|| 4 | 96 | 60.42 | **62.50** |\n|| 5 | 256 | 56.25 | **57.42** |\n|| 6 | 640 | 48.59 | **54.84** |\n|| 7 | 1488 | 46.71 | **52.35** |\n\nBold numbers in this table demonstrate the significant advantage of VoT particularly at higher difficulty levels. It indicates VoT's superior robustness in handling more complex spatial reasoning tasks.\n> **Weakness 3: Anthropomorphism**\n\nWe apologize for some unnecessary anthropomorphism. We used *mental image* and *mind's eye* as metaphors to help readers intuitively understand the described mechanisms, without introducing unfamiliar terminology. We'll revise unnecessary usage which hinders scientific rigors.\n> **Question**\n\nBoth the moving house and person icon are among the 30 symbols LLMs automatically generate to track the navigation progress, as described in the appendix E. In figure 8b, since the direction of each instruction is correct, the left example is a correct solution with imperfect intermediate visualization. Due to the limitation of this inherent visualization ability, even for the most advanced models, it's still challenging to generate a perfect visualization. We provide analysis about the accuracy of the final visualization in Table 2, which are lower than 30% in both visual tasks.\n\nWe hope these clarifications will address the concerns raised and provide a clearer understanding of our work.\n## References ##\n[1] Alan Baddeley. Working memory. Science, 255(5044):556–559, 1992", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FswbJjCaiu", "reviewer": "Reviewer_Af54", "summary": "The paper explores the spatial reasoning capabilities of LLMs. To this end, the paper introduces novel testbeds including multiple tasks and, importantly, a novel approach called Visualization-of-Thought (VoT). This method aims to enhance the spatial reasoning capabilities of LLMs by visualizing their reasoning processes, also inspired by chain of thought CoT reasoning and human cognitive abilities. \nEvaluating on the mentioned testbed the authors show that VoT significantly improves the spatial reasoning performance of LLMs, outperforming e.g. CoT reasoning.", "questions": "- How might VoT be adapted or scaled to work with 3D spatial reasoning tasks or other modalities?", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The introduction of Visualization-of-Thought (VoT) represents a novel problem and significant advancement in eliciting spatial reasoning in LLMs, a relatively unexplored area. \n- The paper is well-structured, balancing necessary details in the main text with extensive supplementary information in the appendix. The problem, tasks as well as the VoT approach are well motivated and described.\n- The paper presents well-conducted and detailed experiments across multiple tasks, providing robust evidence of VoT’s effectiveness. Further, the experimental setup is well described. Additionally, the consideration of openly available models (LLaMA), enhances the study’s reproducibility.  \n- The authors provide a thorough discussion of the results and the limitations providing valuable insights for future work.", "weaknesses": "While novel, the introduced and investigated tasks to evaluate spatial reasoning capabilities of LLMs may still be relatively simple compared to real-world spatial reasoning challenges. However, this is negligible. The paper’s contribution provides significant novelty and lays a robust foundation for future exploration and advances of spatial reasoning in large language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, and we appreciate your support!\n\nSpatial reasoning in LLMs is a less explored research topic, we only scratch the surface of it. Despite this testbed is relatively simple to humans and real-world spatial reasoning challenges, it's still challenging for LLMs. It covers various aspects of spatial reasoning, and offers various difficulty levels to appropriately evaluate LLMs.\n> **Question: 3D spatial reasoning tasks and other modalities**\n\n As LLMs' inherent ability to generate text-form visualizations develops, VoT can be applied to more and more tasks, including 3D tasks. For example, folding a net into a 3D cube, and identifying different perspectives of a 3D cube.\n Moreover, making VoT work with multi-modal LLMs is one of our primary future works. Besides text-form visualizations, VoT can be adapted to generate visual modality by code generation (indirect) or image generation (direct). The code generation approach might be applicable for proprietary advanced MLLMs at current stage.\n We'll explore the approach to directly generate images as visualization to guide subsequent reasoning. For open-source MLLMs, they can be adapted to multimodal generation to gain the visualization ability. This will also benefit proprietary MLLMs for two reason. First, they can leverage external visualizers to ground their reasoning and build more powerful applications. Second, the generated multimodal reasoning trajectory is appropriate for data augmentation, which will benefit interested researchers in the community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After the author rebuttal and the reviewer discussions, the paper received 2 borderline accepts, 1 weak accept, and 1 strong accept recommendations. The borderline reviewers mention that the author rebuttal clarified their concerns. The concerns included lacking technical details and additional experiments to confirm that the model scales.

The AC does not see a particular reason to turn down the opinions of the reviewers.
Decision: Accept (poster)