Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Authors: wenshan  wu
shaoguang  mao
yadong  zhang
yan  xia
li  dong
lei  cui
furu  wei
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 06:33:42

Abstract: Large language models (LLMs) have exhibited impressive performance in language comprehension and various reasoning tasks. However, their abilities in spatial reasoning, a crucial aspect of human cognition, remain relatively unexplored. Human possess a remarkable ability to create mental images of unseen objects and actions through a process known as the Mind's Eye, enabling the imagination of the unseen world. Inspired by this cognitive capacity, we propose Visualization-of-Thought (VoT) prompting. VoT aims to elicit spatial reasoning of LLMs by visualizing their reasoning traces, thereby guiding subsequent reasoning steps. We employed VoT for multi-hop spatial reasoning tasks, including natural language navigation, visual navigation, and visual tiling in 2D grid worlds. Experimental results demonstrated that VoT significantly enhances the spatial reasoning abilities of LLMs. Notably, VoT outperformed existing multimodal large language models (MLLMs) in these tasks. While VoT works surprisingly well on LLMs, the ability to generate mental images to facilitate spatial reasoning resembles the mind's eye process, suggesting its potential viability in MLLMs. Please find the dataset and codes in our [project page](https://microsoft.github.io/visualization-of-thought).

Main paper: 
 introduction: 
 Recently, large language models (LLMs) [BCE + 23, BMR + 20, TLI + 23, JSM + 23] have achieved remarkable performance on various language-related tasks. However, despite their success in math reasoning [KGR + 23], common sense reasoning [LKH + 22], and other reasoning tasks such as symbolic reasoning or logic reasoning [KGR + 23], their abilities in spatial reasoning still remain underexplored [RFD + 21, YBL + 23, MHV + 24]. Spatial reasoning is an essential function of human cognition, allowing us to interact with the environment. It facilitates tasks that require understanding and reasoning about the spatial relationships between objects and their motions. The spatial reasoning of language models largely relies on language to reason about spatial information, whereas human cognitive capabilities extend far beyond verbal reasoning. Humans can not only create task-relevant abstract representations from visual perception [BK18, KC22] , but also imagine unseen scenes through their mind's eye. It remains a research topic called mental image [She78] in domains of neuroscience, philosophy of mind, and cognitive science. Building upon this cognitive function, humans facilitate spatial reasoning by mental image manipulation, such as navigation [Tol48] , mental rotation [SM71] , mental paper folding [SF72] , and mental simulation [MK09] . Figure 1 illustrates the human process involved in a navigation task. Humans enhance their spatial awareness and inform their decisions by creating mental images of a route, utilizing various sensory inputs such as navigation instructions or a map image. Subsequently, they simulate route planning through the mind's eye.Inspired by this cognitive mechanism, we conjecture that LLMs possess the ability to create and manipulate mental images in the mind's eye for spatial reasoning. As illustrated in Figure 1 , LLMs could potentially process and understand spatial information in various formats. They might be capable of visualizing internal states and manipulating these mental images through their mind's eye, thereby guiding subsequent reasoning steps to enhance spatial reasoning. Therefore, we propose the Visualization-of-Thought (VoT) prompting to elicit this ability. This method leverage LLMs to visualize their reasoning steps and inform subsequent steps, implementing the concept of visuospatial sketchpad [Bad92] . VoT adopts zero-shot prompting instead of relying on few-shot demonstrations or text-to-image visualization with CLIP [RKH + 21]. This choice stems from LLMs' ability to acquire various mental images from text-based visual art [SB14, SMM21, Reg19] .To evaluate the effectiveness of VoT in spatial reasoning, we selected three tasks that require spatial awareness in LLMs, including natural-language navigation [YBL + 23], visual navigation, and visual tiling. These tasks require an understanding of space, direction, and geometric shape reasoning. To emulate human-like multisensory perception, we designed 2D grid worlds using special characters as enriched input formats for the LLMs in visual navigation and visual tiling tasks. We compared different models (GPT-4, GPT-4V) and prompting techniques across these three tasks. The findings reveal that the VoT prompting proposed in this paper consistently induces LLMs to visualize their reasoning steps and inform subsequent steps. Consequently, this approach achieved significant performance improvements on the corresponding tasks.The main contributions of this paper include:1. We shed light on LLMs' mental image for spatial reasoning from a cognitive perspective, conducting quantitative and qualitative analyses on the mind's eye of LLMs and its limitations. We also explore cues about the origin of this generalized ability from code pre-training.2. We develop two tasks of "visual navigation" and "visual tiling", along with corresponding synthetic datasets, emulating various sensory inputs for LLMs. These tasks are structured to support varying levels of difficulty, offering a well-designed testbed for the research on spatial reasoning.3. We propose Visualization-of-Thought (VoT) prompting to elicit the mind's eye of LLMs for spatial reasoning and provide empirical evaluations on three tasks. Experiment results prove the effectiveness of VoT prompting compared with other prompting methods and existing MLLMs. This ability to generate mental images to facilitate spatial reasoning resembles the mind's eye process, suggesting its potential viability in MLLMs. 
spatial reasoning: 
 Spatial reasoning refers to the ability to comprehend and reason about the spatial relationships among objects, their movements, and interactions with the environment. This skill is vital for a wide range of real-world applications such as navigation, robotics, and autonomous driving. These fields necessitate action planning based on visual perception and a concrete understanding of spatial dimensions.Although several tasks and datasets [WBC + 15, SZL22, MK22, LB18, RAB + 20] have been developed to probe the spatial semantics embedded in text, existing research efforts often focus on how spatial terms are linguistically structured. Recently, significant achievements and impressive results have been achieved in these benchmarks by converting spatial terms to logical forms through LLMs and adopting logic programming [YIL23] . This implies that excelling in these tasks does not necessarily equate to a genuine understanding of spatial information by LLMs, nor does it provide an accurate measure of their spatial awareness. Spatial awareness involves understanding spatial relationships, directions, distances, and geometric shapes, all of which are essential for action planning in the physical world. To evaluate the spatial awareness and spatial reasoning abilities of LLMs, we have selected tasks that test navigation and geometric reasoning skills, including natural language navigation, visual navigation and visual tiling.Natural language navigation task [YBL + 23] was inspired by prior research on human cognition [GDB17] presenting participants with sequential transitions sampled from a graph structure.In this context, a square map is defined by a sequence of random walk instructions and associated objects at each location, denoted as W = {(l 1 , o 1 ), (l 2 , o 2 ), . . . , (l n , o n )}. Given a square map W , and sequence of navigation instructions I = {i 1 , . . . , i k }, the task for the model is to identify the associated object o ∈ W at the specified location l which is determined by the navigation instructions, as detailed in Equation 1 and exemplified in Appendix B.2.EQUATIONVisual navigation task presents a synthetic 2D grid world to LLM, challenging it to navigate using visual cues. The model must generate navigation instructions to move in four directions (left, right, up, down) to reach the destination from the starting point while avoiding obstacles. This involves two sub-tasks: route planning and next step prediction, requiring multi-hop spatial reasoning, while the former is more complex. Task instructions are available in Figure 6 in appendix.Formulation The model is presented with a grid map M consisting of k consecutive edges E = {e(s 0 , s 1 ), e(s 1 , s 2 ), • • • , e(s k-1 , s k )}, where the starting point and destination are s 0 and s k respectively, as shown in Figure 2 . Route planning task is to generate a sequence of correct directionsD = {d(s 0 , s 1 ), d(s 1 , s 2 ), • • • , d(s k-1 , s k )},as defined in Equation 2. Given M and t navigation instructions D t,0<t<k = {d(s 0 , s 1 ), • • • , d(s t-1 , s t )}, next step prediction task is to identify the correct direction d(s t , s t+1 ) of the next step, as defined in Equation 3. D ∼ p({d(s 0 , s 1 ), d(s 1 , s 2 ), • • • , d(s k-1 , s k )} | M ) (2) (a) k=2 (b) k=3 (c) k=4 (d) k=5 (e) k=6 (f) k=7EQUATIONImplementation The navigation map's underlying graph is semi-Eulerian, alternating between horizontal and vertical edges, with 2 k+1 possible spatial configurations for a k-hop navigation map. For each map and set of k navigation instructions, k -1 question-and-answer (QA) instances,i.e. "what is the next step?" are created. Further implementation details are in Appendix A.1.Introduced by [Gol66] , polyomino tiling is a classic spatial reasoning challenge. We extend this concept to test the LLM's ability to comprehend, organize, and reason with shapes in a confined area, thus enhancing the evaluation of spatial reasoning skills. As depicted in Figure 3 , the task involves a rectangle with unfilled cells and various polyomino pieces, like the I-tetromino made of four aligned squares. The model must select the appropriate polyomino variant, such as choosing the orientation for the I-tetromino, to solve the QA puzzle. Task instructions are provided in Figure 7 in appendix.Formulation The model is presented with a rectangle R masked with k unique polyominoesM P = {mp 1 , • • • , mp k }, 2 corresponding variants of each polyomino v i<=k = {v i1 , v i2}, and a polyomino query q ∈ M P . Visual tiling task is to identify the correct variant of q, as defined in Equation 4. 3 Visualization-of-Thought PromptingEQUATIONConsidering the way humans process spatial information during tasks like navigation, it's common to create mental images , such as maps, to enhance spatial awareness or simulating movements to inform decision-making. Our objective is to elicit the spatial awareness of LLMs and ground their reasoning by visualizing the consequence of their intermediate reasoning steps.We introduce Visualization-of-Thought (VoT) prompting: "Visualize the state after each reasoning step." This new paradigm for spatial reasoning aims to generate reasoning traces and visualizations in an interleaved manner. Qualitative results of this approach are presented in Figure 4 .We use p θ to denote a pre-trained LM with parameters θ, x, y, z to denote a language sequence, and v to denote a visualization sequence in text form. In a multi-hop spatial reasoning task with input x, CoT prompting generates a series of intermediate stepsz 1 , • • • , z n , each step z i ∼ p θ (z i | x, z 1•••i-1) is sampled sequentially, followed by the output y ∼ p θ (y|x, z 1•••n ). As shown in Figure 1 , VoT prompting enhances this process by adding a visuospatial sketchpad to each intermediate step z i , then the subsequent step z i+1 is sampled conditioned on prior stepsz 1•••i and visualizations v 1•••i .As defined in the Equation 5 and 6, it forms interleaved reasoning traces and visualizations. A qualitative comparison between outputs of VoT and CoT is provided in Figure 8a in appendix.EQUATIONStarting from , provide the steps to navigate to .Provided: I T L To fit all the provided polyominoes into the empty squares, what's the correct variation of Tetromino T?Visualize the state after each reasoning step.Visual Tiling Natural Language NavigationYou have been given a 3 by 3 square grid. Initially, you are at the bottom-left corner…find a cassette player…go right…a wool, go right…a conch, go up…a moving van, go left…a confectionery store, go left…a pot pie, go up…a siamang, go right…a black-and-white colobus, go right…a minivan. Now you have all the information on the map. You start at where the cassette player is located, then you go right by one step, go right…go up…go left…go left…go up…go right…go down by one step. What will you find?Visualize the state after each reasoning step. Visualize the state after each reasoning step. EQUATIONThis reasoning paradigm enables LLMs with visual state tracking. We introduce the concept of a state, denoted ass i = [x, z 1•••i , v 1•••i-1] representing a partial solution at step i with the input, the sequence of intermediate steps z 1•••i and the sequence of visualizationsEQUATIONAs shown in Equation 7, visual state tracking is implemented by generating the visualization v i as representation of the internal state s i after each reasoning step z i (e.g. v i could be a grid of the navigation map marked with path or a filled rectangle). Grounded by the visual state tracking sequence, the subsequent state is derived by s i+1 ∼ p θ (s i+1 | prompt V oT , x, s i , v i ).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We confirm the limitations of the work has been fully discussed, which could be find in the Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? The tasks developed include navigation and geometric shape reasoning, which is very unlikely to cause negative societal impacts. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
dataset: 
 Natural Language Navigation We generate 200 square maps of size 3x3 which is described by 9 landmarks in snake order traversal, and a set of navigation instructions.Visual Navigation We generate 496 navigation maps and 2520 QA instances in total, covering various map sizes, up to 7×9 and 9×7. The data distribution is provided in Table 4 in appendix.Visual Tiling We first generate multiple unique configurations to fill a 5 x 4 rectangle with 5 polyomino pieces including two I tetrominoes, two T tetrominoes and one L tetromino. Then we randomly masked two or three pieces of different types and generate QA instance for each masked pieces. The total number of QA instances is 796, and we show dataset details in Table 5 in appendix.We extract the answer from model output by pattern matching. For tasks except for route planning, we calculate accuracy by Equation 8. We adopted sub-string matching † as f correct to determine correctness.EQUATIONFor the route planning task which predicts a sequence of navigation instructions, we reject any sequences exceeding 100 instructions, considering them to be random guesses. We then normalize the navigation instructions by executing each navigation instruction. Those instructions which violate navigation rules will be ignored. The length t of normalized instruction sequence is considered as the temporal distance against the starting point. Given the ground-truth of k navigation instructions, the completing rate of route planning is t/k. For the dataset of n maps, we report two metrics including:1. Average completing rate:n i t i /k i /n.Average completing rate among all instruction sequences, reflecting LLM's effectiveness of route planning. 2. Success rate: n i (t i == k i )/n. This metric represents the proportion of instruction sequences with t = k, i.e., reaching the destination.As illustrated in Table 1 , GPT-4 VoT significantly outperforms other settings in all tasks across all metrics. The significant gap when comparing GPT-4 VoT with GPT-4V CoT and GPT-4 w/o Viz demonstrates that effectiveness of visual state tracking, which allows LLMs visually interpret their actions within an grounded world. And in the natural language navigation task, GPT-4 VoT outperforms GPT-4 w/o Viz by 23.5%. In the visual tasks, the noticeable performance gap between GPT-4 CoT and GPT-4V CoT indicates that LLM grounded with 2D grid could possibly outperform a MLLM in challenging spatial reasoning tasks.On the other hand, performance of GPT-4 VoT is still far from perfect in all tasks, especially in the most challenging route planning task. Despite these tasks are relatively easy for humans, performance of LLMs drops significantly as task difficulty increases. Details on performance trends across difficulty levels are provided in figure 9 1 : Performance of different GPT-4/4V settings in all tasks. Underline denotes statistical significance with p < 0.05 when comparing GPT-4 VoT against all baselines using two-sample z-test, while p < 0.16 is observed compared with GPT-4 CoT in natural language navigation task. 
analysis: 
 As explained in section 3, one of the core aspects of VoT lies in enabling LLMs with visual state tracking. During the experiments, it was observed that GPT-4 CoT occasionally exhibited this reasoning pattern across several tasks with exception of route planning. Besides, incorrect visualizations of VoT are commonly observed in model outputs. In this section, our analysis of VoT primarily focuses on three questions: (1) Do visual state tracking behaviors differ among prompting methods? (2) How visualizations enhance final answers? (3) Can VoT benefit less powerful language models?For each model output, we extract the sequence of visualizations sampled prior to generating the final answer and discard any visualizations generated thereafter. Then we compare the sequence length l v with the number of reasoning steps l s . We calculate Complete Tracking n i (l v == l s )/n when a visualization v i corresponds to each state s i . Similarly, we calculate the Partial Tracking metric as n i (l v > 0)/n when at least one visualization is present before the final answer is generated. Figure 5 shows the significant differences between these settings. In the GPT-4 CoT setting, it demonstrated noticeable tracking rate across almost all tasks except route planning. This observation implies that LLMs inherently exhibit the capability of visual state tracking when spatiotemporal simulation is integral to reasoning.On the other hand, the visual state tracking behavior is sensitive to prompts to varying degrees. As showcased in Figure 8 in appendix, after removing "reasoning" from the prompt of VoT, the visualizations are sampled after GPT-4 generates the wrong answer. Consequently, explicitly prompting LLMs to visualize their reasoning traces with VoT markedly improves the visual tracking rate, thereby enhancing overall performance. The potential contribution of code pre-training to this emergent capability is further explored in Appendix C. Ideally, VoT is supposed to generate an accurate visualization v i at each step, so that subsequent step z i+1 could be determined correctly. This relies on the spatial visualization and spatial understanding capability of LLMs. To evaluate these capabilities of LLMs in these tasks, we extract the final visualization from each model output under the setting GPT-4 VoT in visual navigation and polyomino tiling task. Specifically, for visual navigation task, we extract the visualized map where LLM completed all navigation instructions. For polyomino tiling, we extract the rectangle filled with corresponding polyomino piece. The spatial visualization capability is measured by two criteria:(1) Compliance, indicating whether the manipulation of mental image satisfies requirements such as avoiding overlap and navigating around obstacles. (2) Accuracy, indicating whether the mental image aligns with the corresponding state. The spatial understanding capability is measured by the proportion of correct answers when the corresponding visualization is generated accurately.As could be seen from To evaluate the efficacy of VoT on less powerful language models, we conducted experiments across various model families [BMR + 20, OA + 23, TLI + 23] and model sizes, including GPT-3.5 turbo, LLAMA3-8B-Instruct and LLAMA3-70B-Instruct. We access GPT-3.5 via Azure OpenAI API with model version 1106-preview and apply greedy decoding to all models.As shown in Table 3 , within the same model family, performance improves across all tasks with increases in model size. LLAMA3-70B VoT significantly outperforms the baseline across all tasks except for visual tiling, where it aligns closely with results observed in GPT-4. This consistency suggests that VoT offers a scaling advantage when applied to more advanced models, markedly enhancing performance in larger models. In contrast, less capable models tend to rely on random guessing, especially in spatial reasoning tasks. For instance, in the route planning task, GPT-3.5CoT often resorts to speculative responses, random guessing in nearly half of the instances, which leads to exhaustion of output tokens. While GPT-3.5 VoT effectively minimizes random guesses, such occurrences become increasingly rare with GPT-4 CoT as the model size expands. On the other hand, the reliance on random guessing introduces unpredictability in performance trends for less powerful models.  
conclusion: 
 This study introduces Visualization-of-Thought Prompting (VoT), inspired by the human cognitive function of visualizing and manipulating mental images through the mind's eye. We have demonstrated that VoT enables LLMs to exhibit the mechanism of "the mind's eye", as evidenced by their performance in multi-hop spatial reasoning tasks and our comprehensive analysis of the reasoning traces. Remarkably, VoT enable LLMs to outperform state-of-the-art multimodal large language models (MLLMs) in the tested visual tasks. While VoT demonstrates impressive efficacy in LLMs, this emergent capability to create mental images to enhance spatial reasoning resembles the mind's eye process, suggesting its promise in MLLMs.Building on the success of experiments with GPT-4, we plan to investigate how VoT can futher elicit "the mind's eye" in MLLMs to enhance their spatial awareness. Additionally, our future efforts will focus on automatic data augmentation from real-world scenarios, aiming to identify effective methods for learning generalized internal representations of mental images. This will further improve the mind's eye of LLMs, ultimately contributing to the advancement of their cognitive and reasoning abilities. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All external assets, including models and algorithms utilized in this paper, are duly cited, ensuring proper credit to the original creators. Furthermore, we have meticulously adhered to the specified licensing agreements and terms of use associated with these assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This work only scratches the surface of spatial reasoning of LLMs. Both mental images and visual state tracking rely on the emergent ability of advanced LLMs. Therefore, it might cause performance deterioration in less advanced language models or more challenging tasks. Besides, due to the limited data exposure and a lack of explicit instruction tuning, visual state tracking of current LLMs are sensitive to prompts. For example, when explicitly prompted with "use ascii-art", the tracking rate will significantly increase thereby boosting performance, while removing "reasoning" from the VoT prompt will cause a decrease of tracking rate. Moreover, the mental images tested in our work are limited to 2D grid. To strength the mind's eye of LLMs, more diverse and complicated representation should be explored in the future, such as complex geometric shapes and even 3D semantics shown in Figure 11 in appendix.A Synthetic DataAs depicted in 1, given a specific k, the process of generating a 2D navigation map is composed of 3 steps, which are instruction generation, instruction simulation, map rendering. In instruction generation step, we enumerate all possible instruction sets navigating from the starting point to the destination (e.g move up, then move right). During this step, only the direction of each instruction is determined, while the moving distance is undetermined until next step. In instruction simulation step, simulation is applied in the 2D coordinate system with origin (0, 0) as the starting point. To guarantee an unique answer in each navigation map, the moving distance of each instruction is dynamically calculated to avoid overlapping. Each time when an overlapping is detected, the moving distance of previous instruction will be increased by 1 unit recursively until overlapping is resolved. As the distance is determined, those corresponding points are added to the navigating path. After all instructions are completed, the final point is marked as the destination. In the map rendering step, the bounding box of those points is adopted and normalized to a 2D square grid. The starting point and destination are marked with dedicated squares, and cells along the path are marked by empty squares, while other untouched cells are marked by obstacle squares.Algorithm C solution ← C solution ∪ s i C textual_map ← C textual_map ∪ t i C visual_map ← C visual_map ∪ v i endSince the direction of each navigation instruction is alternating, there are 4 * 2 k-1 = 2 k+1 kinds of spatial configurations for a k-hop navigation map. During the implementation, we simplify the recursive implementation with an early quit when path overlapping could not be resolved within one iteration, the main consideration of which is the size of the map. So the number of generated map is slightly lower than 2 k+1 as the navigating step k increases.The data generation process comprises 3 stages, including configuration generation, question generation and polyomino rendering. In the configuration generation stage, to generate valid spatial configurations of a rectangle and the corresponding polyomino set, we convert a tiling problem to existing formalized problems. One of the problems is an exact cover problem leveraging dancing link algorithm [Knu00] , which could be described as: given a matrix of 0s and 1s, find a set of rows containing exactly one 1 in each column. The conversion is to construct a matrix of 0s and 1s, each row of which represents a possible arrangement of placing a specific polyomino in a rectangle. As illustrated in Equation 9, given k polyomino pieces, and a rectangle of n units to be filled, the first k columns compose an one-hot vector indicating the corresponding polyomino, and the last n columns are marked with 0 or 1 depending on whether the corresponding unit is filled by that polyomino. Then finding a set of polyomino arrangements in a rectangle equals to find a set of rows containing exactly one 1 in each column. Another adaptable problem is the boolean satisfiability problem (commonly known as SAT), for which efficient solvers exist [ES03, GN07] . A tiling problem can be converted to SAT by introducing a boolean variable for each possible arrangement of each piece, and then adding clauses comprising of those boolean variables that ensure at least one arrangement of each piece is achieved, while avoiding conflicts between arrangements of one piece or two different pieces.Given the size of a rectangle and polyominoes to be fit, multiple corresponding solutions are generated by applying those algorithms. Then in the question generation stage, we randomly mask several polyomino pieces in the rectangle, and generate a question answer(QA) pair for each masked polyomino. Finally the rectangle and each polyomino piece are rendered with emoji squares.EQUATIONAfter gathering the textual dataset of 2D square grid, we generate the corresponding visual dataset by drawing text onto an image. Specifically we adopt color emojis for a fair comparison as they're more visual friendly to a multimodal model.Data distribution among various difficulty levels for visual navigation tasks and visual tiling tasks are provided in Table 4 and 5 . It provides flexible difficulty control across different tasks. For visual tiling task, the difficulty is controlled by the number of masked polyomino pieces. As the number increases, the more spatial arrangements LLMs need to consider. Regarding the visual navigation task, as illustrated in figure2, we use the number of roads k to control difficulty, which is corresponding to the size of the map. For visual navigation and visual tiling tasks, the structured input template is comprised of task instruction, input parameters and prompt of specific setting.Task instructions and responses of each visual task under setting GPT-4 VoT are provided as following:• Route Planning Task instruction in Figure 6 , response in Figure 12 .where you will find a baseball player, then you go right, where you will find a harvestman, then you go right, where you will find a neck brace. Now you have all the information on the map. You start at the position where the torch is located, then you go right by one step, then you go right by one step, then you go up by one step, then you go up by one step, then you go left by one step, then you go down by one step, and then you go down by one step. What will you find?Response Example See Figure 15 .As for where this emergent ability stems from, it might derive from tabular data, city grid navigation, maze exploration related coding problems [YBL + 23]. These tasks involves understanding and manipulating objects in a 2D square grid. Besides, we conjecture the exposure of ascii-art comments [Reg19] during LLMs' code pre-training possibly enhances this generalized ability. As a fact to support this conjecture, the visual tiling task is different from navigation tasks because it requires shape understanding and spatial manipulation ability. While tabular data and square grid navigation data boost row-wise or column-wise attention, ascii-art supplements intricate spatial attention to understand and manipulate 2D shapes. Additionally, ascii-art in code comments is presented in various formats, one of which is interleaved ascii diagrams, natural language and programming language. It require LLMs to generate the interleaved mental images and text sequence, thereby enhancing spatial visualization ability and spatiotemporal causality. Interestingly in the natural language navigation task, when GPT-4 is prompted with "use ascii-art to visualize", the complete tracking rate increases to 98.5% (+78.5%), boosting task performance to 62.5% (+3.5%).Ascii-art is commonly used in code comments to represent data structure, diagram, geometry and so on, which could benefit LLMs' spatial understanding and visualization capability. Besides, it's also used to illustrate how an algorithm works or simulate an operation, where reasoning traces and corresponding visualization are presented in an interleaved manner. Below are several examples in open-source projects.• Spatial Causality:Double-ended queue in Rust, Scrolling web pages and tree rotation present triplets of previous visual state, instruction, and updated state of instruction following. • Temporal Causality: Undo systems from emacs provides various temporal states of the undo system when undo operation happens in different timelines and corresponding visualizations in an interleaved manner. Each visualization reflects the temporal casuality of the system state.This kind of interleaved sequence tracks the system state over time, thus reflecting spatiotemporal casuality.In this analysis, we examine performance trends across varying difficulty levels in the next-step prediction task for models utilizing either CoT or VoT methods. These trends are crucial for understanding the inherent unpredictability associated with random guessing. As k increases from 2 to 7 in a k-step navigation map, distinct performance patterns emerge among different models, as depicted in Figure 9 . Larger language models such as GPT-4 and LLAMA3-70B demonstrate a more predictable decrease in accuracy with increasing k. This trend indicates a robust ability to handle progressively challenging tasks, despite the overall decrease in performance. Detailed statistics are provided in table 6. In contrast, less powerful models like GPT-3.5 and LLAMA3-8B exhibit an irregular performance trajectory. These models show variable accuracy, with significant fluctuations at higher difficulty levels, suggesting a reliance on random guessing, particularly under conditions of increased task difficulty. This behavior highlights their limitations in sustaining reliable reasoning processes through more complex scenarios. Furthermore, the VoT method seems to offer a modest improvement in performance for the less powerful models, particularly in scenarios of lower difficulty.This observation suggests that VoT might be advantageous for enhancing reliable reasoning in simpler spatial reasoning tasks, potentially compensating for the inherent weaknesses of smaller language models. We consider visual state tracking similar to spatiotemporal simulation. During the simulation in those tasks, we discovered several interesting behaviors of LLM.1. Diverse visualization formats for state tracking: Nearly 30 different symbols found in the navigation tasks to track the navigation progress, including marking the path, marking the current location. Among those diverse representations, LLM succeeded in some challenging cases where it used directional arrow emojis to indicate both the location and moving direction at each step. More examples could be found in Appendix E.1.Inconsistency between language and visualization: This is commonly observed across all tasks. Due to the limited visualization capability, sometimes LLM generates accurate language instruction but inaccurate visualization. And in other cases, LLM generates wrong answers even the visualization is generated correctly, which reflects its limitation of spatial understanding as discussed in previous section. More examples could be found in Appendix E.2. Now, let's unfold the front face (1, 2, 6, 5) downwards:``` 0 3 *-------* / /| *-------* | 4| | |7 | | * | |/ *-------* | | | | *-------* 1 2 ```Here, the face with vertices 1, 2, 6, and 5 has been unfolded downwards. The vertical lines connecting the top face to the front face have now become horizontal, indicating that the front face is lying flat. The cube is now represented as a two-dimensional net, with the front face unfolded. Keep in mind that this is a simplified representation and that in an actual net, the faces would be connected at the edges without the gaps shown in the ASCII art.The ASCII art you've provided represents a three-dimensional cube with labeled vertices from 0 to 7. To unfold the front face (which appears to be the face connecting vertices 1, 2, 6, and 5), we would lay it flat. Assuming that the front face is the one with vertices 1, 2, 6, and 5, and we unfold it downwards, the net might look something like this: ``` We have two variations for Tetromino I ( ), which can either be placed vertically or horizontally. Given the configuration of the empty spaces, the only way to place the Tetromino I is horizontally at the top row, as the vertical placement would not fit anywhere.0 3 *-------* /| /| *-------* | 4| | | |7 | *-----|-* |/ |/ *-------* 1 2 ``` Ascii cube: ``Ù nfold the front face 0 3 *-------* 1 / | 2 /| *-------* | | |4 | |7 | *-----|-* |/ | / *-------* 5 6After placing Tetromino I ( ), the target rectangle looks like this:``` ``` Next, we have two variations for Tetromino L ( ). We can see that Variation 3 of Tetromino L can fit in the bottom left corner, while Variation 7 would not fit anywhere without overlapping or going outside the rectangle.After placing Tetromino L ( ), the target rectangle looks like this:```(a) Response of Visual Tiling -Page 1```Now, we have two variations for Tetromino T ( ). We need to fit this into the remaining empty spaces. Variation 2 of Tetromino T cannot fit because it would require two empty spaces next to each other vertically, which we do not have. However, Variation 3 of Tetromino T fits perfectly into the remaining space.After placing Tetromino T ( ), the target rectangle looks like this:``` ```The correct variation of Tetromino T that fits into the empty squares is: We have two polyominoes to fit into this rectangle: Tetromino L ( ) and Tetromino I ( ). We need to determine which variation of Tetromino L will fit without overlapping any existing polyominoes or going outside the rectangle.EQUATIONLet's consider the variations for Tetromino L: Since neither Variation 1 nor Variation 6 of Tetromino L fits into the target rectangle without overlapping existing polyominoes or going outside the rectangle, the correct answer is:C. Neither However, there seems to be a mistake because the task guarantees that there always exists a solution. Let's re-evaluate the placement of Tetromino L.We can see that the Tetromino L must fit in the top left corner because that's the only place with enough space for its shape. The Tetromino I fits perfectly in the remaining space using Variation 1 (vertical placement). Therefore, the correct variation of Tetromino L that fits into the target rectangle is:A. 1The initial attempt to place Variation 1 of Tetromino L was incorrect because it was not placed in the top left corner. Upon correcting the placement, we find that Variation 1 does indeed fit, and the Tetromino I can be placed afterward to complete the puzzle.(c) Example case of Self-refine -Page 3 Justification: We confirm that the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.† We use this term for simplicity. In natural language navigation tasks, LLMs often output additional words in the extracted answer besides the expected object name. For example, "Answer: You will find ...". In this case, sub-string matching is adopted without affecting the correctness. Otherwise, exact matching is adopted for multiple choice questions in visual tasks.(a) Response of Natural Language Navigation -Page 1(c) Response of Natural Language Navigation -Page 3 Figure 15: Responses of Natural Language Navigation 


Decision: After the author rebuttal and the reviewer discussions, the paper received 2 borderline accepts, 1 weak accept, and 1 strong accept recommendations. The borderline reviewers mention that the author rebuttal clarified their concerns. The concerns included lacking technical details and additional experiments to confirm that the model scales.

The AC does not see a particular reason to turn down the opinions of the reviewers.

Reviews: review: paper_summary: paper_summary: The paper presents a novel approach called Visualization-of-Thought (VoT) prompting, aimed at enhancing the spatial reasoning abilities of Large Language Models (LLMs). Inspired by the human cognitive process known as the “Mind’s Eye,” the authors propose a method where LLMs visualize their reasoning steps to guide subsequent reasoning. The paper evaluates VoT across three spatial reasoning tasks: natural language navigation, visual navigation, and visual tiling. Experimental results demonstrate significant performance improvements with VoT over conventional prompting methods and existing multimodal large language models (MLLMs).
strength_weakness: strength_weakness: Weaknesses

1. **Limited Task Diversity**: While the selected tasks cover various aspects of spatial reasoning, the evaluation could be broadened to include more complex and varied real-world scenarios. In the main paper, we even didnt see more commonly adopted benchmark performance comparison. Include more varied and complex spatial reasoning tasks to better evaluate the generalizability and robustness of VoT.

2. **Generalizability**: The paper primarily focuses on GPT-4 and GPT-4V. It would be beneficial to see how VoT performs across a wider range of models, particularly less advanced ones. It would be great to see more evaluations of VoT on a wider range of models, including less advanced ones, to assess its applicability across different LLMs.

3. **Prompt Sensitivity**: The effectiveness of VoT appears to be sensitive to the specific wording of prompts, which may limit its robustness in practical applications. Investigate ways to reduce the sensitivity of VoT to specific prompt wording, making it more robust for practical use.
questions: questions: See above Weakness part.
limitations: limitations: See above Weakness part.
paper_summary: paper_summary: This paper focuses on enhancing the spatial reasoning capabilities of Large Language Models (LLMs) by introducing Visualization-of-Thought (VoT) prompting. Inspired by the human cognitive ability to visualize unseen objects - a process known as the Mind’s Eye - VoT visualizes reasoning processes to guide LLMs in tasks like natural language navigation and visual navigation in 2D grid worlds.
strength_weakness: strength_weakness: **Method

Some descriptions are unclear.

1. The method for obtaining the visuospatial sketchpad $v$ remains unclear. This paper describes $v$ as representing a visualization sequence in text form, but it does not specify how visual information (maybe images) is converted into text.

2. If visual information is transformed into text form, how does GPT-4V use it? Is it used as a textual representation of the visual image as well?

**Experiment

1. This paper also introduces the concept of state $s$. What is the performance comparison when this state is included versus when it is excluded?

2. In Table 1, for the visual navigation and visual tiling tasks, why does GPT-4V CoT perform the poorest, even yielding results lower than GPT-4 without visual information?

3. It seems that Section 5.2 aims to discuss “how visualization enhance final answers”, yet the connection between the results in Table 2 and this discussion remains unclear. It would be beneficial to include more detailed descriptions or results related to this.

4. In Section 5.3, Table 3 shows that Llama3-8B with VoT underperforms in most results compared to its counterpart with CoT. Does this suggest that the proposed VoT does not benefit less powerful language models?
questions: questions: Please refer to weaknesses.
limitations: limitations: This paper provides the limitations.
paper_summary: paper_summary: The paper proposes a new prompting method, "Visualization of Thought" (VoT) prompting, to enable LLMs to perform better on spatial reasoning tasks.  In VoT, the LLM is prompted with instructions on performing a multi-step spatial reasoning task, followed by the text "Visualize the state after each reasoning step".  The authors evaluate this prompting method on three types of spatial reasoning tasks, and show that it substantially improves performance on these tasks as compared with Chain-of-Thought prompting, as well as with other variations on prompting.
strength_weakness: strength_weakness: There are three categories of weaknesses which I believe can be improved on by the authors.

1. Some aspects of the paper are unclear and need more explanation.  

-- The paper says "This method augments LLMs with a visuospatial sketchpad".  It's unclear what is meant here by visuospatial sketchpad.  This sentence makes it sound like there is a new architectural feature added to the LLM, but my understanding from the rest of the paper is that VoT just asks the LLM to generate a series of "visualizations" in its output.  Please make it clearer in the description what exactly you mean.  

-- Figure 4 gives examples of VoT prompting.  But it wasn't clear to me whether these prompts include actual images (e.g., the pictures of barriers and people in the navigation task, or the pictures of colored and empty boxes in the visual tiling task) or whether these images in the figures were actually translated into some kind of text encoding for the LLM.  Please make clear exactly what is in the prompt---images or text---and if text, what exactly is the text encoding of the pictures you are showing? 

-- Equation 5 defines v_i as "visualizations".  What exactly are these, in terms of LLMs?  Are they text or image outputs from the LLMs?  

-- In several figures (e.g., Figure 4, 6, 7) prompts and responses are shown but it needs to be made very clear which part is the prompt and which part the response from the LLM.  These were not clear enough for me.  

-- In Figure 8, again, what is part of the prompt and what is part of the response?  

-- Figure 10: The caption only says "mental images for state tracking in visual navigation tasks."  Please say more in the caption about what this figure is showing.  Was "Use round pin" part of the prompt, or is this just describing what the LLM did?  What are you trying to show in this figure?  In general, please make the figure captions more informative.  

-- "Specifically, we adopt GPT-4 and GPT-4 Vision" -- there are several versions of each of these LLMs on the Azure platform -- please specify which ones you used for these experiments.  

-- "We adopted sub-string matching as f_correct to determine correctness."
This didn't really explain how you determined correctness.  Please give more details (enough for someone else to replicate) and explain why this sub-string matching method is reliable for assessing correctness. 

2. Scaling:  The tasks used were relatively simple: 3x3 maps for navigation, 5x4 rectangle for tiling.  Did you do any experiments with scaling up the size of these?  How does this method's spatial reasoning abilities scale as the size is increased?  

3. Anthropomorphism:  The paper is filled with anthropomorphic language, which makes it difficult to understand.  For example: 

-- "LLMs possess the ability to create and manipulate mental images in the mind's eye."   Using terms like "mental images" and "the mind's eye" assumes that the LLM is doing something analogous to the brain, which is not necessarily true!  

-- "visual state tracking is implemented by generating the visualization v_i as
a mental image of the internal state si after each reasoning step z".   A mental image is something a human has.  Using that term for LLMs is unhelpful anthropomorphizing. 

-- The paper calls the intermediate outputs of LLMs "thoughts" (e.g., "this process by adding a visuospatial sketchpad to each thought z_i") -- again, an unhelpful and confusing anthropomorphism.  

-- "We demonstrate LLMs’ zero-shot capability of visualizing their precepts at abstract level". Visualizing is a cognitive ability of brains.  Saying that LLMs "visualize" is an incorrect anthropomorphism.  

-- ""this emergent capability to create mental images to enhance spatial reasoning resembles the mind’s eye process, suggesting its promise in MLLMs."  Same comments as above.
questions: questions: -- I was confused by Figure 8b.  The left side shows the house moving -- is that correct?  Also, it doesn't do what the instructions say: e.g., "move down to the end of the continuous road" -- after the move the house only moved down one square, not two.  Is this considered a correct solution?    The right side shows a person moving.  Where did the notion of moving a person versus moving a house come from?
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper explores the spatial reasoning capabilities of LLMs. To this end, the paper introduces novel testbeds including multiple tasks and, importantly, a novel approach called Visualization-of-Thought (VoT). This method aims to enhance the spatial reasoning capabilities of LLMs by visualizing their reasoning processes, also inspired by chain of thought CoT reasoning and human cognitive abilities. 
Evaluating on the mentioned testbed the authors show that VoT significantly improves the spatial reasoning performance of LLMs, outperforming e.g. CoT reasoning.
strength_weakness: strength_weakness: While novel, the introduced and investigated tasks to evaluate spatial reasoning capabilities of LLMs may still be relatively simple compared to real-world spatial reasoning challenges. However, this is negligible. The paper’s contribution provides significant novelty and lays a robust foundation for future exploration and advances of spatial reasoning in large language models.
questions: questions: - How might VoT be adapted or scaled to work with 3D spatial reasoning tasks or other modalities?
limitations: limitations: The authors have adequately addressed the limitations of their work.

Comments: title: Thank you for the rebuttal
comment: Thanks for your detailed response. It has addressed my concerns, I will raise the score to reflect this.
Thank you for the rebuttal. I have no further questions.

After carefully considering the other reviews and the authors’ responses, I would keep my initial score.
We sincerely appreciate your feedback on our rebuttal. We're glad to hear that our responses helped clarify aspects of our methodology and findings. We will certainly incorporate all the clarifications provided in the rebuttal into a revised version of the paper to ensure it is as clear and comprehensive as possible for all readers.

Your comments have been instrumental in improving the quality and clarity of our manuscript. If you have any remaining concerns, please let us know, and we will be more than willing to address them.
Thank you for the thoughtful rebuttal.  This cleared up some aspects of the method and results for me -- I hope you will include all of these clarifications in a revised version of the paper.