Chain-of-Symbol Prompting For Spatial Reasoning in Large Language Models

Authors: hanxu  hu
hongyuan  lu
huajian  zhang
yun-ze  song
wai  lam
yue  zhang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 09:56:09

Abstract: While conventional Chain-of-Thought prompting shows promising performance on various language tasks for LLMs, the spatial scenarios are nearly unexplored. In this paper, we first investigate the performance of LLMs on complex spatial planning and understanding tasks that require LLMs to understand a virtual spatial environment simulated via natural language and act or reason correspondingly in text. By evaluating on classic spatial planning scenarios through natural language descriptions, we found that current popular LLMs still lack abilities to handle spatial relationships in texts. This arises a question -- do the natural language is the best way to represent complex spatial environments for LLMs, or maybe other alternatives such as symbolic representations are both more efficient and effective for LLMs? To this end, we propose a novel method called CoS (Chain-of-Symbol Prompting) that represents the spatial relationships with condensed symbols during the chained intermediate thinking steps. CoS is easy to use and does not need additional training on LLMs. Extensive experiments indicate that CoS clearly surpasses the performance of the Chain-of-Thought (CoT) Prompting described in natural langauge in all three spatial planning tasks and existing spatial QA benchmark, with even fewer tokens used in the inputs compared with CoT. The performance gain is strong, by up to 60.8% accuracy (from 31.8% to 92.6%) on Brick World scenarios for GPT-3.5-Turbo. CoS also reduces the number of tokens in the prompt obviously, by up to 65.8% of the tokens (from 407 to 139) for the intermediate steps from demonstrations on the Brick World task. Interestingly, we also observed emergent ability of abstract symbols understanding when the size of models scales up.

Main paper: 
 introduction: 
 Given a set of target behaviour examples, large language models (LLMs) demonstrate exceptional abilities to accomplish a wide range of tasks, frequently exhibiting performance that surpasses that of humans (Brown et al., 2020; Srivastava et al., 2022) . Specifically, LLMs exhibit impressive sequential textual reasoning ability during inference, resulting in a significant boost in their performance when encountered with reasoning questions described in natural languages (Nye et al., 2021; Wei et al., 2022) . This phenomenon can be clearly observed with a multi-step chain of intermediate thinking procedure, i.e., a "Chain of Thought" (CoT, Wei et al. 2022) .Conventional CoT usually leverages natural languages as intermediate thinking steps in prompting. Although CoT can enhance LLMs' ability in many cases, redundant natural languages and irrelevant information also can hamper the performance of LLMs (Shi et al., 2023) in some cases. For example, spatial languages and descriptions can be hard for language models to understand Mirzaee et al. 2021; Mirzaee & Kordjamshidi 2022 due to Figure 1 : An example for comparison between Chain-of-Thought (CoT) and Chain-of-Symbol (COS) that elicits large language models in tackling complex planning tasks with higher performance and fewer input tokens. We let the model generate CoT/COS during inference in a few-shot manner. Results were taken in May 2023 with ChatGPT and can be subject to change. complex spatial relationships. Aligning symbols and representing spatial relationships by symbols in word sequences can be a neater representation and thus can be potentially easier to understand by LLMs. We thus explore the use of symbols for LLM prompting, which is still an understudied topic. This is important to study which implies understanding abilities beyond language models for language understanding per se.To explore the role of symbolic representations in prompting, we take the complex spatial understanding and planning as the evaluation scenarios, which require LLMs to understand the virtual spatial environments described through natural language as well as planning and achieving certain goals in such environments. Inspired by existing classic planning competitions and spatial reasoning datasets, we present three domains: (i) Brick World (ii) NLVR-based Manipulation and (iii) Natural Language Navigation. Figure 1 illustrates an example for Brick World 1D, and all these three tasks are described in detail in Section 2.1. These three tasks are all described in natural language. And we also evaluate one existing spatial question answering dataset SPARTUN (Mirzaee & Kordjamshidi, 2022) which uses human-generated questions thus closer to realistic situations. For these tasks, LLMs need to understand a virtual environment in natural language, with the spatial relationship between the objects to be operated on and the restrictions on the operation, which is easy for real humans. However, we found that there are still places for improvement in the performance of LLMs on the tasks.As a major contribution to this study, we investigate the symbolic representations for spatial relationships, and propose a novel method called Chain-of-Symbol (COS) prompting to elicit spatial understanding and planning abilities on LLMs. As in Figure 1 , instead of using intermediate thinking steps described in natural language in CoT prompts shown on the left-hand side, the CoS prompts remove the redundant text description but only using a set of symbols to represent spatial relationships between objects in complex environments. COS achieves noticeable improvement in both performance and efficiency (by up to 60.8% improvements in accuracy and 65.8% for the number of input tokens). We speculate that such an improvement is benefited by the more efficient symbolic representation produced by COS. Our main contributions are three-fold:• We evaluate LLMs on both existing classic spatial understanding tasks and our proposed synthetic spatial planning tasks. We spot that there is still room for performance improvements on current LLMs even with CoT.• We propose a novel method called COS, which prompts LLMs to convert the complex environment described with natural language into condensed symbolic Story: There are three boxes. In the left box, there are one large round in black, one small square in blue, one small square in blue, one small triangle in yellow, one middle square in yellow. In the middle box, there are one large square in yellow, one middle triangle in blue, one large round in black. In the right box, there are one large square in blue, one large triangle in black, one middle triangle in black. Question: How to move all black objects to the left box? Answer:To move all black objects to the left box, we need to identify all the black objects in the three boxes. (Mirzaee & Kordjamshidi, 2022) . Chains of Symbols are highlighted.representations. COS drastically improves LLMs on the spatial tasks. The accuracy gain of COS is large, also with a good reduction in the token consumption for LLMs. • We conduct an in-depth analysis on COS to explore the effect of using different symbols, on different LLMs, and different languages to show the robustness of our method.2 Spatial Planning and Understanding tasks 
natural language spatial planning: 
 Inspired by classic planning domains and tasks described in Liu et al. (2023) and existing spatial reasoning dataset Suhr et al. (2017) , we explore the performance of LLMs in three natural language spatial planning tasks. For all three tasks, we can formulate the problem as given a virtual scenario described by natural language, and a planning question. LLMs should take both the scenario and the question as the input and output correspondingly to solve the question. Such a solution usually contains a series of steps to achieve a final goal.The final test tasks consist of 5,500 evaluation instances, with 4,000 from Brick World, 1,000 from NLVR-based Manipulation, and the remaining 500 from Natural Language Navigation. We use code to generate these instances based on definition of each task.Figure 2 demonstrates an instance for Brick World (top), which requires the LLMs to acquire certain bricks by grabbing the bricks sequentially. We explore 1D and 2D scenarios for the Brick Worlds task. Specifically, in the 1D scenario, the relationship between bricks is only vertical. In the 2D scenario, in addition to the vertical relationship, there is also a horizontal relationship, which we express as "in the front of". To explore the characteristics of language understanding from LLMs, we investigate different levels of difficulty in the way of describing virtual scenarios. We describe them in increasing levels of difficulty as below.• Firstly, we explore labelling bricks from A to Z according to the order of spatial stacking from bottom to top, and the corresponding texts are also described in order from bottom to top, we call this setting "No shuffle". • Secondly, we shuffle the order of the corresponding natural language description while maintaining the labelling rules in alphabetic order called "Shuffle description". • Thirdly, we shuffled the order of labelling so that the spatial relationships do not correspond to the alphabetic order anymore, but are still described in the order from bottom to top in the text description, called "Shuffle label". • Finally, we shuffled both the order of labelling and description. We call it "Shuffle both".We use colors to represent the bricks, which enriches the information and increases the difficulty of the tasks. For each setting with 1D and 2D, we create 500 evaluation instances.The final evaluation set consists of 4,000 instances.Figure 2 demonstrates an instance for NLVR-based Manipulation (middle). We convert the format of Natural Language Visual Reasoning (NLVR, Suhr et al. (2017) ) tasks into a text-based planning task. Based on the creation rules of synthetic images of NLVR, we create 1,000 natural language descriptions for the virtual spatial environments using Python code. Specifically, for each description, we set three boxes just like NLVR, in the left, middle, and right, and in each box, and there are several objects. Each object has three properties: color, shape, and size. Each description has one related question, the question is about how to move all objects that satisfy a certain condition of one property (such as "all objects in black" or "all rounds") to a specific target box. The ground truth is the set of all objects satisfied with this condition which needs to be moved (not in the target boxes).Figure 2 demonstrates an instance for Natural Language Navigation (bottom). Inspired by Vision-and-Language navigation (Gu et al., 2022) , we create a virtual spatial environment that is similar to a 2D map of navigation tasks but using natural language description only. Specifically, we define a set of landmarks: ′ store ′ , ′ bank ′ , ′ house ′ , ′ cinema ′ , ′ garden ′ , ′ school ′ . For each description, there are 7 to 10 landmarks. We create 500 evaluation instances using Python code: the relationship between landmarks is a binary tree structure, with a root node which indicates the start point in the virtual scenario, and each node other than the leaf nodes has one or two child nodes, with a distance of 100 meters or 200 meters between them. Each description has one related question which is about how to reach the nearest one specific kind of landmark from the starting point.We also evaluate COS on manually annotated existing spatial question answering task, SPARTUN (Mirzaee & Kordjamshidi, 2022) , which contains a larger variety of spatial relation types and spatial expressions compared with previous Spatial QA datasets and our three synthetic spatial planning tasks. And the questions in this dataset are manually annotated, which is closer to real-world scenes. The scenarios in this dataset are described in natural languages based on NLVR (Suhr et al., 2017) and SPARTQA (Mirzaee et al., 2021) . 
chain-of-symbol prompting: 
 We propose Chain-of-Symbol (COS) prompting for LLMs, which converts the simulated environment with natural language into a condensed symbolic representation that considers spatial relationship. In order to make our constructing method of COS generalizable and reliable, we adopt a three-step procedure in creating the demonstrations of our COS which can be used in any related tasks:• (i) Automatically prompt the LLMs to generate a CoT demonstration in a zero-shot manner • (ii) Correct the generated CoT demonstration if there existing errors. • (iii) Replace the spatial relationships described in natural languages in CoT with random symbols, and only keep objects and symbols, remove other descriptions.We then use the COS demonstrations to guide the language model in a few-shot manner for prompting LLMs just like CoT (Wei et al., 2022) . Figure 1 depicts an example of a demonstration of CoS produced by models. In this example, we see that both CoT and COS receive the same shared simulated spatial environment in natural language texts. COS depicts a different intermediate thinking process than CoT. The latter represents the environments in a natural language only, while the former use a condensed symbolic representation that considers spatial relationship. Specifically, we use the symbol "/" to represent the spatial relationship "from the top of" here. By doing such a conversion, and removing redundant descriptions, COS effectively improves the model performance as well as reduces the inference costs with LLMs.Figure 2 depicts examples of CoS demonstration for all three planning tasks we proposed.For NLVR-based Manipulation, we convert natural language descriptions for objects to the format of a triplet such as "(large, round, black)". For Natural Language Navigation, we represent the order of landmarks by using symbol "/" to connect them. For Spatial QA task, we use a set of symbols such as "=", "∼" to represent different spatial relationships, and use triplet with "( , , )" to represent objects and their attributes.CoS prompting has multiple properties that are attractive as a prompting approach for LLMs:• First, COS effectively allows a neater, shorter, and condensed intermediate procedure than CoT. It is more structured than natural languages, hence easier for human annotators to analyze, check and correct the intermediate thinking process for LLMs. • Second, COS improves important planning tasks that current LLMs do not tackle well. It provides a better representing method for spatial environments which is easier for LLMs to learn compared with natural language. • Finally, COS clearly reduces the amount of text input into the LLMs and output from LLMs. This makes it much cheaper to access LLMs with API/GPU. 
experiments: 
 In this section, we first introduce our experimental setup in Section 4.1 about the settings of different methods we use, the language models, and the evaluation metrics. Then, in Section 4.2, we report the results of the three spatial planning tasks we proposed. In Section 4.3, we report the results on the SPARTUN dataset.We evaluate CoS and CoT on our proposed three spatial planning tasks and the existing SQA dataset, based on representative LLMs like ChatGPT(gpt-3.5-turbo) and LLAMA-2 series. There are three prompts: zero-shot CoT, few-shot CoT, and few-shot CoS (Ours).We consider zero-shot CoT as our baseline. The reason is that we have found that our choices of LLMs naturally give their intermediate steps (CoT) in their answers, even without specifically asking them to do so. We also found that asking them to remove the thinking steps obviously degrades the results. Therefore, we allow the LLMs to generate CoT, while we do not put any demonstration to the prompt but give prompts like "Let's think step by step" just as Kojima et al. (2022) . For an easier evaluation, we ask the LLMs to output the final results by separating the landmarks with commas. Chain-of-Thought Prompting This baseline uses a few-shot CoT, in which we encourage LLMs to think step by step, and we use five demonstrations to guide the LLMs in the thinking procedure. Note that the intermediate thinking procedure is represented as natural language text, just like the Standard Prompting. Like in (Wei et al., 2022) , we manually crafted five demonstrations for each task to guarantee their correctness. To guarantee the consistency and reliability of the prompts, we follow the format of CoT generated by zeroshot-CoT prompting. We use these fixed five demonstrations for evaluations on each task.As described in Section 3, COS augments the standard CoT prompting with condensed symbolic representation. While CoT has been shown to give large improvements to LLMs on various tasks (Wei et al., 2022) , we argue that using condensed symbolic representations can be an alternative to describing using natural language texts. We manually converted from CoT demonstrations to CoS using the procedure described in Section 3. Five CoS demonstrations of the same examples with CoT are created for each task of Natural Language Planning.We use both Opensource LLMs (Llama-2) and Closed Source LLMs ChatGPT(Gpt-3.5-turbo) for the evaluation of all tasks. We set the temperature to 0 for all the experiments throughout this paper.For planning tasks, we use three evaluation metrics, namely accuracy, precision, and recall. We define accuracy as the success rate in achieving the final goal.We then compute the Longest Common Sequence (LCS) between the ground truth and LLM output sequence to measure their similarity. We compute precision as the ratio of LCS against the length of the LLM output, and we compute recall as the ratio of LCS against the length of the ground truth. For spatial QA task, we only compute accuracy. In addition to the tasks described in English, we also tested COS on Brick World in Chinese. CoT reports 22.9% accuracy, and COS reports 39.1% in 1D scenario of Brick World, which demonstrates the robustness of COS to a language other than English. Robustness to Different Symbols Figure 3 demonstrates the robustness of using different symbols for COS. As we can see, using different symbols brings consistent performance, while not having any symbol drastically impacts the results. Among all the symbols, the comma gives the best results. We conclude that COS is robust to the selection of symbol. Results on Different Language Models Figure 4 reports the results on LLAMA-2 under the 1D scenario. The experimental results align with our previous conclusion that COS outperforms CoT obviously. Saving Tokens for Prompting One advantage featured by COS is that it reduces the number of input tokens to be fed into LLMs. For Brick World (1D scenario), COS reduces the number of tokens for the intermediate steps from 407 to 139 (Table 1 , the numbers are reported from OpenAI Playground2 ). This subsequently saves the costs of accessing LLMs via API/GPUs.For both of these two tasks, we adopt almost the same experimental settings as the ones for Brick World. The only difference is the evaluation metrics we report, we compute precision and recall based on the set rather than the Longest Common Sequence.Table 2 reports the results of NLVR-based Manipulation and Natural Language Navigation with GPT-3.5-turbo. For both of these two tasks, COS reports a higher performance than COT and zero-shot CoT prompting on all of the metrics.Saving Tokens for Prompting One advantage of COS is that it can reduces the number of input tokens. Table 2 reported that for NLVR-based Manipulation, COS reduces the number of tokens for the intermediate steps from 653 to 534, nearly by half of the original intermediate steps (we separate the tokens by space). This subsequently saves the costs of accessing LLMs via API/GPU, which enables easier access to the models. We also explore the effectiveness of COS in a more real-world scenario, by using existing human annotated spatial QA dataset SPARTUN (Mirzaee & Kordjamshidi, 2022) . Specifically, we applied both COS and CoT on GPT-3.5-Turbo and GPT-4. COS gains better performance and uses fewer tokens compared with CoT. In table 3 , we report the results of performance, and both CoT and CoS have 5 shots. It should be noticed that there are far more types of spatial relationships in SPARTUN dataset than our proposed planning tasks, so the results indicate CoS can gain promising performance even when there are a lot of symbols to represent different spatial relationships.We also evaluate CoS on current open-source representative models like LLAMA-2 Touvron et al. (2023) series to further validate the effectiveness and generality of our method. We use LLAMA-2 with different size (7B, 13B and 13B). As shown in Figure 4 , when the model size is small (7B and 13B), CoS cannot outperform CoT in many cases, but in 70B, CoS gain a clear better performance in all three tasks compared with using CoT. It can be seen that as the model size increases, the ratio of model performance to parameters for CoS has a larger slope compared to CoT. This indicate that large language models might can have emergent ability of abstract symbols understanding. 
related work: 
 In-Context Learning Large language models (LLMs) have demonstrated remarkable fewshot learning abilities across various domains (Brown et al., 2020; Srivastava et al., 2022) , which is also called as in-context learning, leading to a paradigm shift in AI to use LLMs as foundational models for language-related tasks, either directly or through fine-tuning (Bommasani et al., 2021; Hu et al., 2022) . While less relevant to COS, a concurrent work converts natural language into executable actions for robots with ChatGPT (Wake et al., 2023) . Another very recent concurrent work uses Symbol Tuning that replaces natural language labels with arbitrary symbols to improve in-context learning (Wei et al., 2023) . The ability of LLMs (Brown et al., 2020; Srivastava et al., 2022) to perform complex reasoning tasks can be significantly enhanced by using a show known as Chain-of-Thought (CoT) prompting, which involves providing them with intermediate reasoning steps (Nye et al., 2021; Wei et al., 2022) . Such a phenomenon also generalizes to the multilingual settings (Shi et al., 2023) . Despite the fact that CoT is powerful, there are reports that demonstrate that CoT is not always useful and that integrating CoT degrades the performance on the task of Machine Translation in their experiment. And this is possibly due to the word-by-word translation (Peng et al., 2023) .Spatial Reasoning Spatial reasoning over natural language texts has been an important research direction in the community (Janner et al., 2018; Mirzaee et al., 2021) . Janner et al. (2018) proposes to leverage representation learning on a navigation task that requires the agent to move a specific location. Rojowiec et al. (2020) proposes a new task on spatial reasoning that requires the language model to generate natural language instructions for 'before' and 'after' image pairs. Mirzaee et al. (2021) proposes a new benchmark for spatial question-answering with 'which' and 'what' questions regarding the environment. In a concurrent work, Tsai et al. (2023) demonstrates that LLMs perform poorly on text-based games with question-answering tasks that require several steps of reasoning.Navigation and Path Planning Language grounding navigation (Gu et al., 2022) refers to the interdisciplinary task that requires the intelligent agent to perceive the visual environment and guide the user to the goal location through natural language instructions (Nguyen et al., 2019; Chen et al., 2019) . Path planning (Panov et al., 2018; Krishna Lakshmanan et al., 2020) refers to the tasks that require the agent to plan its own path to achieve certain goals such as the shortest path or maximizing the cleaning area, typically through the use of reinforcement learning. These areas are highly relevant to the spatial planning tasks we explored and COS, as the spatial environments can be potentially represented by symbolic representations. We leave the investigations of these application areas to future studies. (Sun et al., 2024)  
conclusion: 
 We found that current popular LLMs still lack abilities in complex spatial planning and understanding tasks. To this end, we propose a novel method called COS (Chain-of-Symbol Prompting) that converts spatial relationships described in natural languages to condensed symbolic representations in the chained intermediate thinking steps. COS is easy to use and does not need additional training on LLMs. Extensive experiments indicate that using few-shot COS demonstration clearly surpasses the performance of using CoT described in natural languages on all three spatial planning tasks we proposed and the representative spatial QA benchmark with even fewer tokens (down to about 1/3 tokens of the thinking steps with CoT) used in the inputs compared with CoT prompting. The performance gain is strong, by up to 60.8% accuracy (from 31.8% to 92.6%) on Brick World for ChatGPT.Limitations Refer to the Appendix for the section on Broader Impact. In addition, we only use two models to verify the effectiveness of our method due to the limited time and resources. It would be interesting to apply our method to more models with different sizes to see whether there is an emergent ability of CoS for LLMs. Nevertheless, our choices of foundation models are representative and they are popular LLMs. 
 
 Appendix: A.1 Broader Impact COS is a prompting technique that is easy to use, which can effectively improve the performance of complex planning with LLMS. It also indicates that future training with LLMs can also be well benefited by employing COS in the training procedure to further improve LLM's planning abilities.We have mentioned that we used white spacing for calculating the number of tokens in the intermediate thinking steps. This was a typo and in fact, we accurately measures the number of tokens using the OpenAI Playground. 3 The numbers we reported are correct and there is no need for modification.The intermediate steps we use in the demonstrations for CoT are created and modified from the zero-shot CoT from the LLMs by simply adding "Let's think step by step" before the answer. We then manually correct the intermediate steps from the outputs of using zero-shot CoT for further improvements. We attempted our best efforts in tuning the baselines, and we report the best results we achieved.In the remaining of this section, we demonstrate the few-shot exemplars used in the experiments in our study. We demonstrate the exemplars for both COS and CoT. Question: There is a set of roads and a set of landmarks. The start point is bank H. There is a road which is 100 meters long from bank H to store E. There is a road which is 200 meters long from store E to bank C. There is a road which is 100 meters long from bank C to house A. There is a road which is 100 meters long from house A to house F. There is a road which is 200 meters long from bank C to garden I. There is a road which is 200 meters long from store E to cinema G. There is a road which is 200 meters long from cinema G to school J. There is a road which is 200 meters long from bank H to school D. There is a road which is 100 meters long from school D to store B. From the start point, how to reach the nearest school?Answer: Let's think step by step 1. Start at bank H. 2. There are two schools on the map, school J and school D. 3. The road from bank H to school J is from bank H to store E to cinema G to school J (200 meters + 200 meters + 200 meters = 600 meters). 4. The road from bank H to school D is from bank H to school D (200 meters). 5. Therefore, the nearest school from the start point (bank H) is school D. 6. Follow the road from bank H to school D (200 meters) . So the answer is HD Question: There is a set of roads and a set of landmarks. The start point is garden B. There is a road which is 200 meters long from garden B to cinema D. There is a road which is 200 meters long from cinema D to school J. There is a road which is 100 meters long from garden B to school I. There is a road which is 100 meters long from school I to house C. There is a road which is 100 meters long from house C to bank F. There is a road which is 100 meters long from house C to cinema G. There is a road which is 200 meters long from cinema G to store H. There is a road which is 100 meters long from store H to cinema E. There is a road which is 200 meters long from cinema E to bank A. From the start point, how to reach the nearest school?Answer: Let's think step by step 1. Start at garden B. 2. There are two schools on the map, school J and school I. 3. The road from garden B to school J is from garden B to cinema D to school J (200 meters + 200 meters = 400 meters). 4. The road from garden B to school I is from garden B to school I (100 meters). 5. Therefore, the nearest school from the start point (garden B) is school I. 6. Follow the road from garden B to school I (100 meters). So the answer is BI Story: There are three boxes. In the left box, there are one middle square in yellow, one middle square in black, one small square in blue, one middle square in blue. In the middle box, there are one large square in blue, one middle square in blue, one middle square in black, one large triangle in black, one middle round in blue, one small square in yellow. In the right box, there are one large round in blue, one small triangle in yellow, one large triangle in blue. Question:How to move all rounds to the middle box?Answer: To move all rounds to the middle box, we need to identify all the round objects in the three boxes. They are:-middle box: (middle, round, blue) -right box: (large, round, blue)Then we can move the objects not in the middle box above to the middle box one by one:-(large, round, blue) middle -right Story: There are three boxes. In the left box, there are one small round in blue, one small round in blue, one large round in black, one large square in blue, one small round in yellow, one small round in yellow. In the middle box, there are one large square in blue, one middle triangle in yellow. In the right box, there are one large round in black, one large round in blue, one middle triangle in black, one middle triangle in black. Question:How to move all squares to the right box?Answer: To move all sqaure objects to the right box, we need to identify all the square objects in the three boxes. They are:-left box: (large, square, blue) -middle box: (large, square, blue)Then we can move the objects not in the right box above to the right box:-(large, square, blue) left -right box -(large, square, blue) middle -right box Story: There are three boxes. In the left box, there are one large square in blue, one middle square in blue, one small round in blue, one middle triangle in blue, one middle round in yellow, one large square in yellow. In the middle box, there are one small round in yellow, one middle square in blue, one small triangle in black, one small square in black, one small triangle in yellow, one large round in black. In the right box, there are one small square in yellow, one small triangle in yellow, one middle triangle in black, one large round in yellow, one middle square in blue, one large square in yellow. Question:How to move all black objects to the right box?Answer: To move all black objects to the right box, we need to identify all the black objects in the three boxes. They are:-middle box: (small, triangle, black), (small, square, black), (large, round, black) Then we can move the objects not in the right box above to the right box:-(small, triangle, black) middle -right -(small, square, black) middle -right -(large, round, black) middle -right Story: There are three boxes. In the left box, there are one middle square in yellow, one middle square in black, one small square in blue, one middle square in blue. In the middle box, there are one large square in blue, one middle square in blue, one middle square in black, one large triangle in black, one middle round in blue, one small square in yellow. In the right box, there are one large round in blue, one small triangle in yellow, one large triangle in blue. Question:How to move all rounds to the middle box?Answer: To move all rounds to the middle box, we need to identify all the round objects in the three boxes. They are:-middle box: (middle, round, blue) -right box: (large, round, blue)Then we can move the objects not in the middle box above to the middle box:-(large, round, blue) right -middle Story: There are three boxes. In the left box, there are one large round in black, one small square in blue, one small square in blue, one small triangle in yellow, one middle square in yellow. In the middle box, there are one large square in yellow, one middle triangle in blue, one large round in black. In the right box, there are one large square in blue, one large triangle in black, one middle triangle in black. Question:How to move all black objects to the left box?Answer: To move all black objects to the left box, we need to identify all the black objects in the three boxes. They are:-left box: (large, round, black) -middle box: (large, round, black), -right box: (middle, triangle, black), (large, triangle, black)Then we can move the objects not in the left box above to the left box:1. (large, round, black) middle -left 2. (middle, triangle, black) right -leftTable 12 : Few-shot exemplars for full COT prompt for SPARTQA.Example 1:Background: There are three blocks called A, B, and C. In block A, there is a small blue square far above a small blue circle. There is a small black triangle far to the right of both small blue objects. The black triange is near and above a medium blue square. The medium blue square is touching the right edge of A and is near and above a medium black circle. Near and to the right of the medium black circle is a medium yellow triangle. The yellow triangle is to the left of the small blue circle. In B, which is to the right of A, there is a large blue square near and to the left of a large yellow triangle. There is also a small yellow triangle far below the square. In C, which is to the right of B. there is a small yellow circle near and above a small yellow triangle. The small yellow triangle is touching the top of a medium black triangle. The black triangle is touching the right edge of C. There is a large blue circle far below the black triangle.Question: Are all yellow objects in C above a blue thing?Candidate Answers:["Yes", "No", "Do not Know"]Answer:Let's analyze the information provided step by step: In block C: 1. There a small yellow circle near and above a small yellow triangle. 2. The small yellow triangle is touching the top of a medium black triangle. 3. There is a large blue circle far below the black triangle. Based on the provided information, the small yellow circle and the small yellow triangle in Block C are indeed above a blue thing, which is the large blue circle. So, the correct answer is: "Yes".Example 2:Background: There are three blocks called A, B, and C. A is to the left of B and B is to the left of C. In A, there is a small black triangle near and ot the left of a large blue circle. The large blue circle is near and to the left of a small blue circle. The small blue square is far below a medium yellow square. In B, there is a small yellow circle. In C, there is a medium black triangle that is near and to the left of a small yellow circle. The small yellow circle is near and to the left of a blue triangle. There is a large yellow square touching the bottom of the small yellow circle. Near and below the large yellow square is a large black square.Question: What is the relation between the large black thing and the small yellow thing?Candidate Answers:["left", "right", "above", "below", "near to", "far from", "touching", "Do Not Know"]Answer:Let's analyze the information provided step by step: In block C: 1. There is a medium black triangle that is near and to the left of a small yellow circle. 2. The small yellow circle is near and to the left of a blue triangle. 3. There is a large yellow square touching the bottom of the small yellow circle. 4. Near and below the large yellow square is a large black square. Now, let's determine the relation between the "large black thing" (large black square) and the "small yellow thing" (small yellow circle): 1. The large black square is in block C, and it is described as being "near and below" the large yellow square. 2. The small yellow circle is also in block C, and it is described as being "near and to the left of" the medium black triangle. Based on the provided information, the large black square is below the large yellow square and the small yellow circle is near to the medium black triangle. So, the correct answer: "below" and "near to".Background: There are three blocks called A, B, and C. A is to the right of B and B is to the right of C. In A, there is a medium black circe that is far to the right of a medium blue square. The medium blue square is near and to the right of a small yellow circle. Far above the blue square is a medium yellow triangle. Near above the medium triangle is a small blue circle. In B, there is a large blue square that is touching the right edge of B. There is a medium black triangle that is touching the left side of the blue square. Near and above the medium black triangle is a large blue circle. Near and above the large blue circle is a large yellow triangle. In C, there is a small yellow triangle near and to the right of a medium black circle. The yellow triangle is touching the right edge of C. Far to the left of the medium black circle is a small blue square.Question: What block does not have any objects touching the edge of it?Candidate Answers: ["A", "B", "C"]Answer:Let's analyze the information provided step by step: In block A: 1. There is a medium black circle, which is far to the right of a medium blue square. 2. The medium blue square is near and to the right of a small yellow circle. 3. Far above the blue square is a medium yellow triangle. 4. Near above the medium triangle is a small blue circle. In block B: 1. There is a large blue square that is touching the right edge of B. 2. There is a medium black triangle that is touching the left side of the blue square. 3. Near and above the medium black triangle is a large blue circle. 4. Near and above the large blue circle is a large yellow triangle. In block C: 1. There is a small yellow triangle near and to the right of a medium black circle. 2. The yellow triangle is touching the right edge of C. 3. Far to the left of the medium black circle is a small blue square. Based on the provided information, it's Block A, which does not have any objects touching its edge. So, the correct answer is: "A".Background: There are three blocks called A, B, and C. A is to the left of B and B is to the left of C. In A, there is a large black triangle far above a medium black triangle. The medium black triangle is near and above a large yellow circle. Near and to the right of the large circle is a small blue circle. In B, there is a medium blue circle near and to the left of a small black square. Far below the blue circle is a small yellow circle. In C, there is a small blue square.Question: What object is near the large yellow thing, the medium black triangle or the small blue circle the medium black triangle or the smal blue circle?Candidate Answers:["the medium black triangle", "the smal blue circle", "both of them", "none of them"]Answer:Let's analyze the information provided step by step: In lock A: 1.There is a large black triangle far above a medium black triangle. 2. The medium black triangle is near and above a large yellow circle. 3. Near and to the right of the large circle is a small blue circle. In block B: 1. There is a medium blue circle near and to the left of a small black square. 2. Far below the blue circle is a small yellow circle. In block C: 1. There is a small blue square. Now, let's analyze the position of the large yellow circle: 1. The large yellow circle is in Block A and is near and above the medium black triangle. 2. The small blue circle is also in Block A and is near and to the right of the large yellow circle. Based on the provided information, both the medium black triangle and the small blue circle are near the large yellow circle in Block A. So, the correct answer is: "both of them".Background: There are three blue shapes in a block that we called A. Two medium blue squares are below a big blue square. First medium blue square is to the left of the other medium one which is touching the bottom edge of block A. There is another block below the block A we call it B. A big blue triangle is touching the top edge of this block which is above a small black triangle, a big blue circle and a medium blue triangle. The medium triangle is touching the bottom edge of the block. The circle is below and to the left of the small triangle. It is above the medium triangle.Question: Is the small triangle below the medium triangle?Candidate Answers:["Yes", "No", "Do not Know"]Answer:Let's analyze the information provided step by step: In block B: 1. A big blue triangle is touching the top edge of this block which is above a small black triangle, a big blue circle and a medium blue triangle. 2. The medium triangle is touching the bottom edge of the block. 3. The circle is below and to the left of the small triangle. It is above the medium triangle. Based on the provided information, the small black triangle is Table 13 : Few-shot exemplars for full COS prompt for SPARTQA.Example 1:Background: There are three blocks called A, B, and C. In block A, there is a small blue square far above a small blue circle. There is a small black triangle far to the right of both small blue objects. The black triangle is near and above a medium blue square. The medium blue square is touching the right edge of A and is near and above a medium black circle. Near and to the right of the medium black circle is a medium yellow triangle. The yellow triangle is to the left of the small blue circle. In B, which is to the right of A, there is a large blue square near and to the left of a large yellow triangle. There is also a small yellow triangle far below the square. In C, which is to the right of B. There is a small yellow circle near and above a small yellow triangle. The small yellow triangle is touching the top of a medium black triangle. The black triangle is touching the right edge of C. There is a large blue circle far below the black triangle.Question: Are all yellow objects in C above a blue thing?Candidate Answers:["Yes", "NO", "Do not Know"]Answer:Let's analyze the information provided step by step to determine if all yellow objects in Block C are above a blue thing: C: 1. (small, yellow, circle) ↑ (small, yellow, triangle) 2. (small, yellow, triangle) = top of (medium, black, triangle) 3. (large, blue, circle) ∞↓ (medium, black, triangle). Based on the provided information, C: (small, yellow, circle) ↑ (large, blue, circle) and (small, yellow, triangle) ↑ (large, blue, circle). So, the correct answer is: "Yes".Background: There are three blocks called A, B, and C. A is to the left of B and B is to the left of C. In A, there is a small black triangle near and to the left of a large blue circle. The large blue circle is near and to the left of a small blue circle. The small blue square is far below a medium yellow square. In B, there is a small yellow circle. In C, there is a medium black triangle that is near and to the left of a small yellow circle. The small yellow circle is near and to the left of a blue triangle. There is a large yellow square touching the bottom of the small yellow circle. Near and below the large yellow square is a large black square.Question: What is the relation between the large black thing and the small yellow thing?Candidate Answers:["left", "right", "above", "below", "near to", "far from", "touching", "Do Not Know"] Answer:Let's break down the relationships between the objects step by step: A: 1. (small, black, triangle) < (large, blue, circle) 2. (large, blue, circle) < (small, blue, circle) 3. (small, blue, square) ∞↓ (medium, yellow, square) B: 1. (small, yellow, circle) C: 1. (medium, black, triangle) < (small, yellow, circle) 2. (small, yellow, circle) < (blue, triangle) 3. (large, yellow, square) =↓ (small, yellow, circle) 4. (large, black, square) ↓ (large, yellow, square) Now, let's determine the relation between the (large, black, square) and the (small, yellow, circle): 1. C: (large, black, square) ↓ (large, yellow, square). Therefore, (large, black, square) ↓ (large, yellow, square) 2. C: (small, yellow, circle) < (medium, black, triangle). Therefore, (small, yellow, circle) (medium, black, triangle) So, the correct answer: "below" and "near to".Background: There are three blocks called A, B, and C. A is to the right of B and B is to the right of C. In A, there is a medium black circle that is far to the right of a medium blue square. The medium blue square is near and to the right of a small yellow circle. Far above the blue square is a medium yellow triangle. Near the medium triangle is a small blue circle. In B, there is a large blue square that is touching the right edge of B.There is a medium black triangle that is touching the left side of the blue square. Near and above the medium black triangle is a large blue circle. Near and above the large blue circle is a large yellow triangle. In C, there is a small yellow triangle near and to the right of a medium black circle. The yellow triangle is touching the right edge of C. Far to the left of the medium black circle is a small blue square.Question: What block does not have any objects touching the edge of it?Candidate Answers:["A", "B", "C"]Answer:Let's analyze each block step by step to determine which one does not have any objects touching its edge: A: 1. (medium, black, circle) ∞> (medium, blue, square). 2. (medium, blue, square) > (small, yellow, circle Background: There are three blocks called A, B, and C. A is to the left of B and B is to the left of C. In A, there is a large black triangle far above a medium black triangle. The medium black triangle is near and above a large yellow circle. Near and to the right of the large circle is a small blue circle. In B, there is a medium blue circle near and to the left of a small black square. Far below the blue circle is a small yellow circle. In C, there is a small blue square.Question: What object is near the large yellow thing, the medium black triangle the small blue circle the medium black triangle or the small blue circle?Candidate Answers:["the medium black triangle", "the small blue circle", "both of them", "none of them"] Answer:Let's break down the information step by step to determine which object is near the large yellow circle, the medium black triangle, or the small blue circle: A: 1. (large, black, triangle) ∞↑ (medium, black, triangle). 2. (medium, black, triangle) ↑ (large, yellow, circle). 3. (small, blue, circle) > (large, circle). B: 1. (medium, blue, circle) < (small, black, square). 2. (small, yellow, circle) ∞↓ (blue, circle). C: 1. there is a small blue square. Now, let's analyze the position of the (large, yellow, circle): 1. A: (large, yellow, circle) ↑ (medium, black, triangle). 2. A: (small, blue, circle) > (large, yellow, circle). Based on the provided information, A: (medium, black, triangle) (large, yellow, circle) and (small, blue, circle) (large, yellow, circle). So, the correct answer is: "both of them".Background: There are three blue shapes in a block that we call A. Two medium blue squares are below a big blue square. The first medium blue square is to the left of the other medium one which is touching the bottom edge of block A. There is another block below block A we call it B. A big blue triangle is touching the top edge of this block which is above a small black triangle, a big blue circle and a medium blue triangle. The medium triangle is touching the bottom edge of the block. The circle is below and to the left of the small triangle. It is above the medium triangle.Question: Is the small triangle below the medium triangle?Candidate Answers:["Yes", "NO", "Do not Know"]Answer:Let's analyze the arrangement of objects in block B to determine if the small triangle is below the medium triangle: B: 1. (big, blue, triangle) = top edge of B 2. (big, blue, triangle) ↑ (small, black, triangle), (big, blue, circle) and (medium, blue, triangle) 2. (big, blue, circle) ↓< (small, black, triangle) 3. (big, blue, circle) ↓ (big, blue, triangle) 4. (medium, blue, triangle) = bottom edge B Based on the provided information, the small black triangle is above the medium blue triangle. So, the correct answer is: "No."https://platform.openai.com/playgroundhttps://platform.openai.com/playground(large, triangle, black) right -left 


Decision: The authors propose chain-of-symbol prompting for
spatial-reasoning-from-text tasks. The method involves manually
constructing more compact/symbolic versions of chain-of-thoughts
generated by models in a zero-shot fashion, and using these updated
representations as in-context examples.  This method outperforms a
baseline of not making the proposed symbol substitution, sometimes
significantly, with gpt-35-turbo, and llama-2 (and does so using fewer
tokens).

Reviewers appreciated the simplicity, effectiveness, and efficiency of
the proposed method. A few themes among the weaknesses:

- Some reviewers (e.g., iTYN) hoped for more experiments about
  when/why the CoS method works, while others (e.g., C6LX, cm3f) hoped
  for more models covered, while others (e.g., iTYN) hoped for more
  datasets

- Some reviewers highlighted presentation issues (e.g., ST4m brought
  up the "easy to use" discussion)

In my view, the two biggest remaining shortcomings are:

- Why does CoS help? Are there error analyses that could be conducted
  on the individual instances where improvement occurs?

- Does CoS really generalize to other similar tasks? Or to tasks
  beyond spatial reasoning from text?

Concerns were mostly addressed in conversation. Overall, this work
provides (another) striking illustration of when/how representations
can matter for LLMs --- while CoT and CoS represent the same
information, CoS clearly does so in a manner more accessible to
LLMs. The promised additional experiments the authors mentioned in
their response are likely to provide additional missing context as
requested by reviewers.

Reviews: review: paper_summary: paper_summary: The paper regards the implementation and evaluation of a symbolic language to communicate with a large language model. The translation of the natural language query or instruction to its symbolic form is done manually by the user. The results are compared with those obtained with zero-shot chain-of-thought prompting and with few-shot chain-of-thought prompting where the prompts are also manually drafted but are in natural language.
strength_weakness: strength_weakness: -	The expressiveness of the symbolic language is not theoretically defined and evaluated. It is not clear whether it exhaustively covers complex spatial configurations of objects in a real world settings.
-	The paper claims that the symbolic language is easy to use, but a users’ study that supports this claim is lacking.
-	The contribution towards novel techniques to improve language modeling is not clear. The paper rather suits a human-computer interaction venue.
-	It is not surprising that the obtained results improve compared to zero- and few-shot chain-of-thought reasoning as the symbolic language has less complex patterns than natural language and is an unambiguous controlled language to attend to when querying the LLM, but it requires knowledge engineering expertise from the user.
-	The paper seems to be written in a hurry and contains typos and grammatical errors.
- Overall, the paper is not yet mature to be published but the authors are encouraged to improve their work.
questions: questions: - See the remarks above.
paper_summary: paper_summary: This paper focuses on spatial reasoning using LLMs and it proposes a prompting method called "Chain-of-Symbol (CoS)", which follows the same idea of Chain-of-Thought (CoT) but expresses the reasoning process using concise symbols. This approach was evaluated on four datasets: Brick World, NLVR-based Manipulation, Natural Language Navigation, and SPARTUN. Few-shot CoS was shown to outperform few-shot CoT in most cases.
strength_weakness: strength_weakness: 1. The experimental designs need more justification.

(1) Concern about the validity of the baseline: To show the effect of using concise symbols, a more fair comparison should design CoS and CoT to follow _the same reasoning process_ while being different _only_ in their expression (i.e., symbols vs. texts). Only in this more controlled comparison can one see if symbols are more effective than textual reasoning. However, prompts in the Appendix have shown inconsistent designs between CoS and CoT. For example, on Brick World (Table 4 vs. Table 5), while CoS reasons about the spatial relations before giving the answer sequence, CoT directly presents the answer sequence. Similarly, on Natural Language Navigation (Table 8 vs. Table 9), while CoS enumerates the stops of each path and shows its distance calculation before reaching the conclusion of the shortest choice, CoT seems to miss some of the calculation details (e.g., "4. The road from bank A to store B is shorter than the road from bank A to store G" without saying the calculation of the latter path). As a result, in the current less fair experimental setup, it is hard to understand whether the gain of CoS over CoT is an effect of using symbols, or if it is simply due to prompting the LLM with different reasoning logic. 

(2) Experiments were performed using GPT-3.5-turbo, so it is unclear if the advantage of CoS holds for more advanced LLMs such as GPT-4.

(3) Concern about the choice of datasets for spatial reasoning: I have found the use of "NLVR-based Manipulation" and "Natural Language Navigation" as benchmark datasets for spatial reasoning confusing. They do not seem to focus on spatial reasoning. For "NLVR-based Manipulation", the involved "spatial" aspect is limited to left vs. middle vs. right boxes, whereas the main challenge of this dataset seems to be about "perception" (e.g., recognizing the color, size, and shape of objects). For "Natural Language Navigation", it is more about motion planning while there's no obvious challenge in inferring the spatial relationship among intermediate stops on a path.

2. Novel of CoS: How does CoS compare to Program of Thought, which also uses symbols to represent the reasoning process of problem solving?

Chen, Wenhu, et al. "Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks." Transactions on Machine Learning Research (2023).
questions: questions: Please address my concerns in Reason to Reject.

Figure 2: some texts are not properly displayed in "Natural Language Navigation".
paper_summary: paper_summary: The work proposed the new in-context learning method called Chain of Symbol (CoS)  which utilizes symbolic representations for spatial relation objects rather than natural language representations.  They claim the proposed technique is more effective and reliable than commonly used CoT techniques.  They prove the claim using four tasks involving spatial understanding of various dimensions (1D, 2D, and 3D spatial relations) with closed-source and open-source LLMs.  The results indicate that CoS completely outperformed in performance and resources on all tasks.
strength_weakness: strength_weakness: - The improvement rapidly drops on the complex tasks compared to simple ones, which may show indifference between symbolic representation and natural representation in the complex setting
 - Applying to the realistic scenario will be challenging because the symbolic representation is not straightforward to obtain with reasonable coverage of concepts
 - According to LLama2 results, the proposed techniques only out-performed CoT with the largest model
questions: questions: - You provided the result of GPT-4 on the SPARTUN dataset; what are the results of GPT-4 on the other benchmarks? 
 - Can you visualize the Llama2 results in the same format and details of the GPT results? More details will be helpful since the figure 4 plots is not very conclusive. 
 - Would it be possible to apply CoS approach to the other types of reasoning beyond spatial?
- In the analysis part of BrickWorld, did you experiment with different symbolic representations? What would be the expected results in more complex setting if the symbolic representations are changed? The extreme case would be using <- to represent left instead of right. Would the model still understand it?
- Can you justify using GPT for creating the initial explanation compared to manually crafting CoS? is the motivation reducing the human annotation effort?
- For CoT, the experiment is performed using manually crafted CoT. What about the results from CoT if you follow the same steps as CoS?
paper_summary: paper_summary: This paper proposes to use abstract chain-of-thought prompts that simply use symbols, rather than natural language. This results in notable improvements across a variety of planning and spatial reasoning tasks. This seems like a useful insight that will improve applications of language models, and is also somewhat conceptually interesting.
strength_weakness: strength_weakness: * The paper doesn't quantify thoroughly *when* and *why* CoS is beneficial. While there are a diversity of comparisons, it would be useful to dig into some of them in more depth; for example, are there characteristic patterns of errors that CoT makes but CoS is more likely to avoid? Conversely, where CoS makes an error, does CoT tend to make the same error, or are they uncorrelated? Etc. 
     - It would be even better if it were possible to link some of these analyses to the CoS itself; e.g. showing that it contains more correct steps than the CoT.
     - If possible, it would also be very interesting to quantify the distributional effects of the CoT to CoS switch along the lines of the proposed hypothesis above: for example, is the distribution over next tokens higher entropy in CoT than in CoS? Could that help explain the benefit?
* It would be nice to see some evaluations on more challenging benchmarks like https://arxiv.org/abs/2206.10498
* If I understand correctly, in Fig. 3 "None" means no symbol or space between the letters; it seems to me that having a space would also be a useful comparison condition to run (to see if the eponymous symbols are actually critical, or if it's just necessary to have *some* break in between to avoid tokenizer weirdness).

Comments: comment: Thank you for your response, I have read all other reviews and the response. Based on the responses, it does not seem to me that there was any misunderstanding in my evaluation. My overall evaluation was positive, I kept the same.
1.1 I hope the authors can expose the details and the results in the next version. A sentence of "We attempted our best efforts in tuning the baselines" is very ambiguous:) However, I found the result surprising (less rigorous step-by-step CoT outperforming the more rigorous one) and believe that more discussions are needed.

1.2 I'm sorry for the cost concern, but an experiment using GPT-4 makes more sense and will certainly tell people how much needed the CoS is. 

1.3 The answer is not convincing. The original NLVR was designed to evaluate visional reasoning, including spatial reasoning, but the NLVR-Manipulation in experiments as I understand was annotated by the authors. Based on examples in the paper, the kind of spatial reasoning it needs is very shallow. The Natural Language Navigation benchmark was also created by the authors, not any prior work, hence "they are both representative benchmarks" is very misleading. 

2. Thanks for the justification.

I increased my score as parts of my concerns were addressed.
Thanks to the authors for the interesting work, I am looking forward to seeing these results and the follow-ups in the future!
Thank you for the answers which are clarifying.