Chain-of-Symbol Prompting For Spatial Reasoning in Large Language Models
Authors: Hanxu Hu
Hongyuan Lu
Huajian Zhang
Yunze Song
Wai Lam
Yue Zhang
Venue: COLM (2024)
Keywords: In-Context Learning, Chain-of-Thought Prompting, Spatial Reasoning
TL;DR: A symbolic prompting methods representing complex spatial relationships by using simple symbols for LLMs

Abstract: While conventional Chain-of-Thought prompting shows promising performance on various language tasks for LLMs, the spatial scenarios are nearly unexplored. In this paper, we first investigate the performance of LLMs on complex spatial planning and understanding tasks that require LLMs to understand a virtual spatial environment simulated via natural language and act or reason correspondingly in text. By evaluating on classic spatial planning scenarios through natural language descriptions, we found that current popular LLMs still lack abilities to handle spatial relationships in texts. This arises a question -- do the natural language is the best way to represent complex spatial environments for LLMs, or maybe other alternatives such as symbolic representations are both more efficient and effective for LLMs? To this end, we propose a novel method called CoS (Chain-of-Symbol Prompting) that represents the spatial relationships with condensed symbols during the chained intermediate thinking steps. CoS is easy to use and does not need additional training on LLMs. Extensive experiments indicate that CoS clearly surpasses the performance of the Chain-of-Thought (CoT) Prompting described in natural langauge in all three spatial planning tasks and existing spatial QA benchmark, with even fewer tokens used in the inputs compared with CoT. The performance gain is strong, by up to 60.8% accuracy (from 31.8% to 92.6%) on Brick World scenarios for GPT-3.5-Turbo. CoS also reduces the number of tokens in the prompt obviously, by up to 65.8% of the tokens (from 407 to 139) for the intermediate steps from demonstrations on the Brick World task. Interestingly, we also observed emergent ability of abstract symbols understanding when the size of models scales up.

Published as a conference paper at COLM 2024
Chain-of-Symbol Prompting for Spatial Reasoning in Large Language Models
Hanxu Hu! Hongyuan Lu² Huajian Zhang?, Yun-Ze Song', Wai Lam?, Yue Zhang' Westlake University 2The Chinese University of Hong Kong University of Edinburgh
Juhaaxuu1333@gal) com
wiyluw cuhk. edu, hk v ed. ac. .k
zhangyue@vestlake.
.ew.aaaeeeeeeeeeeee...eeeeee.e.. edu. cn
Abstract
In this paper, we first investigate the performance of LLMs on complex planning tasks that require LLMs to understand virtual spatial environ- ment simulated via natural language and act or reason correspondingly text. By evaluating on classic spatial planning scenarios, we found that current LLMs still lack abilities to handle spatial relationships in texts. This arises a question: Is the natural language the best way to represent complex spatial environments for LLMs, or are other alternatives such symbolic representations more efficient and effective for LLMs? To this end, we propose a novel method called CoS (Chain-of-Symmol Prompting) that represents the spatial relationships with condensed symbols during the chained intermediate thinking steps. CoS easy to use and does not need additional training on LLMs. Extensive experiments indicate that CoS clearly surpasses the performance of the Thaoughf- (CoT) Prompting described in natural language in all three spatial reasoning and planning tasks with even fewer tokens used the inputs compared with COT. The performance gain is strong, by up to 60.8% accuracy (from 31.8% to 92.6%) on Brick World for GPT-3.5-Turb.. COS also reduces the number of tokens in the prompt obviously, by up to 65.8% of the tokens (from 407 to 139) for the intermediate steps from demonstrations on Brick World. Interestingly, we also observed emergent ability of abstract symbols understanding when the size of models scales up.
1 Introduction
Given set of target behaviour examples, large language models (LLMs) demonstrate exceptional abilities to accomplish a wide range of tasks, frequently exhibiting performance that surpasses that of humans (Brown al., 2020 Srivastava al., 2022). Specifically, LLMs exhibit impressive sequential textual reasoning ability during interence, resulting in a significant boost in their performance when encountered with reasoning questions described in natural languages (Nye et al., 2021 Wei et al. (022).. This phenomenon can be clearly observed with a multi-step chain of intermediate thinking procedure, i.e., a "Chain of Thought" (CoT, Wei et al.
Conventional CoT usually leverages natural languages as intermediate thinking steps in prompting. Although CoT can enhance LLMs' ability in many cases, redundant natural languages and irrelevant information also can hamper the performance of LLMs (Shi al. 2023) in some cases. For example, spatial languages and descriptions can be hard for Mirzaee & Kordjamshidi 2022 due to
language models to understand Mirzaee et al. 2021
"Equal Contribution
t *Corresponding Author
1 'Qu code available httpp/////gggtt......o./aaaaaaaa htt.:aaiii....--ommo---llmaaaaaa

---

Published as a conference paper at COLM 2024
Shared Model Input
There bricks. The yellow brick top brick yellow brick on tap of the brick The yellow brick is on top of the brick D The white brick A is on top of the brick B.. For the brick B, the color is white. Now we have to get now grabbed from top bottom, and lower brick grabbed, the upper brick
specific brick, The bricks be removed first. How to get brick D?
Chain-of- Thhinghf-Thought Thought Prompting
Chain-of-Symbol Prompting
The bricks from bottom to top B, A, E,
brick from top of brick
B/A/D/E/C
C/E
2. Remove brick from the top of brick D.
3. Now brick the topmost yellow brick and can be grabbed.
E/D
Model Output
Model Output
get the result A,
X
So we get the result C,
Figure 1: An example for comparison between Chain-of-Thought (CoT) and Chain-of- Symbol (CoS) that elicits large language models in tackling complex planning tasks with higher performance and fewer input tokens. We let the model generate CoT/ CoT/CoS during inference in a few-shot manner. Results were taken in May 2023 with ChatGPT and can be subject to change.
complex spatial relationships. Aligning symbols and representing spatial relationships by symbols in word sequences can be a neater representation and thus can be potentially easier to understand by LLMs. We thus explore the use of symbols for LLM prompting, which is still an understudied topic. This is important to study which implies understanding abilities beyond language models for language understanding per se.
To explore the role of symbolic representations in prompting, we take the complex spatial understanding and planning the evaluation scenarios, which require LLMs to understand the virtual spatial environments described through natural language as well as planning and achieving certain goals in such environments. Inspired by existing classic planning competitions and spatial reasoning datasets, we present three domains: (i) Brick World (ii) NLVR-based Manipulation and (iii) Natural Language Navigation. Figure illustrates an example for Brick World 1D, and all these three tasks are described in detail in Section 2.1 These three tasks are all described in natural language. And we also evaluate one existing spatial question answering dataset SPARTUN (Mirzaee & Kordjamshidi 2022) which uses -umaan-eneeraed questions thus closer to realistic situations. For these tasks, LLMs need to understand a virtual environment in natural language, with the spatial relationship between the objects to be operated on and the restrictions on the operation, which easy for real humans. However, we found that there are still places for improvement in the performance of LLMs on the tasks.
As major contribution to this study, we investigate the symbolic representations for spatial relationships, and propose a novel method called Chain-of-Symbol (COS) prompting to elicit spatial understanding and planning abilities LLMs. As in Figure 1 instead of using intermediate thinking steps described in natural language in CoT prompts shown on the left-hand side, the CoS prompts remove the redundant text description but only using set of symbols to represent spatial relationships between objects in complex environments. CoS achieves noticeable improvement in both performance and efficiency (by up to 60.8% improvements in accuracy and 65.8% for the number of input tokens). We speculate that such an improvement benefited by the more efficient symbolic representation produced by CoS. Our main contributions are three-fold:
We evaluate LLMs on both existing classic spatial understanding tasks and our proposed synthetic spatial planning tasks. We spot that there is still room for performance improvements current LLMs even with COT.
We propose novel method called COS, which prompts LLMs to convert the complex environment described with natural language into condensed symbolic
2

---

Published as a conference paper at COLM 2024
World 1D
bnct o los of brok The white brec ca on tep of brik o for bescc colore bue The dhu heie wall ros he sop lection and Fitee lown to grolundd the oper rocc wrtichi/e//
Spetial
fow the
Background
that un square. The first modium blue square is the left of the or medium are which is touching the wnotthrr block
There three shapes
se grette woutae D.C.R.
nattom wige
A
Dayy Thre are thee booes the lift box there are cine arge sound in dock one small square"
NLVR- based Manipulation
touching the top edge of this block which is above small black traanlcc a big blue circle and medium trtanglee The medium
midele boc, there are one large rquare vellow, one midile trangie in blut one airge round
bage
arge trangla
The bollow and to the left of the small trionglc. it sbove the medium or rannge..
is
triangli black.
Queetion low to mowe al back teti the left
To move all cluck otjects to the isft box we nead to iduntlly all the block obiects in thres
the small triangle below the medium triengle?
Candicats A Answers
peree round. black)
IYes', "NO", "Do not Know!
boc (midsle, triengle, blackl, (Harge, triengls, black)
Ansow::
Let's anelyae the amangumment of objects in block to datermine small trangla below medium briangle:
Tanux sm moss The sexcts sy. Ire sit son abovs to the kol
(aagg, round,
mesale
(middi: brlangli, black; ghn
triengle, blacki
(big. blue, triangle)
Natural Language Navigation Terr whiris
blue, triangle) (small, black, triengla), circke) (medium, trianglo) blue,
200 meturs long from bark A to bork C. There a reed which a 100 metars long from bank Co is 300 mawe ling from Diems In core These ica mad which is 100 mewe long from cois it
(small (big. blue, circk) (big, triangls) (medium blue, triengle) battom B Rosod the provided information, corecct andwer "No."
360 maters from house garden. Thers is road which 100 motes from bank
to cnemal There road whech meters long from onama 2: houas from start the
bunt / baakk C hause / /ioess// cos 8 1200
600)
triengle
conmms Ssas get the mest A / ciemm store
Figure 2: <input, Chain of Symbol, output> example triples for our three proposed tasks: Brick World, NLVR-based Manipulation, and Natural Language Navigation, and SPARTUN dataset (Mirzaee & Kordjamshidi 2022). Chains of Symbols are highlighted.
representations. CoS drastically improves LLMs on the spatial tasks. The accuracy gain of COS is large, also with a good reduction in the token consumption for LLMs.
We conduct an in-depth analysis on CoS to explore the effect of using different symbols, on different LLMs, and different languages to show the robustness of our method.
2 Spatial Planning and Understanding tasks
2.1 Natural Language Spatial Planning
Inspired by classic planning domains and tasks described in Liu et al. (2023) and existing spatial reasoning dataset Suhr et al. (2017), we explore the performance of LLMs in three natural language spatial planning tasks. For all three tasks, we can formulate the problem as given a virtual scenario described by natural language, and a planning question. LLMs should take both the scenario and the question as the input and output correspondingly to solve the question. Such solution usually contains a series of steps to achieve final goal. The final test tasks consist of 5,500 evaluation instances, with 4,000 from Brick World, 1,000 from NLVR-based Manipulation, and the remaining 500 from Natural Language Navigation. We use code to generate these instances based on definition of each task.
2.2 Brick World
demonstaate an instance for Brick World (top), which requires the LLMs to acquire certain bricks by grabbing the bricks sequentially. We explore 1D and 2D scenarios for the Brick Worlds task. Specifically, in the 1D scenario, the relationship between bricks is only vertical. In the 2D scenario, in addition to the vertical relationship, there is also horizontal relationship, which we express as "in the front of". To explore the characteristics of language understanding from LLMs, we investigate different levels of difficulty in the way of describing virtual scenarios. We describe them in increasing levels of difficulty as below.
3

---

Published as a conference paper at COLM 2024
Firstly, we explore labelling bricks from A to Z according to the order of spatial stacking from bottom to top, and the corresponding texts are also described in order from bottom to top, we call this setting "No shuffle".
Secondly, we shuffle the order of the corresponding natural language description while maintaining the labelling rules in alphabetic order called "Shuffle description".
Thirdly, we shuffled the order of labelling so that the spatial relationships do not correspond to the alphabetic order anymore, but are still described the order from bottom to top in the text description, called "Shuffle label".
Finally, we shuffled both the order of labelling and description. We call "Shuffle both".
We use colors to represent the bricks, which enriches the information and increases the difficulty of the tasks. For each setting with 1D and 2D, we create 500 evaluation instances. The final evaluation set consists of 4,000 instances.
2.3 NLVR-based Manipulation
Figure 2 demonstrates an instance for NLVR-based Manipulation (middle). We convert the format of Natural Language Visual Reasoning (NLVR, Suhr et al. (2017)) tasks into text-based planning task. Based on the creation rules of synthetic images of NLVR, we create 1,000 natural language descriptions for the virtual spatial environments using Python code. Specifically, for each description, we set three boxes just like NLVR, in the left, middle, and right, and each box, and there are several objects. Each object has three properties: color, shape, and size. Each description has one related question, the question is about how to move all objects that satisfy a certain condition of one property (such as "all objects in black" or "all rounds") to a specific target box. The ground truth is the set of all objects satisfied with this condition which needs to be moved (not in the target boxes).
2.4 Natural Language Navigation
Digureddemone demonstrates an instance for Natural Language Navigation (bottom). Inspired by VisionanddLaanguae navigation (Gu et al. 2022), we create virtual spatial environment that is similar to a 2D map of navigation tasks but using natural language description only, Specifically, we define a set of landmarks: 'store', bank', house', / cinema', garden', school". For each description, there are 7 to 10 landmarks. We create 500 evaluation instances using Python code: the relationship between landmarks is a binary tree structure, with a root node which indicates the start point in the virtual scenario, and each node other than the leaf nodes has one or two child nodes, with a distance of 100 meters or 200 meters between them. Each description has one related question which is about how to reach the nearest one specific kind of landmark from the starting point.
2.5 Spatial QA
We also evaluate COS on manually annotated existing spatial question answering task, SPARTUN (Mirzaee Kordjamshidi 2022), which contains a larger variety of spatial relation types and spatial expressions compared with previous Spatial QA datasets and our three synthetic spatial planning tasks. And the questions in this dataset are manually annotated, which is closer to real-world scenes. The scenarios in this dataset are described in natural languages based on NLVR (Suhr et al. 2017) and SPARTQA (Mirzaee et al., 2021)
3 Chain-of-Symbol Prompting
We propose Chain-of-Symbol (CoS) prompting for LLMs, which converts the simulated environment with natural language into a condensed symbolic representation that considers spatial relationship. In order to make constructing method of COS generalizable and reliable, we adopt three-step procedure in creating the demonstrations our CoS which can be used in any related tasks:
4

---

Published as a conference paper at COLM 2024
(i) Automatically prompt the LLMs to generate CoT demonstration in zero-shot manner
(ii) Correct the generated CoT demonstration if there existing errors.
(iii) Replace the spatial relationships described in natural languages in CoT with random symbols, and only keep objects and symbols, remove other descriptions.
We then use the CoS demonstrations guide the language model in a few-shot manner for prompting LLMs just like CoT (Wei et al., 2022).
Figure depicts an example of a demonstration of CoS produced by models. In this example, we see that both CoT and COS receive the same shared simulated spatial environment in natural language texts. CoS depicts a different intermediate thinking process than CoT. The latter represents the environments in natural language only, while the former use condensed symbolic representation that considers spatial relationship. Specifically, we use the symbol "/" to represent the spatial relationship "from the top of" here. By doing such a conversion, and removing redundant descriptions, CoS effectively improves the model performance as well reduces the inference costs with LLMs.
Figure 2 depicts examples of CoS demonstration for all three planning tasks we proposed. For NLVR-based Manipulation, we convert natural language descriptions for objects to the format of a triplet such as "(large, round, black)". For Natural Language Navigation, we represent the order of landmarks by using symbol "/ "/" to connect them. For Spatial QA task, we use a set of symbols such as "=", "~" to represent different spatial relationships, and use triplet with "(,,)" " to represent objects and their attributes.
CoS prompting has multiple properties that are attractive as a prompting approach for LLMs:
First, CoS effectively allows a neater, shorter, and condensed intermediate pro- cedure than COT. It is more structured than natural languages, hence easier for human annotators to analyze, check and correct the intermediate thinking process for LLMs.
Second, COS improves important planning tasks that current LLMs do not tackle well. It provides better representing method for spatial environments which is easier for LLMs to learn compared with natural language.
Finally, CoS clearly reduces the amount of text input into the LLMs and output from LLMs. This makes it much cheaper to access LLMs with API/GPU.
4 Experiments
In this section, we first introduce our experimental setup in Section 4.1 about the settings of different methods use, the language models, and the evaluation metrics. Then, in Section 4.2, we report the results of the three spatial planning tasks we proposed. In Section we report the results on the SPARTUN dataset.
4.3
4.1
Experimental Setup
We evaluate CoS and CoT on our proposed three spatial planning tasks and the existing SQA dataset, based on representative LLMs like (ggtGGT((gt-3.5-turi) and LLAMA-2 series. There are three prompts: zero-shot CoT, few-shot CoT, and few-shot CoS (Ours).
Zero-shot Chain-of-Thought Prompting We consider zero-shot CoT as our baseline. The reason is that we have found that our choices of LLMs naturally give their intermediate steps (CoT) in their answers, even without specifically asking them to do so. We also found that asking them to remove the thinking steps obviously degrades the results. Therefore, we allow the LLMs to generate CoT, while we do not put any demonstration to the prompt but give prompts like "Let's think step by step" just as Kojima et al. (2022). For easier evaluation, we ask the LLMs to output the final results by separating the landmarks with commas.
5

---

Published as a conference paper at COLM 2024
Table 1: The results of ChatGPT(ggt-335-turbe on Brick World. We report the results with four settings as described in Section 2.2 under both 1D and 2D scenarios, We adopt N = 5, where N₂ represents the number of demonstrations for COS and CoT. The best results are bolded. For CoS and CoT, we report the average and the standard deviation from three runs with different sets of demonstrations. Acc. represents accuracy, Pre. represents precision, and Rec. represents recall. zs-Co'T represents zero-shot CoT. We report the average number of tokens in the intermediate steps.
Model
No Shuffle
Shuffle Description
Shuffle Label
Shuffle Both
Acc.
Pre.
Rec.
Acc.
Pre.
Rec.
Acc.
Pre.
Rec.
Acc.
Pre.
Rec.
Tok.
Scemario
2s-CoT
61.0
77.2
71.9
60.4
77.5
77.5
318
63.4
59.8
28.2
58.6
55.3
CoT
81.01110
87.744.5
90.1126
71.5192 90.7
81.17.1
75.1110.1
88.01
90.1100
43.0144 71 714133 75.7116
407
Cos
96.61:199
98.0+0.8
97.7108
95.9:12
97.9
97,5:03 92.6+2.0
97.0:
95.9:111
69.7.5.1
86.7:42
83.61:16 139
2D Scemario
zs-CoT
32.7
53.8
60.6
14.8
31.9
46.9
13.0
32.0
42.3
9.8
30.4
38.4
CoT
25.0+1566
49,8+98
45.0=105
21.5+8.2
45.6-54
41.2+63
21.8+233
44.7+59
43.2+40
14.9+34
38.1+29 3644+3.5
546
Cos
60.7+19 67.2111 71.3+13 33.7+32 46.7=08 50.0=1.5 23.5+5,0
45.9=aa
63.0=1211
28.9+23
46.3=11
44.4=28 341
Chain-of-Thought Prompting This baseline uses few-shot CoT, in which we encourage LLMs think step by step, and we use five demonstrations guide the LLMs in the thinking procedure. Note that the intermediate thinking procedure is represented as natural language text, just like the Standard Prompting. Like in (Wei et al. 2022), we manually crafted five demonstrations for each task to guarantee their correctness. To guarantee the consistency and reliability of the prompts, we follow the format CoT generated by zeroshot-CoT prompting. We use these fixed five demonstrations for evaluations on each task.
Chain-of-Symbae Prompting As described in Section CoS augments the standard CoT prompting with condensed symbolic representation. While CoT has been shown to give large improvements to LLMs on various tasks (Wei et al., 2022), we argue that using condensed symbolic representations can be an alternative to describing using natural lan- guage texts. We manually converted from CoT demonstrations to CoS using the procedure described in Section 3 Five CoS demonstrations of the same examples with CoT created for each task of Natural Language Planning.
Language Models We use both Opensource LLMs (Llama-2) and Closed Source LLMs ChatGTT(Gtt..5-ttub))) for the evaluation of all tasks. We set the temperature to 0 for all the experiments throughout this paper.
Evaluation Metrics For planning tasks, we use three evaluation metrics, namely accuracy, precision, and recall. We define accuracy as the success rate in achieving the final goal. We then compute the Longest Common Sequence (LCS) between the ground truth and LLM output sequence to measure their similarity. We compute precision as the ratio of LCS against the length of the LLM output, and we compute recall as the ratio of LCS against the length of the ground truth. For spatial QA task, we only compute accuracy.
4.2 Results of Spatial Planning Tasks
4.2.1 Brick World
T the results of COS against the zs-CoT and CoT on the task of Brick World. First of all, we see that the complexity increases both from the 1D scenario to the 2D scenario and from the setting of No Shuffle to the setting of Shuffle Both, together with drop in the performance. ChatGPT with zs-CoT does not perform well, with only 9.8%

---

Published as a conference paper at COLM 2024
accuracy on the most difficult setting Shuffle Both under the 2D scenario. Although CoT brings some improvements, the performance for CoT still not satisfying, with an accuracy of 43.0% which is just below the 50% bar for setting Shuffle Both under the 1D scenario. In contrast, we see that CoS gives very good improvements on this setting (from 28.2% to 69.7%). We found that COS gives consistent improvements to all the settings on Brick World, clearly surpassing COT.. The largest gain is on the setting of Shuffle Label under the 1D scenario, with 60.8% improvements in accuracy (from 31.8% to 92.6%).
4.2.2 Further Analysis of Brick World
Randomness in the Experiments To investigate the randomness in our experiments, we run multiple trials with three different sets of demonstrations for CoT and COS. Table[] their means and standard deviations. We see general trend here that CoS usually reports a lower standard deviation than CoT (for example, a standard deviation of 1.9 for Acc. for No Shuffle under the 1D scenario for CoS, against 11.0 for Co'T). This represents that COS is more stable than CoT on Brick World.
Brick World 1D (shufile both) success nite (%)
100
CoS on the Different Language In addition to the tasks described in English, we also tested CoS on Brick World in Chinese. CoT reports 22.9% accuracy, and CoS reports 39.1% in 1D scenario of Brick World, which demonstrates the robustness of CoS to a language other than English.
None
80
60
Robustness to Different Symbols Figure S demon- strates the robustness of using different symbols for CoS. As we can see, using different symbols brings consistent performance, while not having any symbol drastically impacts the results. Among all the symbols, the comma gives the best results. We conclude that COS is robust to the selection of symbol.
40
Results on Different Language Models Figure re- ports the results on LLAMA-2 under the 1D scenario. The experimental results align with our previous con- clusion that CoS outperforms CoT obviously. Saving Tokens for Prompting One advantage featured by COS is that it reduces the number of input tokens to be fed into LLMs. For Brick World (1D scenario), CoS reduces the number of tokens for the intermediate steps from 407 to 139 (Table ! the numbers reported from OpenAl Playground(]. This subsequently saves the costs of accessing LLMs via API/GPUs.
0-
GPu-
Figure 3: Performance of us- ing different symbols for CoS Brick World 1D (Shuffle Both) in accuracy.
4.2.3 NLVR-based Manipulation and Natural Language Navigation
For both of these two tasks, we adopt almost the same experimental settings the ones for Brick World. The only difference is the evaluation metrics we report, we compute precision and recall based on the set rather than the Longest Common Sequence.
Main Results Table 2 reports the results of NLVR-based Manipulation and Natural Lan- guage Navigation with GPT-3.5-turbo. For both of these two tasks, COS reports a higher performance than CoT and zero-shot CoT prompting on all the metrics.
Saving Tokens for Prompting One advantage of CoS is that it can reduces the number of input tokens. Table reported that for based Manipulation, CoS reduces the number of tokens for the intermediate steps from 653 to 534, nearly by half of the original intermediate steps (we separate the tokens by space). This subsequently saves the costs of accessing LLMs via API/GPU, which enables easier access to the models.
Pttpss//paafomm.eeen.c..m//ppppp

---

Published as a conference paper at COLM 2024
Table 2:
The automatic evaluation results with gpt-3.5-turbo on Natural Language Navi- gation and NLVR-based Manipulation. We set N, = 5, where N, represents the number of demonstrations for prompting with COS and CoT. The best results are bolded. We report the average and the standard deviation from three runs with different demonstrations. Acc. represents accuracy, Pre. represents precision, and Rec. represents recall (precision and recall are computed with sets in this case).
Task
Natural Language Navigation
NLVR-based Manipulation
Acc.
Pre.
Rec.
Tok.
Acc.
Pre.
Rec.
Tok.
zs-CoTT
52.8
74.0
79.6
18.6
26.9
19.7
Col
53,6+288
76.3+1.1
81.7+0.8
390
64.
64.1338
81.7=1.
84.5+00.7
653
Cos
74.9+3344
87.9=1.8
86.74:3.0
239
68.4+22.3
71.2:1199
82.9+211
534
Table 3:
The automatic evaluation results with GPT-3.5-Turbo and GPT-4 on SPARTUN dataset. We apply CoT with 5 shots, and CoS with 5 shots (Ours) respectively. We report the number of tokens in the intermediate steps from demonstrations the last column.
Model
GPT-3.5-Turbo
GPT-4
Tokens
CoT-5
47.1
69.8
198
CoS-5
49.4
72.6
167
4.3 Spatial Question Answering
We also explore the effectiveness of COS in more real-world scenario, by using existing hu- man annotated spatial QA dataset SPARTUN Mirzaee & Kordjamshidi 2022). Specifically, we applied both CoS and CoT on GPT-3.5-Turbo and GPT-4. COS gains better performance and uses fewer tokens compared with COT. table we report the results of performance, and both CoT and CoS have 5 shots. It should be noticed that there are far more types of spatial relationships in SPARTUN dataset than our proposed planning tasks, so the results indicate CoS gain promising performance even when there lot of symbols to represent different spatial relationships.
4.4 Results of different size LLAMA-2
We also evaluate CoS on current open-source representative models like LLAMA-2 Touvron et al. (2023) series to further validate the effectiveness and generality of our method. We use LLAMA-2 with different size (7B, 13B and 13B). As shown in Figure when the model size small (7B and 13B), CoS cannot outperform CoT in many cases, but 70B, CoS gain clear better performance in all three tasks compared with using CoT. It can be seen that as the model size increases, the ratio of model performance to parameters for CoS has larger slope compared to CoT. This indicate that large language models might can have emergent ability of abstract symbols understanding.
Related Work
In-Context Learning Large language models (LLMs) have demonstrated remarkable few- shot learning abilities across various domains (Brown et al. 2200 Srivastava et al.. 2022), which is also called as in-context learning, leading to a paradigm shift in Al to use LLMs as foundational models for language-relater tasks, either directly through fine-tuning (Bommasani et al.) 2021 Hu et al.) 2022). While less relevant to COS, a concurrent work converts natural language into executable actions for robots with ChatGPT (Wake et al.) 2023). Another very recent concurrent work uses Symbol Tuning that replaces natural language labels with arbitrary symbols to improve in-context learning (Wei et al. 2023).
8

---

Published as a conference paper at COLM 2024
-
CoS
Co'I
Brick-1D ID
Navigation
Temporal Sequence
Solve rate (%)
80
60
0
7
13
70
13
Model Scale (B)
Figure 4: Scaling curve of CoS and CoT of Llama-2 on three tasks.
Shain-of-Thoughh Reasoning The ability of LLMs (Brown et al. 2020 Srivastava al., 2022) to perform complex reasoning tasks can be significantly enhanced by using a show known as Chaun-of-Thought (CoT) prompting, which involves providing them with interme- diate reasoning steps (Nye et al. 2021 Wei et al. 2022). Such phenomenon also generalizes to the multilingual settings (Shi et al. 2023). J. Despite the fact that CoT' is powerful, there are reports that demonstrate that Col is not always useful and that integrating CoT degrades the performance on the task of Machine Translation in their experiment. And this is possibly due to the woordd translation (Peng et al.) 2023
Spatial Reasoning Spatial reasoning over natural language texts has been an important research direction in the community (Janner et al., 2018 Mirzaee et al. 2021). Janner et al. (2018) proposes to leverage representation learning on a navigation task that requires the agent to move specific location. Rojowiec et al. (2020) proposes a new task on spatial reasoning that requires the language model generate natural language instructions for 'before' and 'after' image pairs. Mirzaee et al. (2021) proposes a new benchmark for spatial question-answering with 'which' and 'what' questions regarding the environment. In concurrent work, [Tsai et al. (2023) demonstrates that LLMs perform poorly on text-based games with question-answering tasks that require several steps of reasoning.
Navigation and Path Planning Language grounding navigation (Gu et al. 2022) refers to the interdisciplinary task that requires the intelligent agent to perceive the Visual environ- ment and guide the user to the goal location through natural language instructions (Nguyen
et al. 2019:Chen et al. 2019). Path planning (Panov al. 2018 Krishna Lakshmanan et al. 2020 reters to the tasks that require the agent to plan its own path to achieve certain goals such as the shortest path or maximizing the cleaning area, typically through the use of reinforcement learning. These areas are highly relevant to the spatial planning tasks we explored and CoS, as the spatial environments can be potentially represented by symbolic representations. We leave the investigations of these application areas to future studies.
(Sun et al. 2024
Conclusion
We found that current popular LLMs still lack abilities in complex spatial planning and understanding tasks. To this end, we propose novel method called COS (Chain-or Schii--of-Symblee Prompting) that converts spatial relationships described natural languages to condensed symbolic representations in the chained intermediate thinking steps. COS is easy to use and does not need additional training on LLMs. Extensive experiments indicate that using few-shot CoS demonstration clearly surpasses the performance of using CoT described in natural languages on all three spatial planning tasks we proposed and the representative spatial QA benchmark with even fewer tokens (down to about 1/3 tokens of the thinking steps with CoT) used in the inputs compared with CoT prompting. The performance gain is strong, by up 60.8% accuracy (from 31.8% to 92.6%) Brick World for ChatGPT.

---

Published as a conference paper at COLM 2024
Limitations Refer to the Appendix for the section on Broader Impact. In addition, only use two models to verify the effectiveness of our method due to the limited time and resources. It would be interesting to apply method to more models with different sizes to see whether there is an emergent ability of CoS for LLMs. Nevertheless, our choices of foundation models representative and they are popular LLMs.
Acknowledgments
We acknowledge funding support from the NSFC Key project 62336006, and Center for Perceptual and Interactive Intelligence (CPII) Ltd. under the Innovation and Technology Commission's innoHK scheme.
References
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterri, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei- Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Mun- yikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models. arXiv e-prints, art. arXiv:2108.0722 072558, August 2021. doi: 10.48550/ att..aavivvv..2222222222
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin Advances
Adumccss in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12530-12539, 2019. doi: 10.1109/CVPR..21990.
Jing Gu, Eliana Stefani, Qi Wu, Jesse Thomason, and Xin Wang. Vision-and-langugge navigation: A survey of tasks, methods, and future directions. Proceedings of the 60th
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
7606-7623, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653
524. URL.htpss:
https:://///. ltttt::s/aacaanthol.g. wwwpoolloyyooog////0 org/20222 acl- acl-lllg.......... 524
10

---

Published as a conference paper at COLM 2024
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6: 49-61, 2018. doi: (atll//tallaa000000 URL htttp:::/cclatthhlllee www.g.....oooooo................ org/Q18- -000
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 09-2219922222 2022.
Anirudh Krishna Lakshmanan, Rajesh Elara Mohan, Balakrishnan Ramalingam, Anh Vu
Le, Prabahar Veerajagadeshwar, Kamlesh Tiwari, and Muhammad Ilyas. Complete coverage path planning using reinforcement learning for tetromino based cleaning and maintenance robot. Automation in Construction, 112:103078, 2020. ISSN 0926-5805. doi:
URL http://wwwww.ccncceireeee... com/
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone, Lim+p: Empowering large language models with optimal planning proficiency, 2023.
Roshanak Mirzaee and Parisa Kordjamshidi. Transfer learning with synthetic corpora for spatial role labeling and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6148 6148-6165, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v /022//////222 emnlp-main.413. URRLlattss://caaathoolooo .acpooggoorrg/ogg 2022/022... comlp-mainn. ma3..413
Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi SPARTQA: A textual question answering benchmark for spatial reasoning. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 4582-4598, Online, June 2021. As- sociation for Computational Linguistics. doi: 10. http////222..aalmmmmmm
URL
https://aclantthologg org/ org/2020 2021. www.c-amaan.nnn.. 364
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 12519-12529, 2019. doi: (0.1099/CVPR.219.00228)
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computa- tion with Language Models. Xrv e-prints, art. arXiv:2112.00114, November 2021. doi: 00.48550/arxiv21150
Aleksandr Panov, Konstantin S. Yakovlev, and Roman Suvorov. Grid path planning with deep reinforcement learning: Preliminary results. Procedia Computer Science, 123: 2018.01.55 URL ktttps:/www.ce. www.sscceecedreer.cccm wttpiiirretttcoeeeeeeeeemeeeeeee womm//cience//rtiico.mm htt.c//eerrtileeeeeeeoo httpaa//e///////////00111111111/
347-353, 2018 ISSN 1877-0509 doi: http:///o/oo.....000101/////////
8th An- nual International Conference on Biologically Inspired Cognitive Architectures, BICA 2017 (Eighth Annual Meeting of the BICA Society), held August 1-6, 2017 in Moscow, Russia
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Towards Making the Most of ChatGPT for Machine Translation. arXiv e-prints, art. arXiv:2303.133: 13780, March 2023.
Robin Rojowiec, Jana Götze, Philipp Sadler, Henrik Voigt, Sina Zarrieß, and David Schlangen. From "before" to "after": Generating natural language instructions from image pairs simple visual domain. In Proceedings of the 13th International Conference on Natural Language
11

---

Published as a conference paper at COLM 2024
Generation, pp. 316-326, Dublin, Ireland, December 2020. Association for Computational inl.--ingg.. 1.38
Linguistics. URL
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large Language Models Can Be Easily Distracted by Irrelevant Context. arv e-prints, art. arXiv:2302.00093, January 2023. doi: (0.48550/arXiv.22 00093.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, 2023..
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. lyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chen- lin Meng, Chitta Baral, Chiyu Wu, Chris Callison- Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debaiyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekkn Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez- Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mari- ani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Jacon,Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenen- baum, Joshua S. Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Centteras-Ochande Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Man- aal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast,
Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik
12

---

Published as a conference paper at COLM 2024
Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McEIrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michat Swedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipcormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Mitkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Ëtta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S, Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh I'admakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the Imítation Game: Quantifying and extrapolating the capabilities of language models. arXiv e-prints, art M...:22006004155,, June 2022. doi: 10.48550/ arXiv.2206.04615.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi A corpus of natural language for visual reasoning- In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-223, Vancouver, Canada, 2017, Association for Computational Linguistics. doi: 10.18653 1111665//v1/11--203 URL htt. org/P17-2034
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jian- ing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arxiv preprint Akii:2403.147344 2024.
2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang,
13

---

Published as a conference paper at COLM 2024
Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Chen Feng Tsai, Xiaochen Zhou, Sierra S. Liu, Jing Li, Mo Yu, and Hongyuan Mei. Can Large Language Models Play Text Games Well? Current State-of-the-Are and Open Questions. arXiv e-prints, art. arXiv:2304.0286. April 2023, doi: 10.48550/ 10.48550/arxivv2044.00 231.
Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Chatgpt empowered long-step robot control in various environments: A case application, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc V. Le. Symbol tuning improves in- context learning in language models. Xiv e-prints, art. arXiv:2305.08299 May 2023. doi: 10.48550/ http://aaaiivv.....222
A Appendix
A.1
Broader Impact
CoS is prompting technique that easy to use, which can effectively improve the perfor- mance of complex planning with LLMS. It also indicates that future training with LLMs can also be well benefited by employing CoS in the training procedure to further improve LLM's planning abilities.
A.2 Extended Settings
A.2.1 Number of Tokens
We have mentioned that used white spacing for calculating the number of tokens in the intermediate thinking steps. This was a typo and in fact, we accurately measures the number of tokens using the OpenAl Playground The numbers we reported are correct and there is no need for modification.
A.2.2 Designing the Intermediate Steps
The intermediate steps we use in the demonstrations for CoT are created and modified from the zero-shot CoT from the LLMs by simply adding "Let's think step by step" before the answer. We then manually correct the intermediate steps from the outputs of using zero-shot CoT for further improvements. We attempted our best efforts in tuning the baselines, and we report the best results we achieved.
A.3 Few-shot Exemplars
In the remaining of this section, we demonstrate the few-shot exemplars used in the experi- ments in our study. We demonstrate the exemplars for both CoS and CoT.
14

---

Published as a conference paper at COLM 2024
Table 4: Few-shot exemplars for full Chain-of-Symbol prompt for brick 1D.
Question:
There bricks. For brick color yellow. The yellow brick is of the brick B. yellow brick. grabbed from top
get specific the upper brick must be removed first. How to get brick
brick
grabbed,
Answer:
A/7C
sum: A//C//B
get result C,
There of brocks. The yellow brick A top of brick C. yellow brick top of brick A. For brick C, the white. Now to get specific the upper brick must be removed first. How to get brick A?
Question:
must grabbed top to hottom, and the brick grahbed,
B7TA
sum: B//A7/C
get the result B, A.
bricks. brick The white brick top the brick D. blue brick on top of brick C. Now we have to get a specific brick. The bricks must now be grabbed from bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How get brick D?
Question:
top
C,
A7/E
E7/D
D//8
B//C
So the result
There is set bricks, The white brick on top of brick C. The white brick C on top the brick The yellow brick is on top of the brick A. For the brick B, the color white. The white brick D on top the brick F. The blue brick A top the brick D. Now we have get specific brick. The bricks must be grabbed from top to bottom, and the lower brick grabbed, the upper brick to get
Question:
removed
Answer:
EZZA
A7/D
D//F
F//C
C//B
sum: B///A///////////////
So the result as D,
There is set bricks The white brick is on lop of the brick D. The yellow brick is on top of brick A. The blue brick D on top of the brick C. The white brick E the brick G. For the brick A, the color is blue. The blue brick C is top of the brick E. The while brick G top brick B. Now have get specifie brick. The bricks grabbed, upper brack removed How get brick
Question:
grabbed top bottom, the
Answer:
D//E
C//D
G//B
So we get the result F, G.
15

---

Published as a conference paper at COLM 2024
Table 5: Few-shot exemplars for full COT prompt for brick 1D.
There a set of bricks. The yellow brick is on top of the brick D. For the brick D, the color is white. The yellow brick A is on top of the brick The yellow brick E is on top of the brick A. The blue brick C is on top of the brick B. Now we have to get a specific brick. The bricks must now be grabbed from top to bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How get brick D?
Question:
Answer: The bricks from bottom to top is B, D, C, A, E1. Remove brick E from the top of brick
A. 2. Remove brick A from the top brick C. 3. Remove brick C. from the top of brick B. 4. Now be grabbed, but we need remove to get to brick D. 5. Remove brick B from the top of brick D. 6. Now brick D is the topmost white brick and
brick B is the topmost yellow brick and
can be grabbed. So we get the result as E, A, B, D.
There a set of bricks. For the brick A, the color is blue. The white brick B is on top of the brick C. The blue brick C is on top of the brick A Now we have to get specific brick. The bricks must now be grabbed from top to bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How to get brick B?
Question:
Answer: The bricks from bottom to top is A, C, B 1. Remove brick C from the top of brick A.
Now brick A is the topmost brick and can be grabbed, but we need to remove it to get to brick B. 3. Remove brick A from the top of brick C. 4. Now brick is the topmost white brick and can be grabbed. So we get the result as C, A, B.
There is a set of bricks. The white brick B is on top of the brick A. The white brick A is on top of the brick G. The white brick G is on top of the brick D. The blue brick D is on top of the brick F. The white brick H is on top of the brick B. For the brick C, the color is yellow. The white brick E is on top of the brick C. The white brick F is on top of the brick E. Now we have to get a specific brick. The bricks must now be grabbed from top bottom, and the lower brick is to be grabbed, the upper brick must be removed first. How to get brick F?
Question:
Answer: The bricks from bottom top C, E, F, D, G, A, H1. Remove brick H from the top of brick B. 2. Remove brick B from the top of brick A. 3. Remove brick A from the top of brick G. 4. Remove brick G from the top of brick D. 5. Remove brick D from the top of brick F. 6. Now brick is the topmost white brick and can grabbed. So we get the result as H, B, A, G, E
There is set of bricks. The white brick B is on top of the brick C. The yellow brick D on top of the brick A. For the brick A, the color is yellow. The yellow brick É is on top of the brick B. The white brick C is on top of the brick D. Now we have to get a specific brick. The bricks must now be grabbed from top to bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How to get brick D?
Question:
The bricks from bottom to top is A, D, C, B, E 1. Remove brick E from the top of brick B. Remove brick from the top of brick C. 3. Now brick C is the topmost white brick and can be grabbed, but we need to remove it to get brick D. 4. Remove brick C from the top of brick D. 5. Now brick D is the topmost yellow brick and can be grabbed. So we get the result as F, B,
Answer:
C, D.
Question: There is set of bricks. The yellow brick C is on top of the brick B. The white brick B is on top of the brick A. For the brick A, the color is white. Now we have to get specific brick. The bricks must now be grabbed from top bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How to get brick
The bricks from bottom to top is A, B, C 1. Now brick C is the topmost brick and can be grabbed, but we need to remove it to get to brick B. 2. Now brick B is the topmost white brick and can be grabbed. So we get the result as C, B.
Answer:
16

---

Published as a conference paper at COLM 2024
Table 6: Few-shot exemplars for full COS prompt for brick 2D.
the on top of the brack B. The brick E is on top the brick The brick is on top of brick C. The brick G on top of the brick The brick is of the brick G. Now we have get specific brick. The bricks must now be grabbed from top bottom, and the lower brick is grabbed, upper brick removed first. How get brick
Question:
of
Answer:
can get
A//E
C//E
///HH
So we the result as E,
There is set bricks, the brick The yellow brick is on top of the brick A. The blue is on top of the brick B. The blue brick is on top of the brick C. Now we have to get specific brick. The bricks must now be grabbed from top to bottom, and if the lower brick grabbed, the upper brick must be removed first. How to brick
Question:
Answer:
We gel
A//B//C//DX
we get the result C,
Question: is brickss. the brick The white brick D is un top of brick C. The white brick E on top the brick D. The yellow brick F on top of the brick E. Now we have to specific brick. The bricks must new be grabbed from top to bettom, and the lower brick is to be grabbed, the upper brick must removed first. How to brick C?
A, is yellow
yellow
front
Answer:
We can gel CSB-AS C//D//E//E
the result D.C.
There is set of bricks. The brick P Pis of the brick R. The brick top the brick B. The brick The brick
Question:
of the beick P. the brick The brick K in front brick M. The brick B top beick D. the brick the color blue.. front the brick Now we have to get specific brick. bricks must now be grabbed
The brick is on top the brick J. The brick from top bottom, and the lower brick be grabbed, the upper brick must he removed first. How to get brick M?
Answer:
We
H//R/ ///R///DD////////C
So we the result as M directly.
There is of bricks. The brick is on top of brick F. The brick front the brick E. The brick on top of the brick K For the brick O, the color is blue. The brick G is on tup of the brick A. The brick iss in front the brick J. The brick is front of the brick O. The brick
Question:
the brick Now have get speciiic brick. The bricks now grabbed from top bottom, and the lower grabbed, upper brick
first. to
Answer:
We can gel
////////////
get directly.
17

---

Published as a conference paper at COLM 2024
Table 7: Few-shot exemplars for full COT prompt for brick 2D.
There are set of bricks. There is a brick A. The brick B is in front of the brick A. The brick C is in front of the brick B. The brick D top of the brick B. The brick E on top of the on top of the brick C. The brick G is on top of the brick D The brick H is on top of the brick G. Now we have to get a specific brick. The bricks must now grabbed from top to bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How to get brick A?
Question:
brick A, The brick
Answer: Let's think step by step:
To get brick A, we find is on top of A.
We find E on the top.
3 We need to remove brick E first, it is top of brick A.
Brick A is now accessible and can be grabbed.
So we get the result F, A.
There are set of bricks. For the brick A. The yellow brick B is on top of the brick A. The blue brick C is on top of the brick B. The blue brick D is on top of the brick C. Now have to get specific brick. The bricks must now be grabbed from top bottom, and if the lower brick is to grabbed, the upper brick must be removed first. How get brick B?
Question:
Answer: Let's think step by step: 1. To get brick B, we find C on top of B 2. We find D is top of C. We find D is on the top 4. We need to remove brick D, as it is top of brick C. 5. We need to remove brick C, as it is on top of brick B. 6. Brick is now accessible and can be grabbed. So we get the result D, C,
set of bricks. The brick P is top of the brick R. The brick J is top of the brick The brick D on top of the brick The brick R on top of the brick H. The brick front of the brick M. The brick blue. The brick Cs on top of the brick J. The brick H is in front of the brick K. Now we have to get a specific brick. The bricks must now be grabbed from top to bottom, and if the lower brick to be grabbed, the upper brick must be removed first. How to get brick M?
Question:
There
on top the brick D For the brick M, the color is
Answer: Let's think step by step: 1. To get brick M, we find there is no brick on top of brick M, So we get the result M directly.
There are set of bricks. The brick K is on top of the brick F. The brick M is in front of the brick F. The brick N on top of the brick K. For the brick O, the color is blue. The brick G on top the brick A The brick is in front of the brick The brick is front of the brick O. The brick A is on top the brick N. Now we have to get a specific brick. The bricks must now be grabbed from top to bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How to get brick O?
Question:
Answer:
Let's think step by step: 1. To get brick O, we find there is no brick on top of brick O So we get the result O directly.
There are set of bricks. For the brick A, the color is blue. The yellow brick B is in front of the brick A. The yellow brick C is in front of the brick B. The white brick D is top of the brick C. The white brick F on top of the brick D. The yellow brick F on top of the brick E. Now we have to get a specific brick. The bricks must now grabbed from top to bottom, and if the lower brick is to be grabbed, the upper brick must be removed first. How to get brick C?
Question:
Answer: Let's think step by step: 1. To get brick C, we find C is in front of B. We find is on top of C.
We find E is on top of D.
4. We find F is on top of E.
5. We need to remove brick F, as it is on top of brick E,
We need remove brick E, as on top of brick D.
7. We need remove brick as it is on top of arrckk C.
8. Brick C is now accessible and can be grabbed.
So we get the result as F, F, D, C.

---

Published as a conference paper at COLM 2024
Table 8: Few-shot exemplars for full CoS prompt for Natural Language Navigation
Question: There is set of roads and set of landmarks. The start point is bank There is road which is 200 meters long from bank A to bank There a road which 100 meters long from bank C to house H. There is a road which is 100 meters long from house H to cinema F. There is a road which is 200 meters long from cinema F to store B. There is road which is 100 meters long from store B to store G. There is road which 200 meters long from bank C to house D. There a road which is 200 meters long from house D to garden J. There is road which 100 meters long from bank A to cinema 1. There is a road which is 100 meters long from cinema I to house E. From the start point, how to reach the nearest store?
Answer: There are two stores on the map, store
store G.
bank bank C house H cinema F store B (200 100 100 200 600)
bank A / bank C house H / cinema F store B store G (200 100 + 100 200 + 100 700)
So we get the result as bank A bank C house H / cinema F store
So the answer is ACHFB
Question: There is a set roads and a set of landmarks. The start point is bank H. There is road which is 100 meters long from bank H to store E. There is a road which is 200 meters long from store E to bank C. There is road which is 100 meters long from bank C to house A. There road which 100 meters long from house A to house F. There is road which is 200 meters long from bank C to garden There a road which 200 meters long from store E. to cinema G. There is a road which is 200 meters long from cinema G to school J. There is road which is 200 meters long from bank H to school D. There is road which is 100 meters long from school D to store B. From the start point, how to reach the nearest school?
Answer: There are two schools on the map, school J and school D. bank H / store E E/ bank C garden / cinema G school] (200 + 200 200 600)
bank H school D (200)
We get the result as bank H school D.
So the answer is HD
There is a set of roads and a set of landmarks. The start point is garden B. There is road which is 200 meters long from garden B to cinema D. There is road which is 200 meters long from cinema D school J. There
Question:
road which 100 meters long from garden to school I. There a road which is 100 meters long from school to house C. There is road which is 100 meters long from house C to bank F. There is road which is 100 meters long from house C to cinema G. There is road which is 200 meters long from cinema G to store H. There road which 100 meters long from store H cinema E. There is road which is 200 meters long from cinema E to bank A. From the start point, how to reach the nearest school?
Answer: There are two schools the map, school| J and school T. garden B cinema D school J (200+200- = 400)
garden B school (100)
We get the result as garden B school (100).
So the answer is
19

---

Published as a conference paper at COLM 2024
Table 9: Few-shot exemplars for full COT prompt for Natural Language Navigation
Question: There is set of roads and set of landmarks. The start point is bank A. There is a road which is 200 meters long from bank A to bank C. There road which is 100 meters long from bank C to house H. There is road which 100 meters long from house H cinema There is a road which is 200 meters long from cinema F to store B. There is road which is 100 meters long from store B to store G. There is road which is 200 meters long from bank C to house D. There is road which is 200 meters long from house D to garden J. There is road which is 100 meters long from bank A cinema I. There is road which 100 meters long from cinema I house E. From the start point, how to reach the nearest store?
Answer: Let's think step by step 1. Start at bank A. 2. There two stores the map, store and store G. 3. The road from bank A to store B is from bank A to bank C to house H to cinema to store B (200 meters + 100 meters 100 meters + 200 meters = 600 meters) 4. The road from bank to store is shorter than the road from bank A to store G. 5. Therefore, the nearest store from the start point (bank A) is store B. So the answer is ACHEB.
There is a set of roads and a set of landmarks. The start point is bank H. There is road which is 100 meters long from bank H to store F. There is road which is 200 meters long from store E to bank C. There is road which is 100 meters long from bank C to house A. There a road which 100 meters long from house A to house There is road which is 200 meters long from bank C to garden I. There is a road which is 200 meters long from store E to cinema G. There is a road which is 200 meters long from cinema G to school J. There is road which is 200 meters long from bank H to school D. There is road which is 100 meters long from school D to store B. From the start point, how reach the nearest school?
Question:
Answer: Let's think step by step 1. Start at bank H. 2. There are two schools on the map, school
J and school D. 3. The road from bank H to school is from bank H to store to cinema G to school (200 meters+ 200 meters 200 meters 600 meters). The road from bank H school D is from bank H to school D (200 meters). 5. Therefore, the nearest school from the start point
(bank H) is school D. 6. Follow the road from bank H to school D (200 meters). So the answer is HD
There is a set of roads and a set of landmarks. The start point is garden B. There is road which is 200 meters long from garden B to cinema D. There is road which is 200 meters long from cinema D to school] J. There to school I. There a road which is 100 meters long from school to house C. There is a road which is 100 meters long from house C to bank F. There is road which is 100 meters long from house C to cinema G. There is road which is 200 meters long from cinema G to store H. There is a road which is 100 meters long from store H to cinema F. There is road which is 200 meters long from cinema E to bank A. From the start point, how to reach the nearest school?
Question:
road which is 100 meters long from garden B
Answer: Let's think step by step 1. Start at garden B. 2. There are two schools on the map, school) J and school 1. 3. The road from garden to school] J is from garden to cinema D to school| (200 meters + 200 meters 400 meters). The road from garden garden B to school I (100 meters). 5. Therefore, the nearest school from the start point (garden B) is school I. 6. Follow the road from garden B to school (100 meters). So the answer is BI
school I is from
20

---

Published as a conference paper at COLM 2024
Table 10: Few-shot exemplars for full CoS prompt for NLVR-based Manipulation.
Story: There are three boxes. In the left box, there one middle square in yellow, one middle square in black, one small square in blue, one middle square in blue. the middle box, there are one large square blue, one middle square in blue, one middle square black, one large triangle black, middle round in blue, small square in yellow. In the right box, there are one large round in blue, one small triangle in yellow, one large triangle in blue. Question:How to move all rounds middle box?
Answer:
all rounds the middle box, need to identify all the round objects the three boxes. They
middle box: (middlee, round, blue)
right (large, round, blue)
Then we can move the objects not the middle box above to the middle box one by one: (large, round, blue) middle right
Story:
There are three boxes. In the left box, there are one small round in blue, one small round in blue, one large round in black, one large square blue, one small round yellow, one small round yellow. In the middle box, there are one large square in blue, one middle triangle in yellow. In the right box, there are one large round in black, one large round in blue, one middle triangle in black, one middle triangle in black. Question How to move squares to right box?
Answer: To move all sqaure objects to the right box, we need identify all square objects
boxes. They
left box: (large, square, blue)
middle box: (large, square, blue)
Then we can move the abjects not in the right box above to the right box:
(large, square, blue) left right box
(large, square, blue) middle- right
Story:
There are three boxes. In the left box, there are one large square in blue, middle square in blue, one small round blue, one middle triangle in blue, one middle round in yellow, one large square small triangle square in yellow. In the middle box, there are black, small square black, small
small round in yellow, one middle square blue, triangle in yellow, one large round in black. In the right box, there are one small square in yellow, one small triangle in yellow, one middle triangle in black, one large round in yellow, one middle square in blue, one large square yellow. Question How to move all black objects to the right box?
Answer: To move all black objects to the right box, we need to identify all the black objects in the three buxes. They are:
#middle bux:
(small, triangle, black),
(small, square, black),
(large, round, black)
Then can move the objects not the right box above the right box
(small, triangle, black) middle- right
(small, square, black) middle right
(large, round, black) middle - right
Story: There are three boxes. In the left box, there are one middle square in yellow, one middle square in black, one small large square blue, middle square in blue, one middle square in black, one large triangle in black, one middle round in blue, one small square in yellow. In the right bux, there one large round in blue, one small triangle yellow, one large triangle blue. Question:How to move all rounds to the middle box?
square blue, middle square blue. the middle box, there
Answer: To move all rounds to the middle bux, we need to identify all the round objects in the three boxes. They ane e middle box: (middle, round, blue)
right box (large, round, blue)
Then we can move the objects not in the middle box abuve to the middle box:
(large, round, blue) right- middle
Story: There are three boxes. the left box, there are one large round in black, one small square in blue, one small square in blue, small triangle in yellow, one middle square in yellow. In the middle bux, there one large square in yellow, one middle triangle in blue, one large round in black. In the right box, there are one large black, middle triangle black. Question:How to move all black objects to the left box?
in blue, one large triangle
Answer:
all black objects the left box, need identify the black objects the three boxes. They
left box: (large, round, black)
middle box: (large, round, black),
right box (middle, triangle, black), (large, triangle, black)
Then can move the objects not left bux above the left box:
(large, round, black) middle - left
21
2. (middle, triangle, black) right- left
3. (large, triangle, black) right- left

---

Published as a conference paper at COLM 2024
Table 11: Few-shot exemplars for full CoT prompt for NLVR-based Manipulation.
Story: There are three boxes. In the left box, there are one middle square in yellow, one middle square in black, one small square in blue, middle square blue. In the middle box, there
large square blue, one middle square in blue, one middle square in black, one large triangle in black, one middle round in blue, one small square in yellow. In the right box, there one large round in blue, one small triangle yellow, one large triangle blue. Question:How to move all rounds to the middle box?
Answer: To move all rounds to the middle box, we need to identify all the round objects in the three boxes. They are: One middle round in blue in the middle box- One large round in blue in the right box
Then we can move the boxes above not the middle
the middle box one by one:
1. Move the large round in blue from the right box to the middle box. Now all the rounds are in the middle box.
Story:
There are three boxes. In the left box, there are one small round in blue, one small round in blue, one large round in black, one large square blue, one small round in yellow, one small round yellow. In the middle box, there are one large round in black, one large round in
large square in blue, one middle triangle in yellow. In the right box, there blue, one middle triangle in black, one middle triangle in black. Question:How to move all squares to the right box?
Answer: To move all sqaure objects to the right box, we need to identify all the square objects in the three boxes. They are:
One large square blue in the left box
large square in blue
middle box
Then we can move the box above with the square object to the right box:
1. Move the large square in blue from the left box to the right box. 2. Move the large square in blue from the middle box to
the right box.
Now all the square objects are in the right box.
There three boxes. In the left box, there are large square in blue, middle square blue, one small round in blue, one middle triangle in blue, one middle round in yellow, one large square in yellow. In the middle box, there are small round in yellow, one middle square blue, one small triangle triangle in yellow, one large round in black. In the right box, there are one small square in yellow, one small triangle in yellow, one middle triangle in black, one large round in yellow, one middle square in blue, one large square in yellow. Question:How to move all black ubjects to the right box?
Story:
black, one small square black, one small
Answer: To move all black objects to the right box, need to identify all the black objects in the three boxes, They are: One small triangle in black in the middle box- - One middle triangle in black in the middle box- - One large round in black in the middle box
Then we can move the boxes above not in the right box to the right box one by one: Move the small triangle black from the middle bux the right bux. 2. Move the middle triangle black from the middle bux the right box. 3. Move the large round in black from the middle hox to the right box. the right box.
Now all the black objects
Story: There are three boxes. In the left box, there are one middle square in yellow, one middle square in black, one small square blue, one middle square blue. the middle box, there are one large square blue, one middle square in blue, one middle square in black, one large triangle in black, one middle round in blue, one small square in yellow. In the right box, there one large round in blue, one small triangle yellow, one large triangle blue. Question: to move all rounds to the middle box?
Answer: To move all rounds to the middle box, we need to identify all the round objects in the three boxes. They are: Onne middle round in blue in the middle box - One large round in blue in the right box
Then we can mave the boxes above not the middle to the middle box one by one:
1. Move the large round in blue from the right box to the middle box. Now all the rounds are in the middle box.
Story: There are three boxes. In the left box, there are one large round in black, one small square in blue, small square blue, one small triangle in yellow, one middle square in yellow. In the middle box, there are one large square in yellow, middle triangle blue, in black, one middle triangle in black. Question:Hov How to move all black objects to the left box?
large round in black. In the right box, there
large square blue, one large triangle
Answer: To move all black objects to the left box, we need to identify all the black objects in the three boxes. They are:
One large round in black in the left box- One large round in black in the middle box- One large triangle in black the
right box- One middle triangle in black in the right box
Then
move the boxes above not in the left box to the left box ane by
1. Move the large round in black from the middle box to the left box. 2 Move the large triangle in black from the right box
the left box. Move the middle triangle in black from the right box to the left box.
Now all the black objects are in the left box.
22

---

Published as a conference paper at COLM 2024
Table 12: Few-shot exemplars for full COT prompt for SPARTQA.
Example
There are three blocks called A, and C, In block there small blue square far above small blue circle. There is small black triangle the right of both small blue objects. The black triange near and abuve medium
Background:
square, The medium trangle. The yellow triangle is to the left of the small blue circle. In B, which is to the right of A, there is large yellow träangle. There also small yellow triangle far below the square, In C, which to the right there small yellow triangle. The black triangle circle near above small yellow triangle. The small yellow trangle as louching top medium touching the right edge of C. There is large bloe circle far below the black triangle.
square touching the right edge of and is near and above a medium black circle. Near and to the right of the medium black circle a medium yellow
blue square near and to the left
Question:
Are all yellow objects in C abbve blue thing?
Candidate Answers! [ YYes', "No", "Do not Know") ] Answer:
Let's andlyze the information provided step by step: In block C: There a small yellow circle near and above a small yellow triangle 2. The small yellow triangle is touching the top a medium triangle is large blue cirile far below black träangle. Based on the provided information, the small yellow circle and the small yellow triangle in Block C are indeed above a blue thing, which is the large blue cinlle So, correct answer is: 'Yes..
Example
Bockgroand: There are three blocks called A, and A is the left of B and is to the left of C, In A, there is a small black triangle near and at the left of a large blue circle. The large blue circle near and to the left of a small blue carcle. The small blue square is far below a medium and to the left small yellow cirdl. The
yellow square. In there small yellow circle there is medium black triangle that blue triangle. There a large yellow square tauching bottom of small yellow circle. Near
small yellow circle near and below the large yellow square a lange black square.
the
Question: What is the relation between the large black thing and the small yellow thing?
Candidate Answers: ('eett, "right, "ahove', , "below", "hear to", "far from', "touching"; 'Do Not Know"] Answer:
Let's analyze the information provided step by step: In bluck C: 1. There a medium black triangle that is near and to the left of small yellow a blue triangle, 3. There a large yellow square touching
crdle The small yellow circle near
bettom of small
yellow circle. 4. Near and below the large yellow square a large black square, Now let's determine the relation between the "large black Whing" (large black square) and the "small yellow thing" (small yellow cticlee: 1. 'The large black square is in block C, and it described as beinng "hear and below" the large yellow square. 2 The small yellow cieele also in black C, and is deseribed being "near and to the left of the medium black triangle Hased on the provided information, the large black square is below the large yellow square and the small yellow circle medium black triangle the correct answer: "below' and "near to'.
Example 3:
medium black that for to the right of a medium blue square. The medium blue square near and the right a small yellow cirde. For above the blue square a medium yellow triangle. Near above the medium triangle a small blue circle. In B, there is large blue square that is touching the right edge of B. There is medium black triangle that touching the left side of the blue square. Near and above the medium black triangle large blue circle. Near above large blue medium black circle. The yellow triangle is touching the right edge of Far to the left of the medium black cirole is a small blue square.
Background: There
blocks called A,
the right B is the right of C. A,
is large yellow triangle. In C, there a small yellow triangle near and to the right of
Question: What block does not have any objects touching the edge of?
Candidate Answers: ['A', "B, 'C7
Answer:
Let's analyze the information provided step by step: In block I. There is medium black circle, which is far to the right of mediumm blus square. 2. The medium blue square near and the right a small yellow circle 3. Far above blue square medium yellow trangle.
Near above the medium triangle is small blue cirlle in block B: 1. There a large blue square that is touching the right edge of B. 2 There as medium black triangle that touching the left side of the blue square. 3. Near and above the medium black triangle is lange blue circle and the right a medium black circle. 2. The yellow triangle is touching the right edge of C 3. For to the left of the medium black circle is small blue square Based on the provided informatioo, Block which does not have any objects touching its edge So, the correct answer is: "A".
Near and above the large blue diecle large yellow trlangls In block C: There is small yellow triangle
Example 4:
the of B is to left C. In there large black trangle above a medium black triangle. The medium black triangle is near and above large yellow circle. Near and to the right of the large circle is small blue circle. In B1 there is medium blue citcle near and to the left of a small black square. Far below the blue circle is small yellow circle. In C, there small blue square.
Background;
blocks called
Question: What object is near the large yellow thing, the medium black triangle or the small blue circle the medium black triangle or the smal blue circle?
Candidate Answers l'eee medium black trlangle", Answer:
blue circle", "both them", / "none of them")
Let's analyze the information provided step by step: In lock A: 1. There is lange black triangle far above a medium black triangle. The medium black triangle
and above lorge yellow ciicle.3. Near and the right the large citcle small blue circle. In block
There is medium blue circle near and to the left of small black square 2. For below the blue circle is a small yellow circle. In block C: I. Block
There is small blue square. Now, let's analyze the position of the large yellow circle:
large yellow circle
ahove the medium black triangle 2 The small blue cuclle is also in Black A and is near and to the right of the large yellow circle, Based on the provided information, both the medium black triangle and the small blue circle are neur the large yellow circle in Block A, So, the correct answer is: bboth of them'.
Example
Background: There are three blue shapes in a block that we called A. Two medium blue squares are below a big blue square. First medium blue square the left of the other medium which kouching the bottom edge of black There another block below the block we call B.A big blue irtangle is touching the lop edge of black which above amall black triangle. a big blue cacle medium blue triangle. The medium triangle is bouching the botiom edge of the block. The circle below and to the left of the small triangle. It is above the medium triangle
Question: the small trlangle below the medium triangle?
Candidate Answess! YYss", 'No", "Do not Know')
23
Answer:
Let's analyze ze the information provided step by step: In block B: A big bluee triangle is touching the top edge of this block which above smalll black triangle a big blue circle and medium blue triangle 2. The medium triangle is touthing the buttom edge of the block, 3. The circle triancie It above medium triangle, Based provided information, small balack triancle
below and to left

---

Published as a conference paper at COLM 2024
Table 13: Few-shot exemplars for full COS prompt for SPARTQA.
Example 1:
Background: There are three blacks called
C. In block there small blue square far above small blue circle. There small black triangle far to the right of both small blue objects. The black triangle near and abuve a medium blue square The medium blue square touching the right edge of A and is near and above a medium black circle Near and to the right of the medium black circle a medium yellow triangle. The yellow triangle large yellow triangle. There
the left of the small blue circle In 8, which is to the right of A, there is large blue square near and to the left a small yellow triangle far below the square.
C, which is to right of B. There amall yellow circle near and above a small yellow triangle. The small yellow triangle is touching the top of medium black triangle. The black triangle touching the right ediee of C. There is large bloe circle far below the black triangle.
Question:
Are all yellow objects in above blue thing?
Candidate Answens YYss, 'NO'; 'Do not Know'1
Answer:
Let's analyze the information provided step by step to determine it all yellow objects in Block C are above a blue thing C: 1. (small,, yellow circle) T (small, yellow, triangle) 2. (small, yellow, triangle) = top of (meddian,, black, triangle) 3. (Jange, blue, circle) oot (medium, black, provided Information, (smail, yellow, citcle) I (largo, clrcke) and (small, yellow, triangle) (lange, blue,, citele))
Friangle). Based So, the correct answer is:
Example 2:
Backgroand: There are three blocks cilled A, and A is the left of B and is to the left C. In A, there is a small black triangle near and to the left of a large blue circle. The large blue circle is near and to the left of a small blue circle. The small blue square is far helow a medium yellow square. In there small yellow circle. C, there a medium black triangle that
and to the left small yellow circle. The trinngle. There large yellow square touching bottom of
small yellow citcle
the
yellow circle. Near
and below the large yellow square a lange black squa..
Question: What is the relation between the large black thing and the small yellow thing?
Candidate Answen:!"leff "right", "above', "below", 'hear to", 'far from', "truching', 'Do Not Know']
Answer:
Let's break down the relationships between the abjects step by step: A: I. (smalll, black, Irangle) ic clarge, blue, circle 2. (large, blue, circle) e [small, blue, circle) 3. (small, blue, square) (medium,, yellow, square) B: 1. (small, yellow, circley C: (medium, black, triangle) (small) yellow, carcle 2. (small, yellow, circle) < (blue, triangle) 3. (large, yellow, square) -i (amall, yellow, circle) 4. (large, black, square) I (largs, yellow, square] Now, let's determine the relation between the (laggs, black, square) and (small, yellow, cirdle: 1. C: (large, black, square) (argg, yellow, squire). Therefore, (lange, black, square) (lange, yellow, square] 2 C (small, yellow, circle) < (medium,, black, triangle) Therefore, (small,, yellow, circle) (medium, black, triangee) So, the cornect answer: "below" and "near to".
Example
Background: There are thrse blocks called A, and C. is to the right of and B to the right of In A, there is a medium black circle that for the right of medium blue square. The medium blue square near and to the right of a small yellow circle. Far above the blue square touching the right edge
medium yellow trlangle. Near medium triangle small blue circle. träangle that is touching the left side of blue square. Near and above medium black triangle is large blue circle. Near and ahove the large blue circle is large yellow triangle. In C, there is a small yellow triangle near and to the right of a medium black circle. The yellow triangle is louching the right edge C, Far to the left of the medium yellow
there
large blue square
There is medium
cirel is small blue square.
Question: What block does not have any objects truching the edge it?
Candidate Anewensl"A A", "B"'"""
Answer:
Let's analyze each block step step determune which does have any abjects touching its edge: A: I. (mediiu,, black, circle) ees [medium,, blue, square). 2. (medium,, blue, square) > (small, yellow, circle). 3. (medium, yellow, trlangle) sot (hlue, square). 4. (small, blue; (argg, blue, square) edge
(medium, black, triangle) (blune square). (lange, blue, circle) (medium, black, Irtangle). 4. (large, yellow, triangle) (large, blue, circle). C: I. (small, yellow, triangle) > (medium, black, circley, 2. (yellow, triangle) mo edge of C. (small, blue, square) ooc (mediium, Black, circle) Based on the provided information, let's determine which block does not have any objects bouching its edge It's Block B So, the correct answer is: 'A".
circle) (medioom, triangle).
Example 4:
Background There three blocks called A, and is the left of B and is to the left C. In A. there lage black trangle far above a medium black triangle. The medium black triangle near above large yellow cirile Near the right the large circle small blue circle. In B, there is medium blue circle near and to the left a small black square. Far below the blue cincle is a small yellow circle. In C, there is small blue square.
Question: What object is near the large yellow thing. the medium black triangle the small blue circle the medium black triangle or the small blue cirele?
Candidate Answers] the medium black triangle", "the small blue cirele", 'both of them', 'none them')]
Answer:
medium black triangle, or the small blue circle: A: (large, black, tranglee) oof (medium,, black, triangle). (medium, black, triangle) (large, yellow, circle) (smail, blue, strcle) >s (lrrge,circle)
Let's break down information step by step to determine which object is near lange yellow
(medium, blue, circle) (small, black, square), 2. (small, yellow, circle) mot (blue, circle). C: 1, there is small blue square Now, let's analyze the position of the (largs, yellow, circley L. A: Darge, yellow, circle) (medium, black, triangle), A: (small, blue, cirde) Dlarge, yellow, cirdle. Based on the provided information, A: (medium, black, triangle) (large, yellow, cirdl) and (small) blue, circle) (langs) yellow, circle), the correct answer 'both of them!.
Example 5:
medium blue squares are below a big blue square. The first medium blue square the left of the other medium one which touching the bottom edge black A. There is another block below black we call it B. A big blue triangle is touching the top edge of this black which is above small black triangle, a big blue circle and medium blue triangle. The medium triangle louching the bottoe edge of the block. The circle below and to the left the small triangle. above the medium triangle.
Background: There are three shupes in black we call
Question: Is the small triangle below the medium triangle?
Candidate Answers Yes' "NO", not Know")
Answer:
Let's analyze the arrangement of objects in block to determine the small triangle is below the medium triamgler B: (big blue, triangle) lop edge B 2. (bsg. blue, trangle) 7 (small, black, trianglel, (big, blue, caclet and (mediuum blue, triangle) 2. (bsg. blue, cacle) UK (small, black triangle) 3. (big, blue, circle) L (big, blue, Irangle) 4 (medium, blue 2luee2duugg 2dngle) bottom edge Based on the provided information, the small black triangle above medium blue triangle So the correct answer is: 'No.'

---

Published as a conference paper at COLM 2024
Table 14: Comparison of CoS(above) and CoT(below) prompts for BBH's Temporal Se- quence task.
Example
Today, Emily went the museum. Between what times could they have gone?
We
Emily woke
Elizabeth saw Emily reading at the library from 2pm to 9pm.
Jessica Emily watching movie the theater from 9pm
Leslie saw Emily waiting at the airport from 5pm to 6pm.
William Emily buying clothes the mail from 7pm.
The museum was closed after 7pm.
Between
could Emily gone
Options:
(A) 1pm to 2pm
(B) 6pm to 7pm
(C) 5pm to 6pm
(D) 2pm to 4pm
Let's think step by step.
Jpm/wokke-pp
ppm-2pm//fiee
2pm-spn/rading
ppn- Son-Spmmwwatching
spm/waiting
6pm-7pm/uuying
7pmnktouure
so 2pm/pmm/// /free. ffree, So the answer (AI)
Today, Emily went museum, Between
they gone?
We know that
Emily up at 1pm.
Elizabeth saw Emily reading the library from 2pm to 4pm.
Jessica saw Emily watching a
4pm 5pm.
Leslie Emily waiting the airport from to
William saw Emily buying clothes at the mall from 6pm 7pm.
The museum clased after
Between what times could Emily have gone to the museum??
Options:
(A) 1pm to 2pm
6pm 7pm
(C) 5pm to 6pm
(D) 2pm to 4pm
A: Let's think step by step.
Wake-ap time: 1pm
1pm-22m:: free)
2pmn-4pen reading library
#pm-5pm watching movie the theoter
5pm-6ppm: waiting airport.
6pm-7pm: buying clothes at the mall
The museum clesue time 7pm.
The only time when Emily could have the museum 2pm, So the answer
25


Reviews:
[{"review_id": "jcmnPEkUIp", "reviewer": "Reviewer_ST4m", "summary": "The paper regards the implementation and evaluation of a symbolic language to communicate with a large language model. The translation of the natural language query or instruction to its symbolic form is done manually by the user. The results are compared with those obtained with zero-shot chain-of-thought prompting and with few-shot chain-of-thought prompting where the prompts are also manually drafted but are in natural language.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks for your reply,\n\nFor your concern: **The expressiveness of the symbolic language is not theoretically defined and evaluated.**\n\nCoS is an empirical method, so we doubt the necessity of a theoretical definition of this method, while we clearly described the detailed pipeline of constructing CoS and transforming from CoT to Cos in Section 3 and in Appendix A.2.2. \n\nFor your concern: **The paper claims that the symbolic language is easy to use, but a users’ study that supports this claim is lacking.**\n\nOur claim about 'easy to use' is based on the fact that our method doesn't need additional training and doesn't need a complex pipeline, it only needs simple transformation from CoT. Our method requires fewer token numbers compared with CoT. Because of its simplicity and using structured symbolic languages, it is easy for users to read and understand (see Figure 1).\n\nFor your concern: **The contribution towards novel techniques to improve language modelling is not clear. The paper rather suits a human-computer interaction venue.**\n\nThis conference is not only limited to topics about 'improving language modelling', and as you can see, we have chosen the research area of this paper as Alignment, Evaluation, LMs and interactions, which we think is aligned with the content of our paper based on https://colmweb.org/cfp.html. Using symbolic languages for in-context learning and prompting is worthy of exploration, and we believe the LLM community is open to diverse and meaningful topics like this.\n\nFor your concern: **'It is not surprising that the obtained results improve compared to zero- and few-shot chain-of-thought reasoning as the symbolic language has less complex patterns than natural language and is an unambiguous controlled language to attend to when querying the LLM, but it requires knowledge engineering expertise from the user.'**\n\nIn fact 'symbolic languages have less complex patterns and an unambiguous language' is also our motivation for using symbols rather than natural languages, which we described in the introduction section.  'requires knowledge engineering expertise from the user', from the construction process of CoS in the paper, we only use some random symbols to replace objects relationships in CoT, which can't indicate it requires knowledge engineering expertise. \n\nAgain, Thank you for your effort in evaluating the paper. Have a nice day.\n\nBest Regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe idea is original.\n-\tThe idea is evaluated with different tasks (brick world spatial reasoning, natural language visual reasoning, natural language navigation and spatial question answering).", "reasons_to_reject": "-\tThe expressiveness of the symbolic language is not theoretically defined and evaluated. It is not clear whether it exhaustively covers complex spatial configurations of objects in a real world settings.\n-\tThe paper claims that the symbolic language is easy to use, but a users’ study that supports this claim is lacking.\n-\tThe contribution towards novel techniques to improve language modeling is not clear. The paper rather suits a human-computer interaction venue.\n-\tIt is not surprising that the obtained results improve compared to zero- and few-shot chain-of-thought reasoning as the symbolic language has less complex patterns than natural language and is an unambiguous controlled language to attend to when querying the LLM, but it requires knowledge engineering expertise from the user.\n-\tThe paper seems to be written in a hurry and contains typos and grammatical errors.\n- Overall, the paper is not yet mature to be published but the authors are encouraged to improve their work.", "questions_to_authors": "- See the remarks above.", "extra_scores": {}}, {"review_id": "tZAiVZLGoh", "reviewer": "Reviewer_C6LX", "summary": "This paper focuses on spatial reasoning using LLMs and it proposes a prompting method called \"Chain-of-Symbol (CoS)\", which follows the same idea of Chain-of-Thought (CoT) but expresses the reasoning process using concise symbols. This approach was evaluated on four datasets: Brick World, NLVR-based Manipulation, Natural Language Navigation, and SPARTUN. Few-shot CoS was shown to outperform few-shot CoT in most cases.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\nThanks for your reply. \n\n**For your first concern**,\n\n\n(1). Just as we described in Appendix A.2.2, we create exemplars based on the zero-shot CoT results, and manually convert them to the CoS version, and just as we described there, 'For the results of templating We attempted our best efforts in tuning the baselines, and we report the best results we achieved.' There are versions which are concisely parallel but we find using CoT is even lower than the current version, but your point makes sense, and we will add a more fair comparison in the latter version of the paper. In Table 8 and Table 9, there is only one place (just as you mentioned) which is not aligned with each other, this is due to some generation mistakes of LLMs, but other exemplars are aligned, which might mitigate this mistake and unfairness, and at the same time our results are significant.\n\n(2). We want to conduct experiments using GPT-4 but unfortunately we don't have enough budget for the API of GPT-4.\n\n(3). For the choice of dataset, we choose NLVR-based manipulation and Natural Language Navigation because they are both representative benchmarks and scenarios of reasoning/planning, and they are both easy to create and easy to understand. As you can see in the CoT results in Table 2 for these two tasks, they are still challenging, and it can also indicate that the CoS method can generalize to diverse scenarios not only in pure spatial reasoning tasks.\n\n**For your second concern**, \n\nFor the novelty of CoS, we are indeed different with Program of Thought [1] : they generate code to execute the generated program (which leverage external tools), while we only uses LM **itself** to reason through symbolic languages. \n\nThe core novelty of this idea is that we are the first to discuss how to directly use symbols (without any other external tools or executors of programs) for language models, and we find it can both save tokens (inference time) and improve performance in some planning and reasoning tasks.\n\nAgain, Thank you for your effort in evaluating the paper. Have a nice day.\n\nBest Regards,\n\nAuthors\n\n[1] Chen, Wenhu, et al. \"Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.\" Transactions on Machine Learning Research (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a new method called Chain-of-Symbol (CoS), which offers the benefit of shorter prompts and hence more cost- and time-efficient than CoT. It was also shown to outperform CoT.", "reasons_to_reject": "1. The experimental designs need more justification.\n\n(1) Concern about the validity of the baseline: To show the effect of using concise symbols, a more fair comparison should design CoS and CoT to follow _the same reasoning process_ while being different _only_ in their expression (i.e., symbols vs. texts). Only in this more controlled comparison can one see if symbols are more effective than textual reasoning. However, prompts in the Appendix have shown inconsistent designs between CoS and CoT. For example, on Brick World (Table 4 vs. Table 5), while CoS reasons about the spatial relations before giving the answer sequence, CoT directly presents the answer sequence. Similarly, on Natural Language Navigation (Table 8 vs. Table 9), while CoS enumerates the stops of each path and shows its distance calculation before reaching the conclusion of the shortest choice, CoT seems to miss some of the calculation details (e.g., \"4. The road from bank A to store B is shorter than the road from bank A to store G\" without saying the calculation of the latter path). As a result, in the current less fair experimental setup, it is hard to understand whether the gain of CoS over CoT is an effect of using symbols, or if it is simply due to prompting the LLM with different reasoning logic. \n\n(2) Experiments were performed using GPT-3.5-turbo, so it is unclear if the advantage of CoS holds for more advanced LLMs such as GPT-4.\n\n(3) Concern about the choice of datasets for spatial reasoning: I have found the use of \"NLVR-based Manipulation\" and \"Natural Language Navigation\" as benchmark datasets for spatial reasoning confusing. They do not seem to focus on spatial reasoning. For \"NLVR-based Manipulation\", the involved \"spatial\" aspect is limited to left vs. middle vs. right boxes, whereas the main challenge of this dataset seems to be about \"perception\" (e.g., recognizing the color, size, and shape of objects). For \"Natural Language Navigation\", it is more about motion planning while there's no obvious challenge in inferring the spatial relationship among intermediate stops on a path.\n\n2. Novel of CoS: How does CoS compare to Program of Thought, which also uses symbols to represent the reasoning process of problem solving?\n\nChen, Wenhu, et al. \"Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.\" Transactions on Machine Learning Research (2023).", "questions_to_authors": "Please address my concerns in Reason to Reject.\n\nFigure 2: some texts are not properly displayed in \"Natural Language Navigation\".", "extra_scores": {}}, {"review_id": "Iku4b6XR9I", "reviewer": "Reviewer_cm3f", "summary": "The work proposed the new in-context learning method called Chain of Symbol (CoS)  which utilizes symbolic representations for spatial relation objects rather than natural language representations.  They claim the proposed technique is more effective and reliable than commonly used CoT techniques.  They prove the claim using four tasks involving spatial understanding of various dimensions (1D, 2D, and 3D spatial relations) with closed-source and open-source LLMs.  The results indicate that CoS completely outperformed in performance and resources on all tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nThanks for your reply and efforts in evaluating our paper. There could be some miscommunication we would like to clarify.\n\n**For your first concern: The improvement rapidly drops on the complex tasks compared to simple ones, which may show indifference between symbolic representation and natural representation in the complex setting** \n\nThis paper encourages the community to pre-train LLMs in a symbolic manner in order to higher performance on complex tasks.\n\n**For your second concern: Applying to the realistic scenario will be challenging because the symbolic representation is not straightforward to obtain with reasonable coverage of concepts**\n\nThis is true for some tasks. Yet, in addition to spatial reasoning, coding and logical reasoning in first-order logic can also benefit from CoS. They are important useful tasks where CoS could be potentially easily applied.\n\n**For your third concern: According to LLama2 results, the proposed techniques only out-performed CoT with the largest model**\n\nThis paper encourages the community to pre-train LLMs in a symbolic manner so that smaller LLMs will also benefit from CoS.\n\n**For your first question: You provided the result of GPT-4 on the SPARTUN dataset; what are the results of GPT-4 on the other benchmarks?**\n\nWe will include more results in the near future with more resources.\n\n**For your second question: Can you visualize the Llama2 results in the same format and details of the GPT results? More details will be helpful since the figure 4 plots is not very conclusive.**\n\nThis will be included in the camera-ready version.\n\n**For your fourth question: Would it be possible to apply CoS approach to the other types of reasoning beyond spatial?**\n\nSee the reply to the second concern above.\n\n**For your fourth question.**\n\nWe think the results in Table 6 might provide insights into your question. Yes, the model still understands the case you suggested.\n\n**For the fifth and sixth questions.**\n\nThere can be some miscommunication. CoT is generated by GPTs automatically, and we convert them manually into CoS for a fair comparison.\n\nAgain, Thank you for your effort in evaluating the paper. Have a nice day.\n\nBest Regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed technique provides another strong representation for performing spatial reasoning on LLMs. \n The notable reasons are stated below:\n - The proposed method outperforms CoT in several benchmarks including both simple and complex spatial relations as well as resource-saving\n - The benchmarks used in the experiment have a good coverage of application regarding spatial relations\n - Provide the results that support the performance of the model on both open-source and close-source LLMs \n - Based on analysis of one of the benchmarks, the proposed method is effective in different language setting\n - Easy to apply in-context learning", "reasons_to_reject": "- The improvement rapidly drops on the complex tasks compared to simple ones, which may show indifference between symbolic representation and natural representation in the complex setting\n - Applying to the realistic scenario will be challenging because the symbolic representation is not straightforward to obtain with reasonable coverage of concepts\n - According to LLama2 results, the proposed techniques only out-performed CoT with the largest model", "questions_to_authors": "- You provided the result of GPT-4 on the SPARTUN dataset; what are the results of GPT-4 on the other benchmarks? \n - Can you visualize the Llama2 results in the same format and details of the GPT results? More details will be helpful since the figure 4 plots is not very conclusive. \n - Would it be possible to apply CoS approach to the other types of reasoning beyond spatial?\n- In the analysis part of BrickWorld, did you experiment with different symbolic representations? What would be the expected results in more complex setting if the symbolic representations are changed? The extreme case would be using <- to represent left instead of right. Would the model still understand it?\n- Can you justify using GPT for creating the initial explanation compared to manually crafting CoS? is the motivation reducing the human annotation effort?\n- For CoT, the experiment is performed using manually crafted CoT. What about the results from CoT if you follow the same steps as CoS?", "extra_scores": {}}, {"review_id": "fWxXVxgabI", "reviewer": "Reviewer_iTYN", "summary": "This paper proposes to use abstract chain-of-thought prompts that simply use symbols, rather than natural language. This results in notable improvements across a variety of planning and spatial reasoning tasks. This seems like a useful insight that will improve applications of language models, and is also somewhat conceptually interesting.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you very much for taking the time to evaluate the paper, as well as for your appreciation towards our paper.\n\nWe sincerely thank you for your suggestions for the potential development of CoS. We will definitely incorporate them as soon as we obtain the results in the near future work. They are:\n\n1. Deeper analysis of when and why CoS is beneficial.\n2. Check carefully on the number of correct steps with CoS and CoT.\n3. Check the next token entropy to dig CoS.\n4. Evaluate CoS on more challenging benchmarks like https://arxiv.org/abs/2206.10498.\n5. Adds more comparison to Fig 3.\n\nAgain, Thank you for your effort in evaluating the paper. Have a nice day.\n\nBest Regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Clear, simple idea — that illustrates a more general point about natural language not necessarily being the most effective way to get language models to reason.\n     - I actually felt that this point could be more strongly emphasized than it is; perhaps including natural language in the CoT \"distracts\" the model with the fuzzier processes of next word prediction and/or makes the distribution higher entropy, while the CoS prompting is more formulaic and rigid and therefore avoids one or both of these. It seems worth discussing these possibilities, and ideally investigating them in some way (see below). \n* Strong improvements on the tasks evaluated.\n    - as a bonus, saves some compute/tokens.\n* Some nice robustness & scaling experiments.", "reasons_to_reject": "* The paper doesn't quantify thoroughly *when* and *why* CoS is beneficial. While there are a diversity of comparisons, it would be useful to dig into some of them in more depth; for example, are there characteristic patterns of errors that CoT makes but CoS is more likely to avoid? Conversely, where CoS makes an error, does CoT tend to make the same error, or are they uncorrelated? Etc. \n     - It would be even better if it were possible to link some of these analyses to the CoS itself; e.g. showing that it contains more correct steps than the CoT.\n     - If possible, it would also be very interesting to quantify the distributional effects of the CoT to CoS switch along the lines of the proposed hypothesis above: for example, is the distribution over next tokens higher entropy in CoT than in CoS? Could that help explain the benefit?\n* It would be nice to see some evaluations on more challenging benchmarks like https://arxiv.org/abs/2206.10498\n* If I understand correctly, in Fig. 3 \"None\" means no symbol or space between the letters; it seems to me that having a space would also be a useful comparison condition to run (to see if the eponymous symbols are actually critical, or if it's just necessary to have *some* break in between to avoid tokenizer weirdness).", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose chain-of-symbol prompting for
spatial-reasoning-from-text tasks. The method involves manually
constructing more compact/symbolic versions of chain-of-thoughts
generated by models in a zero-shot fashion, and using these updated
representations as in-context examples.  This method outperforms a
baseline of not making the proposed symbol substitution, sometimes
significantly, with gpt-35-turbo, and llama-2 (and does so using fewer
tokens).

Reviewers appreciated the simplicity, effectiveness, and efficiency of
the proposed method. A few themes among the weaknesses:

- Some reviewers (e.g., iTYN) hoped for more experiments about
  when/why the CoS method works, while others (e.g., C6LX, cm3f) hoped
  for more models covered, while others (e.g., iTYN) hoped for more
  datasets

- Some reviewers highlighted presentation issues (e.g., ST4m brought
  up the "easy to use" discussion)

In my view, the two biggest remaining shortcomings are:

- Why does CoS help? Are there error analyses that could be conducted
  on the individual instances where improvement occurs?

- Does CoS really generalize to other similar tasks? Or to tasks
  beyond spatial reasoning from text?

Concerns were mostly addressed in conversation. Overall, this work
provides (another) striking illustration of when/how representations
can matter for LLMs --- while CoT and CoS represent the same
information, CoS clearly does so in a manner more accessible to
LLMs. The promised additional experiments the authors mentioned in
their response are likely to provide additional missing context as
requested by reviewers.
Decision: Accept