Chain of Thoughtlessness? An Analysis of CoT in Planning
Authors: Kaya Stechly
Karthik Valmeekam
Subbarao Kambhampati
Venue: NeurIPS 2024 poster (2024)
Keywords: LLMs
Planning
Reasoning
Chain of Thought
Primary area: other
TL;DR: We carefully examined the performance of Chain of Thought techniques on classical planning problems and found that, contrary to previous claims, they do not lead to generalizable improvement..

Abstract: Large language model (LLM) performance on reasoning problems typically does not generalize out of distribution. Previous work has claimed that this can be mitigated with chain of thought prompting--a method of demonstrating solution procedures--with the intuition that it is possible to in-context teach an LLM an algorithm for solving the problem.
This paper presents a case study of chain of thought on problems from Blocksworld, a classical planning domain, and examines the performance of two state-of-the-art LLMs across two axes: generality of examples given in prompt, and complexity of problems queried with each prompt. While our problems are very simple, we only find meaningful performance improvements from chain of thought prompts when those prompts are exceedingly specific to their problem class, and that those improvements quickly deteriorate as the size n of the query-specified stack grows past the size of stacks shown in the examples.
We also create scalable variants of three domains commonly studied in previous CoT papers and demonstrate the existence of similar failure modes.
Our results hint that, contrary to previous claims in the literature, CoT's performance improvements do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts. This spotlights drawbacks of chain of thought, especially the sharp tradeoff between possible performance gains and the amount of human labor necessary to generate examples with correct reasoning traces.

Chain of Thoughtlessness? An Analysis of CoT in Planning
Kaya Stechly"
Karthik Valmeekam" SCAI, Arizona State University kvalmeekOasu.ede
SCAL, Arizona State University kstechleasu.ce edu
Subbarao Kambhampati SCAL, Arizona State University raošasu. edu
Abstract
Large language model (LLM) performance on reasoning problems typically does not generalize out of distribution. Previous work has claimed that this be mitigated with chain of thought prompting- a method of demonstrating solution procedures-with the intuition that it is possible to in-context teach an LLM an algorithm for solving the problem. This paper presents a case study of chain of thought on problems from Blocksworld, classical planning domain, and examines the performance of two state-of-the-are LLMs across two axes: generality of examples given in prompt, and complexity of problems queried with each prompt. While our problems are very simple, we only find meaningful performance improvements from chain of thought prompts when those prompts are exceedingly specific to their problem class, and that those improvements quickly deteriorate as the size of the query-specified stack grows past the size of stacks shown in the examples. We also create scalable variants of three domains commonly studied in previous CoT papers and demonstrate the existence of similar failure modes. Our results hint that, contrary to previous claims in the literature, CoT's performance improvements do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts. This spotlights drawbacks of chain of thought, especially the sharp tradeoff between possible performance gains and the amount of human labor necessary to generate examples with correct reasoning traces]
1 Introduction
While originally designed for text completion, Large Language Models (LLMs) have shown promise on diverse of unrelated tasks. While initial anecdotal results were unexpectedly impressive (8)
followup systematic studies showed tuat-outside of limited, non-generalizable classes of problems- these models generally perform poorly on basic, multi-hop reasoning tasks [17 ranging from arithmetic [35] and logic puzzles [1] to constraint satisfaction [42] 2 and classical planning [47).
At the same time, the subfield of prompt engineering [36] has grown rapidly, promising improvements in performance without retraining. core tenet of this subfield is that LLMS are capable of powerful
equal contribution
Ressurcess and source code for planning experiments can be found at https://github com/ http//arbikk///2//c--llaaamm and for other domains https://gittub..comtt wtwp:/thww...om//sttehh...../
cot not-scheduling
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

in-context learning [12 [6], that is, capable of intelligently using additional context provided in prompt to correctly respond to queries that would otherwise be answered incorrectly. Generally, this requires operationalizing algorilbnicpproceuuudd advice, and, in principle, learning such procedures includes being able to effectively apply them beyond syntactically similar instances.
The foundational method for inducing in-context learning is the chain of thought approach, which has been claimed "unlock the reasoning abilities of LLMs" [50]. To create a chain of thought (CoT) prompt, a user annotates similar problems with intermediate reasoning steps and prepends them to the standard prompt. These annotations are meant as demonstrations, intended to teach a procedure applicable to both the examples and the new query. When prompted like this, the LLM is expected to output a similar series of reasoning steps prior to the new answer. Numerous studies have claimed that this procedure significantly enhances LLM performance in complex reasoning tasks 54 56 52 43). However, in general it unclear how "similar" the examples need to be to the problem, how broadly any given chain of thought prompt will apply, and-most importantly-how much human effort is necessary to craft prompts specific to each problem subclasses. Followup work has claimed that merely adding magic phrases ("let's think step by step") to every prompt is sufficient for some improvement [26]. While in some domains, this technique has proven to be even more brittle than manual CoT, it has achieved the same performance increases in others, hinting that improvements observed with CoT may not indicate as much about LLMs' general in-context learning abilities as previously thought.
49
We are interested in the tradeoff between possible performance gains from chain of thought prompt engineering and the amount of human labor necessary to generate examples with useful reasoning traces. Ideally, a properly constructed prompt should teach the L.LM how to robustly generalize a basic algorithmic procedure in order to increase performance on large class of problems, thereby converting a modest amount of human teaching effort into a significant capability boost. Unfortunately, this only seems to be possible to very limited extent [J4]
In the current work, we examine the limits of chain of thought in solving classical planning problems. Test domains commonly used in previous chain of thought studies (e.g. GSM8K [10], CommonSense QA (44) present two significant issues: (a) they lack a systematic method to scale instances, which is essential for evaluating whether LLMs can extend provided procedures to larger instances of the same type, and (b) due to their static nature, are more likely to be well-represented on the web(51], increasing the chance that they were part of LLM training data, a factor which could obscure the true reasoning capabilities of L.Ms.. Planning is well-studied kind of sequential tasks an agent with devising plan that takes given initial state to pre-specified goal state. New, diverse, and unique problem instances are easy to generate, but potentially hard to solve.
eeciiiionmmaiing which
We focus on Blocksworld, simple commonsense domain widely recognized and utilized in Interna- tional Planning Competitions (23), where a set of blocks in an initial configuration must be rearranged step-by step-by-step into goal configuration. For a subset of our results, we simplify even further, and only consider problem instances where every block starts on the table and the goal is a single stack of blocks. These instances require very minimal reasoning: one need only figure out which block is on the bottom, and then stack the remaining blocks in the sequence directly defined in the goal. For 3 < n < 20, we generate a variety of instances where the goal requires a specific n height stack, while providing examples of how solve 2 and 3 height instances.
We consider different chain of thought prompts, where each is more specific-and provides more problem-specifie knowledge-than the last: a zero-shot variant, a general progression proof, a subopti- mal algorithm specific to Blocksworld, a table-to-stace specific simplification of that algorithm, and a lexicographic version of the simplification. The most general could be applied to any problem, while the least is specific to an easier version of the stacking problem. The three duman--rafiee prompts all teach algorithms which could, in principle, solve any of the instances they are tested on. We test on three state of the art models: GPT-4 (3, Claude-3-Opus,
S and GPT-4-Turbo.
Our results reconfirm that LLMs generally incapable of solving simple planning problems [47] and demonstrate that chain of thought approaches only improve performance when the hand-annotated examples and the query are sufficiently similar the current query. As goal stack size increases, accuracy drops drastically, regardless of the specificity of the chain of thought prompt. As generality of the prompt increases, performance on even the smallest goal stacks also decreases, and often falls short of standard prompting. Even state of the art extensions of CoT (like self-consistency (49). show similar or sometimes even worse performance. Overall, this case study calls into question
2

---

assumptions about the generalizable effectiveness of chain of thought, and suggests that LLMs do not learn new, general algorithms in context, but instead rely on some form of pattern matching to achieve prompt-design--seeciie performance increases. This in turn increases the burden on humans giving advice.
To better compare to previous work, we construct scalable versions of three previously studied synthetic problems-Coin Flip, Last Letter Concatenation, and multi-step arithmetic 44 50 26 48
and replicate reported chain of thought prompts. While these domains do not have a corresponding notion of prompt granularity. they do cover range of difficulties. When testing on 4PT----Trbb, We see a similar lack of generalization on these problem sets as we saw in Blocksworid.
In the rest of this paper, first review related work, then describe the chain of thought approaches we have developed in the context of planning, analyze the overall effectiveness of chain of thought prompting on Blocksworld problems, and extend results to three synthetic tasks well-represented in the COT literature.
2 Related Work
Modifying text prompts to elicit intermediate problem-solvingg steps from LLMs originally took the form of scratchpads (33 (50) proposed a similar prompt style in natural language, dubbing this approach chain of thought (CoT), and claiming that-with some human hand-annotation of examples-this not only boosts performance without retraining, but "allows reasoning abilities to emerge naturally". They argued that by merely interspersing intermediate reasoning steps in natural language into examples, they were inducing the LLM to "learn via a few examples", motivating this idea with anthropomorphizatioss (" ("Consider one's 's own thought process when solving a complicated reasoning task such multi-step math word problem"). (26) argued that some of the performance of CoT could be retained without providing any examples, and instead just appending the magic phrase "let's think step by step" to the end of prompt. This has been called zero-shot COT.
However, CoT has long been known to be imperfect and incomplete. Previous work has investigated improving the consistency of CoT through self-consistency [49], multi-agent debate 13. least-to- most prompting (55). deductive verification 228. and other approaches. Unfortunately, many of these involve prompting the LLM multiple times for single problem, which can balloon the cost of inference. Other work has examined the possibility of reducing or removing the need for human annotation of examples by using LLMs to generate their own examples automatically [54 9 To avoid
well-known issues with the brittleness of LLM self-verification and self-teaching 42 22 20 19 24].
we restrict this paper's scope manually written chains of thought.
Previous papers have analyzed CoT from multiple perspectives [5] 37 finding that there only a loose relationship between the presented chain and the final answer provided annotations has little effect on resultant performance 38 are also known to be unfaithful to the underlying reasoning process 29 25 M In particular, the
6 and that the correctness of LLM p-roduced chains of thought
way the examples are presented can bias a model into giving some answer (c.g. if all the example answers are A, the model will be more likely to output A), but its CoT will not reflect this [45).
Motivated by claims that CoT prompts allow models learn in context how to reason-that is, to learn how to execute numan-specifeed algorithms- we focus on CoT prompting's out-of-domain generalization. [4] previously showcased a lack of generalization in multiplication, puzzles, and a number sequence problem, even when the model was fine-tuned on CoT examples. However, they only examined one set of prompts, did not experiment with levels of prompt specificity, and were much more interested in local failures of compositionality arising from cumulating error. More broadly, previous work has examined generalization limits of LLMS in arithmetic tasks 335 formula simplification 34 and theorem proving 4
While early accounts claimed LLMs. despite not being trained for it, were capable of reasoning and
planning (8) later work showcased serious brittleness across these domains [7]. [50] claims that
"standard prompting only provides a lower bound on the capabilities of large language models", with proper prompting allowing reasoning to "emerge naturally." " Recent work seems to maintain this optimism [7]. In this paper, we examine the effectiveness of CoT in the context of classical planning problems, which have well-defined and algorithmically checkable ground truths, can be generated with arbitrary size and difficulty, and are unlikely to be the training data. If CoT induces more than just pattern matching, and can in fact teach LLMS to perform generalizable, compositional reasoning.
3

---

then we should expect that to be reflected in robust and maintainable improvements on a simple commonsense benchmark set like Blocksworid, and we should expect these results hold for scaled variants of the very benchmarks tested in 50 and later CoT work.
3
Background
Classical planning problems task a planner with finding a sequence of actions that, when exccuted, will take agent from a pre-specified initial state desired goal state. STRIPS planning is discrete, deterministic formalism that encompasses this class. Problems are represented using the Planning Domain and Definition Language (PDDL) (0) and have long featured in various planning challenges and competitions. Our main experiments are all on the Blocksworld PDDL domain.
A PDDL specification consists of three components. The domain doesn't I change between problems and consists of a set of predicates-wiie whose truth values deseribe the state of the world-and set of irtions-defined by their preconditions and effects- that the agent is allowed to take. The initial state a list of predicates that are true at the outset of the specific problem (an example predicate: "Block A is on the table"). The goal is a boolean expression of predicates (a goal: "Block A is on Block B.").
plan is sequence of actions. The solution to a PDDL problem a plan in which the preconditions of every action are satisfied at execution time, and which arrives at a satiffigi final state. To verify a plan, follow the actions in order and check that these two desiderata are achieved. In this work, we convert natural language responses into PDDL [46] and evaluate them with VAL 21
4 Chain of Thought Setups for Planning
We examine the influence of prompt selection on LLM performance within subsets of the Blocksworld domain. formally specified problem instance can be translated into many possible prompts. The most basic of these is input/output (I/O) prompting: the problem is translated directly from PDDL into natural language and provided to the LLM 47). While this directly tests the LLM's ability to solve the problem, it is not always the most effective strategy for maximizing performance.
All POOLEstttemm
All Blocksward Problems
Drawing metaphors of human learning, recent liter- ature has claimed that LLMs are capable of in-context learning. The basic idea is that-by first presenting the model with examples of similar problems-it is possible cause LLM to acquire relevant new skills within the current context window. n--hot prompts operational- ize this by prepending number of relevant examples. Chain of thought [50] approaches take this further, pre- senting human- cuman-crafted "thoughts" which the LLM is intended to imitate in its response. Practitioners argue that, intuitively, these augmented examples teach the LLM how to solve problems in the given set.
Toble to Stock Problermm
Tobdee to indoogrsptie Shook mobiems
Figure 1: Target Distributions of Problems.
This figure shows the levels of expected generality for each prompt.
However, this method relies human labor [53] to provide task-specific knowledge and an (at least rough) algorithmic or procedural approach the problem. The more general the provided knowledge is, the more problems it can be applied to, and the less human prompt-crafting it requires. On the other hand, the more granular and specific is, the more performance can be expected.
In experiments, we consider subsets of Blocksworld problems. We follow a prompt structure similar to that described [47] [ but include "thoughts" our n-shot prompts. These thoughts are written follow algorithmic procedure for solving the example problem.
Not every procedure is applicable to every problem. From the point of view of a human hand- crafting chain of thought prompt, there is intuitively an expected target distribution on which the
Prompp and response examples for each approach can be found in the Appendix.

---

demonstrated algorithm generally works. For instance, prompt designer detailing how to stack C on top of B on top of A will expect that a model that learns this procedure will also be capable of stacking B on top of A on top of C, but may not expect it to know how to first properly dismantle an existing tower of blocks to access a necessary block. However, this distribution often differs from the effective target distribution- that is, the actual set of problems on which the prompt gives robust improvements in performance. We explicitly describe the gap between these two distributions.
Zero-Shot Chain of Thought (Universal): This is the most general approach, and involves merely appending "let's think step by step" to the end of the prompt
Progression Proof (Specific to PDDL): Versions of this CoT could, in principle, be prepended to any PDDL problem prompt. as the generation of annotated examples is easy to automate without knowledge of the specific PDDL domain. [47] This prompt includes (1) a meta-prompt explaining plan correctness and (2) example where each action is annotated with the state prior the action, the reason why the action is applicable in that state, and the resulting state after the action is applied. Examples start from an arbitrary block configuration and construct single stack of blocks from it.
Blocksworld Universal Algorithm (Specific to the Domain): Blocksworld, it possible to reach any goal state from any initial state by simply unstacking all the blocks, placing them on the table, and then reassembling them into the required stacks. Resulting plans are not only executable and goal-reaching, but will never exceed twice the length of the optimal plan for any given instance [40]. This prompt demonstrates an annotated version of this approach, explaining and performing both the deconstruction and reconstruction steps of the algorithm. The same examples are used as in the previous prompt. The expected target distribution encompasses all Blocksworld problems.
Every example is mablett-sstacl problem: every block starts on the table, and the goal to create a single specific stack of blocks. This specificity simplifies the problem greatly, and allows near-direct pattern matching between the examples and the I.LM's output; however, is infeasible to specify prompts with this level of detail for every problem class. The expected target distribution is table-to-stack Blocksworld problem, as they are the only problems that can be solved by the described algorithm.
Stacking Prompt (Specific to Narrow Problem Class):
Lexicographic Stacking (Specific to Particular Syntactic Sequences): We simplify the problem further by focusing a particular syntactic form of the goal. This prompt is very similar to the stacking prompt. but is specific to a subset of the target distribution: the goal state is always a lexicographic prefix (e.g.. A, AB, ABC, etc.).
Blocksworld Results
We perform two parallel studies. The first tests each chain of thought prompt on its intended problem distribution, as explained in the previous section. Then, we focus on a specific subclass of Blocksworld problems and test every prompt on just that subclass. Together, we expect these two studies to give us a good picture of how effective LLMs are in applying advice beyond the specific instances.
5.1 Testing on Intended Problem Distributions
We evaluate the performance of GPT-4 and Claude-3-Opus on Blocksworld problems with both standard 2-shot prompts and chain of thought prompts of varying granularity. Each prompt is tested on its intended problem class, as discussed in the previous section.
Chain of thought does not meaningfully enhance performance except on the narrowest problem distributions. While providing this chain of thought advice becomes significantly harder as the level of specificity increases, is necessary, as the LLM succeeds only when the problem is reduced to level where basic pattern matching suffices: at each stage, stack the next letter on top; if that letter does not exist on the table, then stop.
key advantage of planning domains is that they provide the ability to easily and systematically gen- erate larger test sets, including arbitrarily more challenging instances. The difficulty of Blocksworld instance scales with the number of blocks involved, allowing us to clearly assess the out-of-domain generalization achievable with and without chain of thought. As shown in Figure 2 chain of thought
5

---

% of correct instances by GPT-4-Turbe
correct
100
Fast Downward Planner
Zero Shot
80
Progression
Specific n-ehot
40
Blocksworld Universal Algorithm
Specitic n-shot
20
Stocking Prompt
3
4
5
6
7
10
13
14
15
16
17
18
19
20
#
Blocks
% of correct instances by Claude-3-
Faet Downward Planner
80
Shot CoT
% Instonces
80
Progression Proof
Damain Specific n-shot
40
Blocksworld Universal Algorithm
Problem Class Specific n-shot
20
Stocking Prompt
3
5
19
20
# of Blocks
% of correct instances by GPT-4
Shot
80
Zero Shot CoT
instances
60
Progression Proof CoT
Domain Specific n-shot
Blocksworld Universal Algorithm
Problem Class Specific n-shot
20
Stocking Prompt
# of Blocks
Figure 2: Accuracy of GPT-4-Turbo, Claude 3-Opus and GPT-4 across chain of thought prompting methods in their intended problem distributions with increasing number of blocks.
does generalize beyond handful of blocks. Note that sound planning systems (such Fast. Downward) have a 100% accuracy on all problems tested.
5.2 Testing only on Table-to-Stack
As mentioned before, a table-to-stack problem is any problem in the intended target distribution of the stacking prompt. The initial state has every block the table, with goal of arranging all the blocks into single, pre-specified stack. While simple problem, GPT-4's zero-shot performance over 261 instances is 3.8%. With the stacking CoT prompt, performance improves to 59.3%. Is this a result of the model learning in-context how to reason correctly over this type of problem? so, might expect it to perform the same when presented with more general CoT prompt that demonstrates the same procedure, but is applicable to a greater variety of problems.
To check this, we evaluate performance of our prompts on table-to-stack problems with prompts of varying granularity: standard I/O prompting, general n-shot (drawn from arbitrary Blocksworld problems), goal-specific n--hot (drawn from table-to-stack problems), and three levels of CoT specificity. Table I shows the results: only the most specific and least applicable prompt retains anywhere near this performance improvement. Figure A.I.I in the appendix further illustrates that none of the prompts provide robust black-hiight generalizability. We also tested self-consistency on these prompts, but found that performance dropped. Details can be found in Appendix A.2

---

Prompt
GPT-4-Turbo
Claude-3-Opus
GPT-4
zero-shot
19.1%
9,96% 10.34%
3.83%
zero-shot CoT
21%
4.98%
Domain-Specific n-shot
13.7%
16.4%
6.13%
Progression Proof CoT
15.3%
4.59%
6.89%
Domain-Specinn n-shot
13.7%
16.4%
6.13%
Blocksworld Universal Algorithm
37.1%
37.1% %
51.3%
Problem Class Specific n-shot Stacking Prompt
18%
15.7% 24.5%
8.81%
40.6%
59.3%
Table 1: Accuracy across CoT and example granularities over 261 instances in table-to-stack Blocksworld.
If chain of thought is meant to replicate human thinking learning, it should generalize beyond the most direct pattern matches and allow for more robust reasoning across similar problems. However, results only show modest improvement in performance some domains, with specific enough prompting strategies, which quickly deteriorates when the problems shown become slightly larger.
6 Extension to Scalable Synthetic Benchmarks
Previous work on CoT mainly constrained evaluations to static test sets ranging from commonsense
domains (Sports Understanding [41, Strategy [88] CommonSense@A [44)) few-hop math word
problems (AsDiv (31), GSM8k [TO] MAWPS (27)) to number of basic "symbolic reasoning"
6] Many of these benchmarks are difficult to scale, but a number of them can be modified to allow for the generation of arbitrary new instances which nevertheless have clear ground truths. We examine CoinFlip, LastLetterConncaeeatioo and a synthetic proxy for multi-step arithmetical reasoning. Exact prompt details can be found in the appendices/A.7]/.a
tasks (CoinFlip 126]. CastLetterConcateeatiio (26.. Shuffledd Objects [41).. 26 50 55
AA.9 When possible we used the manual CoT prompts found [50] and the zero-shot Col prompt described in [26]. Number of examples ranges from to 3 for both CoT and direct prompts. Results for all three domains in Table 2 and Figure 3
CoinFlip is natural language version of this task introduced in (50) to showease performance of CoT, though that paper only studies up to four flip problems. An example prompt is "A coin is heads up. Craig flips the coin. Alice does not flip the coin. the coin still heads up?". The correct answer is "no". Note that chance performance this domain is 50%, there are only two possible answers. Our extension to the domain is detailed in
CoinFlip: Parity tests have long history in machine learning 32
LastLetterConcateee Concatenation: Also introduced in (50), the text processing task that asks for the concatenation the last letters of series of words. An example prompt is "Take the last letters of each word in 'Craig Alice" and concatenate them." for which the correct answer is "ge". The set of possible answers this task much larger than in CoinFlip, but previous work has claimed significant performance increases on this kind of task with COT. Modeling something similar to our Blocksworld granularity experiments, we create two other test sets, using the same underlying words in the same distribution, but which differ in what they ask the model to do. Last VowelConcateention requires using only the last vowels of words. requires using the first letter of the first word, the second letter of the second word, and so forth. If the rith word docs not have an nth letter, the problem specifies that a should concatenated to the string instead.
task is a simple
Multi-step Arithmetic on Single-Digit Numbers: CoT is often tested on math word problems. However, many of these test sets only include problems which require very small numbers of reasoning steps. GSM8k was designed partly so that its problems would "require more steps to solve". but its problems only range 2 to 8 steps and, in fact, previous analyses have found that only 10% of those problems require more than five steps- the majority is 2, 3, or 4. [16]
7

---

CoinFlip
LostLeteeConnattentiion
AE One- One-Digit Arithmetic
100
Vorkant
100
Explanctiion
% of Instances correct
Arbitrory
80
neom_dleaw
80
sing ee-iigt
60
Direct
40
40
40
20
20
20
Col
Dfrect
2
17 # of People
23
26
10
14
16
20
8
14
20 23
29
# of Words
# of Operations
Figure 3: Accuracy of GPT-4-Turbo with chain of thought prompting across variations of our synthetic datasets. "Direct" means direct prompting without any COT.
Prompt
CF
LLC
LVC
FLC
Arithmetic
Zero-Shot
56.38%
10.00%
5.75% N/A
1.81 %
24.13% 56.
45.60% 42.76%
Zero-Shot CoT
95.71%
52.54%
N/A
Manual CoT
98.89%
51.06%
27.00%
26.00%
50.43%
69.31%
Incorrect Cot
96.76%
48.15%
N/A
N/A
N/A
N/A
Table 2: Accuracy across CoT types and problem variations over all instances in our synthetic datasets. CF is CoinFlip, LLC is EastLetterConnatenatioo LVC is LastVowelConcatenan FoomaetterComcaltnattio,, Arithmetic is baseline single-digit Arithmetic, AE is the same problems but with the explanation provided that all intermediate answers are single digit.
FLC is
To sidestep this issue, we construct a synthetic dataset that involves linearly simplifying parenthesized expressions that consist of repeated applications of the four basic arithmetical operations on one digit numbers. An example prompt is "Simplify the following expression into single number: / (9 (5 + (1)))., where the correct answer is We filter our problems so that no operation ever results in a number that isn't in the range 9 This can be seen as a deeply simplified variant of the arithmetical expression simplification dataset presented in (3 where no modular arithmetic, negative numbers, or non-linear nesting is required. However, we extend our maximum number of required reasoning steps much further and we construct prompts which are more specific and spell out every single step explicitly. More details on the dataset can be found A.5
6.1 Results
Length Generalization The only synthetic domain that shows any hints of generalization is CoinFlip. Using (50)'s prompt, performance is perfect for through 4 step problems, starts to show the occasional mistake after, and only dips below 90% 3 3-step problems (as shown in Figure 3 However, the problems in this domain are very simple. Parallel to the lexicographic stacking case of Blocksworld, it does not require much reasoning beyond counting up average of half given problem's step count.
LastLectterConcattnatiio and multi-step arithmetic show behavior almost identical to main experiments. While sufficiently specific CoT prompts do increase performance on small instances, this performance increase quickly degrades as the number of steps necessary increases. Notably, the brring-based nature of the improvement CoT is inducing. We examine the data with different metrics and find that the only properties that do generalize with CoT are syntactic. In particular, while overall accuracy plummets back that of direct prompting, CoT consistently improves the Levenshtein distance the correct answer and ensures that the final response string contains exactly the right letters, just not in the
problem does allow us to examine what exact
We exclude O, since any number multiplied by zero is zero, and this would quickly lead to zero representing around 50% of correct answers for larger numbers of reasoning steps.
8

---

right order or number. We take this as further evidence that CoT, rather than teaching algorithms or procedures, modifies the syntactic style of the LLM's output, and that this pattern matching is what leads to observed increases in performance on smaller instances.
Prompt Granularity and Problem Variation Because of the simplicity of these problems, prompt granularity is much harder to examine than in Blocksworld. There isn't t enough variation in possible problems. However, across the three types of letter concatenation and two types of arithmetic expression simplification that we test, we see very similar patterns as before: CoT's performance improvements are maintained much longer in easier cases, and take longer to collapse back to direct performance. There still seems to be a "sweet spot" where the problem is just barely hard enough that CoT makes a difference, but not so hard that this difference doesn't 't matter.
Examining Intermediate Reasoning The construction of our synthetic arithmetic task gives some hints as to what part of CoT may be failing. [4] argues that compositional reasoning fails because LLMs perform linearized subgraph matching and act noisy estimators of intermediate functions (see e.g. proposition 4.2 in [14]) and that performance collapses follow from the fact that repeated application of any error- poor-prone function estimator leads to exponentially accumulating error.
In our problem, it possible to exhaustively check whether this is the case. There are exactly 118 possible 1-digit binary arithmetic problems which result in a 1-digit number. We tested GPT-4-Turbo, GPT-4, GPT-3355Turbb Llama3-70b, and Llama3-8b on this dataset at various temperatures and every single model scored 100%. However, despite perfect performance intermediate function, CoT still does not lead to robust generalization to arbitrary length problems. Therefore, at least on this problem set, the issue isn't due to accumulating error. The problem must be with the LLM's inability to learn the correct algorithm from contextual demonstrations, rather than with its inability to execute that algorithm.
application of the required
Overall, we see that our results on planning are not a fluke. These three synthetic domains showcase similar generalization failures, but these failures only become clear when the problems tested on require sufficiently many reasoning steps or when the minor modifications of the domain are studied. This illustrates the need for testing on benchmarks which can generate arbitrary new instances of increasing difficulty. Without such testing, conclusions drawn from static test sets of limited size are unlikely to be robust. We implore the community at large to adopt more rigorous evaluation mechanisms, especially when making claims about the poorly-understood yet much-hyped algorithmic reasoning abilities of black box models.
7 Conclusion
In this paper, we conducted systematic evaluation of the effectiveness of chain of thought in large language models specific elassical planning problem. Our case study indicates that, contrary to previous claims in the literature, providing examples of procedural reasoning does not induce the general ability apply that procedure novel instances in current state-of-the-art large language models. In fact, the performance improvements seen when prompting LLMs in this manner quickly vanish when queries differ in generality from the examples, despite the fact that the same algorithmic procedure applies to the larger or more general instance. Very specific prompts are more likely to work, but they can require significantly more human labor to craft. Our results indicate that chain of thought prompts may only work consistently within problem class if the problem class is narrow enough and the examples given are specific to that class. Both of these facts show that chain of thought approaches provide less generalization than previous claims seem to indicate, and hint that basic pattern matching rather than context learning of general algorithmic procedures may better explain the improvements seen from chain of thought.
8 Acknowledgements
This research is supported in part by ONR grant NO00 -0001423-1-2409, and gifts from Qualcomm, J.P. Morgan and Amazon.
References
[1] The United States Social Security Administration SSA ssa.gov. ssa.gov

---

[2] Marah Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Ra- hee Ghosh Peshawaria, Ranjita Naik, and Besmira Nushi. Kitab: Evaluating llms on constraint satisfaction for information retrieval. arXiv preprint arXiv:2310. 255. /55/1, 2023.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt. Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint DrXii:23303.88722 2023.
[4] Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length gen- eralization in large language models. Advances in Neural Information Processing Systems, 35:38546-38556, 2022.
[5] Anthropic. Introducing the next generation of claude, Mar 2024.
[6] Guangsheng Bao, Hongbo Zhang, Linyi Yang, Cunxiang Wang, and Yue Zhang. Llms with thain-of-thought are non-causal reasoners. arXiv preprint arXiv:2402. 16048, 2024.
[7] Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwašniewski, Jürgen Müller, Lukas Gianinazzi, et al. Topologies of reasoning: Demystifying chains, trees, and graphs of thoughts. arXiv preprint ar2iv:2401.14295, 2024.
[8] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprini arXiv:2303.
2023.
[9] Xiaoxue Cheng, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Chainlm: Empowering large language models with improved chain-of siiin-of-thoon ofothhught prompting. arXiv preprint wxXv:2403.14331. 2024.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1r 2021.
[11] Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. arXiv
preprint arXiv:2208. 14271, 2022.
[12] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint 3xXiv:3301.00233 2022.
[13] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. /4325, 2023.
[14] Nouha Dziri, Ximing Lu, Melanic Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers compositionality. Advances in Neural Information Processing Systems, 36, 2024.
[15] Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang- Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
[16) Silin Gao, Jane Dwivedi-Ys Yu, Ping Yu, Xiaoqing Ellen Tan, Ramakanth Pasunuru, Olga Golovneva, Koustuv Sinha, Asli Celikyilmaz, Antoine Bosselut, and Tianlu Wang. Efficient tool use with Mflain-of-abstractin reasoning. arXiv preprint 1oXiv:2401.1444 /7464, 2024.
[17] Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbic. Large language models are 2023.
not abstract reasoners. arXiv preprint
[18) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
[19] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint 2rXiv:2305.1173: 2023.
[20] Ruixin Hong. Hongming Zhang, Xinyu Pang, Dong Yu, and Changshui Zhang. A closer look the self-verrfication abilities of large language models in logical reasoning. arXiv preprint arXiv:2311.0795. 2023.
10

---

[21] Richard Howey, Derek Long, and Maria Fox. VAL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL. In 16th IEEE International Conference on Tools with Artificial Intelligence, pages 294-301. IEEE, 2004.
[22] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song. and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2023..
[23] IPC. International planning competition, 1998.
[24] Dongwei Jiang, Jingyu Zhang. Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi. Self-[in] correct: Lims struggle with refining self-generated responses. arXiv preprint arXiv:2404.04298 2024.
[25] Yeo Wei Jie, Ranjan Satapathy, Goh Siow Mong, Erik Cambria, et al. How interpretable are rea- soning explanations from prompting large language models? arXiv preprint arXiv:2402./1863, 2024.
[26] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213,, 2022.
[27] Rik foncel-Kedziorski Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152-1157, 2016.
[28] Zhan Ling, Yunhao Fang, Xuanlin Li, Zhiao Huang, Mingu Lee, Roland Memisevic, and Hao Su. Deductive verification of chain-of oftthhughttthtttth reasoning. Advances Neural Information Processing Systems, 36, 2024.
[29] Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apid- ianaki, and Chris Callison-Burch. Faithful chain-of-thought reasoning. arXiv preprint arXiv:2301.13377 2023.
[30] Drew McDermott, Malik Ghallab, Adele E. Howe, Craig A. Knoblock, Ashwin Ram, Manuela M. Veloso, Daniel S. Weld, and David E. Wilkins. Pddl-the planning domain definition language. 1998.
[31] Then-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. arXiv preprins arXiv:2 10.::2206..7772 15772, 2021.
[32] Marvin Minsky and Seymour Papert. An introduction to computational geometry. Cambridge tiass., HIT, :794 1969.
[33] Maxwell Nye. Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2/ 2021.
[34] Flavio Petruzzellis, Alberto Testolin, and Alessandro Sperduti. Benchmarking gpt-4 on algorithmic problems: A systematic evaluation of prompting strategies. arXiv preprint arXiv:2402.17396 2024.
[35] Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction. arXiv preprint arXiv:2208. (000:22000.00511,, 2022.
[36] Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang. and Huajun Chen. Reasoning with language model prompting: A survey. arXiv 2022.
preprint arXiv:2212.09553
[37] Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint 00.0:22200.01200, 2022.
[38] Rylan Schaeffer, Kateryna Pistunova, Samar Khanna, Sarthak Consul, and Sanmi Koyejo. Invalid logic, equivalent gains: The bizarreness of reasoning in language model prompting. arXiv preprint arXiv:2307.1057. 2023.
[39] Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual thain-of-though reasoners. In The Fleventh International Conference on Learning Representations, 2022.
11

---

[40] John Slaney and Sylvie Thiébaux. Linear time near-optimal planning in the blocks world. In Proceedings of the National Conference on Artificial Intelligence, pages 1208-1214, 1996.
[41] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. 2022.
arXiv preprint arXiv:2206.0460
[42] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the selfiverification limitations of large language models on reasoning and planning tasks, arXiv preprint arXiv:2402.08/ 15, 2024.
[43] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint 1rX1v,2210.09261, 2022.
[44] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: A question answering challenge targeting commonsense knowledge. Proceedings of NAACL- HLT, pages 4149-45 1159,118, 2019.
[45] Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36, 2024.
[46] Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36, 2024.
[47] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities large language models-a critical investigation. Advances in Neural Information Processing Systems, 36, 2024.
[48] Jianing Wang, Qiushi Sun, Nuo Chen. Xiang Li, and Ming Gao. Boosting language models reasoning with Staii-of--kooleedee prompting. arXiv preprint arXiv:2306.0647 06427. 2023.
[49] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Kleventh International Conference on Learning Representations, 2022.
[50] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of- thouhgt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-244 244377, 2022.
[51] Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang. An empirical study on challenging math problem solving with gpt-4. arXiv preprint vrXiv:2306.01337 2023.
[52] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
[53] JD Zamfirescu-Percin Richmond Wong, Bjoemn Hartmann, and Qian Yang. Why johnny can't prompt: how non-ai experts try (and fail) to design Ilm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-21, 2023.
[54] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representa- tions, 2022.
[55] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
[56] Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprins urXiv:2211.09066 2022.
12

---

A
Appendix
Contents
A
Appendix
13
A.1
Broader Impacts
13
A.2
Self Consistency on Table Stack problems
13
A.3
Further details on modifications to the CoinFlip domain
13
A.4
4
Further details on modification to the Lastl etterC Concalenation domain
14
A.5
Further details on the multi multi-step Arithmetic dataset
14
A.6
Planning Prompts and Responses by GPT-4
15
A.6.1
Domain Information
15
A.6.2
Progression Proof Prompt
16
A.6.3
Blocksworld Universal Algorithm Prompt
18
A.6.4
Stacking Prompt
21
A.6.5
Lexicographic Stacking Prompt
23
A.7
Coinilip Prompts
24
A.8
LasilletterCConaeeenaioo Prompts
25
A.9
Single Digit Arithmetic Prompts
28
A.1
Broader Impacts
Chain of Thought (CoT) has become one of the most widely adopted ideas for improving planning and reasoning abilities of LLMs. Almost every system routinely, and uncritically, uses some prompting strategy attributed to COT. On the flip side, whenever LLMs are shown have limitations in any sphere, practitioners tend to question those studies by attributing it to unskilled use of CoT methodology. Our study, based on both in planning and other more standard tasks, calls into question the prevalent belief that LLMs are capable of operationalizing and generalizing the CoT advice effectively. instead suggests that CoT is effective only when the LLM can straightforward pattern matching between the example and the problem. We believe that the lessons of this study will be helpful in mitigating the applications of LLMs to tasks requring planning and reasoning with false confidence.
A.2 Self Consistency on Table to Stack problems
We evaluated self consistency 49 a state-of-thhean extension of CoT, on table stack problems. We sampled 5 different reasoming paths (with temperature 0.7) and chose the most frequent plan breaking ties randomly. As our results show (in Table FigureA..2.0 self-consistency does not lead to generalization breakthrough, and in fact is generally worse than the original results in Table 1. This likely because the solution space for planning problems much larger than that studied in previous (often multiple choice) benchmarks. In fact, most queries led to five unique responses, forcing us to choose the final answer from them at random.
A.3 Further details on modifications the CoinFlip domain
Given list of names, generating new instances is just a matter of filling in a template. We source our list of names from the U.S. Social Security Administration [ and only keep names with least 50 occurrences. We scale our problems with the number of names (potentially repeated) mentioned in the prompt. The main test set consists of 120 instances, with 40 instances per number of names and 28 different numbers of names, ranging from to 28, We also tested [50]'s prompt on an extended
13

---

% correct instances by GPT-4-Turbo
% of Instances correct
100
Fost Dowmward Planner
80
Zero Shot CoT
80
Problem Class Specific n-shot
Stacking Prompt
Blocksworld Universal Algorithm
Progression Proof CoT
correct instances by Claude-3-on 3-Opus
% instances correct
100
Fast Downward Planner
80
Zero Shot CoT
60
Problem Closs Specilic n-shot
Stacking Prompt
Blocksworld Universal Algorithm
Progression Proof CoT
20
16
Blocks
of correct instances by GPT-4
x ofstoncee correct
100
Fast Dowmward Planner
80
Zero Shot GoT
Problem Closs Specilic n-shot
Stacking
40
Blocksworld Universal Algorithm
Progression Proof
20
10
13
14
16
19
Blocks
Figure A.1.1: (Table-to-stack) Accuracy of GPT-4-Turbo, Claude-3-Opus and GPT-4 across chain of thought prompting methods with increasing number of blocks.
Prompt
GPT-4-Turbe
Claude-3-Opus
zero-shot
18.3%
9.19%
zero-shot CoT
18.3%
11.8%
Problem Class Specific n-shot
15.7% 24.5%
14.9%
Stacking Prompt
26.4%
Table 3: Accuracy of Self-consistency over 261 instances in table-to-stack Blocksworld.
set of 2960 instances, with 40 instances per number of names, but only stopping at 75 names, finding that performance did begin to decrease more significantly past 30 names.
A.4 Further details on modifications the Cest..ctterConcattnattio domain
We use the same database as in CoinFlip to generate words. | To scale instances, we simply increase number of words whose last letters must be concatenated. Our problems range from to 20 words, with 40 instances per word, giving total of 800 problems.
A.5 Further details on the multi-step Arithmetic dataset
The number of reasoning steps in this domain corresponds directly the number of operations that need to be performed to simplify given expression. Our test set consists of 1160 total problems, spread to 29 operations, with 40 instances per number operations. Again mirroring
14

---

% of correct instances by GPT-4-Turbo
100
% of correct
Fast Downward Pianner
Zero Shot
80
CoT
Specific
Stocking Prompt
40
3
8
9
10
12
13
14
15
16
18
19
20
% of correct instances by Claude-3-O0l
100
Fast Downward Pianner
Instances correct
Zero Shot
80
Zero CoT
Problem Specific n-shot
60
Stacking Prompt
40
3
8
9
10
13
14
15
16
18
19
20
Figure A.2.1: (Table-to-stack) Accuracy of GPT-4-Turbo and Claude-3-Opus across chain of thought prompting methods with self consistency.
our Blocksworld test cases, we experiment with two variants: prompting as if these were any expression simplification problems, and prompting with the explicit additional information that every intermediate step must be a single digit number.
A.6
Planning Prompts and Responses by GPT-4
A.6.1
Domain Information
Domain Information
pleying
arrange
actious can do
Pick block
Unstack block from on top of another block
Put dovn block Stack block on top of another block
I have the folloving restrictions my actions: only pick up unstack block I can only pick
unstack block if hand is ampty. I can only pick up block if the block on the table and the block clear. block is clear the block
other blocks top of it and if the block is not picked up. I can only unstack a black from on top of another block if the block an unstacking vas really on top of other black.
I can only unstack block from ea top of another block if the black I an unstacking is clear.
holding pick up unstack block, holding.
down block that I can only stack block on top of another block if an holding the block being stacked.
I can only
I only stack block top of another block if the block onto which stacking the block clear.
ny hand dace you stack block on of second block, the second block is no longer clear.
ampty.
15

---

A.6.2 Progression Proof Prompt
Progression Proof Chain of Thought Prompting and GPT-4 Response
The plan correctness is defined in terus of states resulting from executing the actious in the preconditions
plan.
executable state resulting from the action execution consista of everything in the previous state with the addition and deletion add and delete effecte the action. Plan correctneas is defined followe: if the firet action in the plan is applicable in the initial state, 1.e., ite proconditions are all present there; and the second action is applicable in the state this
state
state.
resulting from applying the first
continues until the state resulting from the application of the last action in the last but one atate givea state shere the goals satisfied.
initial
Here 1s an example problem:
[STATEMENT]
initial conditions is eupty, Block B is on top of Black D, Block A is on the table, Block D 1s on the table and is to have that Block D is top of Block
that,
Black
the hand
plan follows:
[PLAX]
1. Current State: Block & is clear, Block B is clear, Block C is clear, the hand is eapty, Block B on top Black D, Black on the table, Block on the table and Block 1a
table
Action: unstack Block B from en top of Block
because precanditions; Block B - in clear, the hand is empty and Black B is on top of Block D, are satistied in the currant
Reason:
The abova action applicable in the
Resulting State: Block A is clear, Block C is clear, Block D is clear, the hand is currently holding Block E, Block A is the table, Block C is the table and Block on the table
Block clear, Black is clear, hand currantly holding Block B, Black is on the table, Block C ls on the table and Block D is on the table Action: put
Current State: Block
Reason: The above action is applicable in the current state because its preconditions; the hand is currently holding Block B, are satisfied in the current state. Resulting State: Block is clear, Block clear, Block is clear, Block is clear, the hand is empty, Block is an the table, Block is on the table, Black is on the table and Black
table
clear, Block clear, empty, Block A is on the table, Block B is en the table, Block C is the table and Block
clear,
clear,
Action: pick up Block D
Reason: The above action is applicable in the current state because its precanditions; Block D table, satisfied in
clear, hand empty and Block
- state.
Resulting
clear, Block holding Black D, Block A is on the table, Black B is the table and Black is on the table
clear,
is currently
Current State: Block A is clear, Black is clear, Black C is clear, the hand in currently the table Block
table, Action: stack Block on top of Block B
holding D,
because preconditions; + is clear and the hand is currently holding Block D, are satistied in the current state. Reaulting State: Block is clear, Block C is clear, Block la clear, the hand is enpty, Block the table, Block
Reason: above
applicable
top Block B, Black the table
the table and Block C
Final State: Block & is clear, Block is clear, Block D is clear, the hand is empty, Block D is top
table, Block
table
The goal conditions are satiafied the final state. Hence, the above plan valid.
[PLAX END]
another example:
[STATEMENT]]
As Initial conditiona have that, Black B 1s clear, Block B is of Block C, Block top of Block A, the hand is eupty, Block A is on the table, and Block is on the table. Block ia top Black Block top Block c.
goal
16

---

on plan la as follows:
[PLAN]
Current State: the Block is clear, the Block D is clear, the hand is anpty, the Block B is on Block ia the tabla and the
top Block the Block
top the Black
Block D is on the table
Action: unetack
top Reason: The above action is applicable in the current state because its preconditions; the Block is clear, the band is eupty and the Block is on top of the Block c, are satiefied in the current state.
Resulting State: the Block C in clear, the Block is clear, the hand is currently holding table and the Block
Black
top
Block
ie on the table
Current State: the Block ie clear, the Block D is clear, the hand is currently holding Block B, the Block is on top of the Block 4, the Block A is on the table and the Block D is on the table
Action:
put down the Block B
action applicable is currently holding Block B, are satiefied in the current state. clear, Block clear, of the Block A, the Block A is on the table, the Block B ia - the table and the Block D is on the table
Reason:
because preconditions;
-
Resulting
the empty, the Block G is
clear,
+
3. Current State: the Block is clear, the Block C is clear, the Block D ia clear, the hand ia table,
the Block the table and the Block D is on the table
enpty,
top
Action:
top The above action is applicable in the current state because its preconditions; the Block C is clear, the hand is ampty and the Block is on top of the Block A, are satisfied state.
unstack
Reason:
in the
Resulting State: the Block A is clear, the Block B is clear, the Block is clear, the hand is table,
the Block Black D is on the table
currently holding
Current State: the Block
clear, the Block B is clear, the Block 1a clear, the hand 1s currently holding Block c, the Black A is on the table, the Black B is on the table and the Block D is the table
Action: stack the Block on top of the Block B
Reason:
preconditione; Black B is clear and the hand is currently holding Black C, are satisfied in the current
action applicable
state
state.
Resulting State: the Block A is clear, the Block is clear, the Block clear, the hand is empty, the Block C in on top of the Block B, the Block A is on the table, the Block B in on the
is the table
clear, empty, the Block C is on top of the Black B, the Black A is on the tabla, the Black is on the table and the Block on table
State:
clear,
Action: pick up the Block
Reason: The above action is applicable in the current state because its precanditional the satiefled
Block
eapty clear,
current state.
Resulting
currently holding Black D, the Block C in on top of the Black B, the Block A is on the table and the Block B 1s on the table
clear,
Current State: the Block in clear, the Block C is clear, the hand is currently holding Block B, Block
the The above action is applicable in the current state because its preconditions: the Block C is clear and the hand is currently holding Block D, are satisfied in the current
on top
table
Action:
top
Reason:
state.
Resulting State: the Block A is clear, the Block is clear, the hand is empty, the Black is Block
on top
Block and the Block B is on the table
top
on table
Final State: the Block
clear, the Block D is clear, the hand is empty, the Block C la top the Block B, the Block D is on top of the Block C, the Block on the table and the Block B is the table
The goal conditions are satinfied in the final state. Hance, the above plan in valid.
[PLAN ZKD]
Here is the problem to solve:
[STATEMENT]
As initial conditions have that: Block B is clear, Block C is clear, the hand empty, Block B top Black A, Block the table and Black C in the table.
17

---

goal la for the following to be true: Block 1a on top of Block and Block C is of Block
My plan is as follows:
GPT-4 Reeponse
[PLAX]
Current State: Block ie clear, Block clear, the hand ie eapty, Block ie on top Block Block A the table and Block is the table
Action: unstack Block B from an top of Block
Reason:
because precanditiona, ie clear, the hand is enpty and Block B is on top of Block 4, are estisfied in the current
action applicable in
Resulting State: Block is clear, Block C is clear, the hand currently holding Block Block is on the table and Block is on the table
2. Current State: Block A is clear, Black is clear, the hand is currently holding Block B, Block table
Action: put down Block B
because preconditions; + is currently holding Block 8, are satinfied in the current state.
Reason: above action applicable
Resulting State: Block A is clear, Block is clear, Block is clear, the hand is empty, Block table, Block
the table and Block
the table
clear, is on the table, Block B is en the table and Block C is on the table
clear,
eapty,
pick up
Reason: The above action is applicable in the current state because its precanditions; Block A is clear, the hand is empty and Block A is on the table, are satisfied in the current state.
Resulting State: Block B is clear, Block C is clear, the hand is currently holding Block À, Block
is clear, B is the table and Block is on the table Action: stack Black on top of Black B Reason:
clear,
currently holding A,
action is applicable in + ie clear and the hand is currently holding Block A, are satisfied in the current state. clear,
because preconditions; Block B
Resulting
eapty, Black B, Block is on the table and Block C is the table
is top
Current State: Block A 1s clear, Block is clear, the band is enpty, Block Block B, Block B in on the table and Block in on the table
Action: pick up Block
Reason:
The above action is applicable in the current state because its preconditione. Block C enpty and Block
is clear,
table, are satisfied in the
state.
Reaulting State: Block clear, the hand is currently holding Block c, Block on top Block and Block B 1s the table
clear, Block B and Block B is on the table
curreatly holding
is top
Action:
top Reason: The above action is applicable in the current state because its precanditions: Block A is clear and the hand currently holding Block c, are satisfied in the curreat state. top Block B, Block
Resulting State: Block is clear, the hand is empty, Block wa on top of Block and Block B is on the table
Final State: Block C is clear, the hand is ampty. Block A is top of Block B, Block C is on top is The goal conditions are satiafied in the final state. Hence, the above plan valid. [PLAN END]
VALID PLAN---
A.6.3 Blocksworld Universal Algorithm Prompt
Blocksworld Universal Algorithm Chain of Thought Prompting and GPT-4 Response
Here is an example problen:
[STATEMENT]
initial conditions have that, Block is clear, Block is clear, the hand eapty, Block C top Block A, Block the table, Block B is the table.
18

---

goal la to have that Block A 1a top of Block C and Block on Block
plan is follows:
[THOUGHTS]
1. I take apart all the tovers, and pat all the blocks en the table. check every
left there. Any of another block muat be put on the table, but can only unstack then if they clear, so will check that before trying to unetack any of then. If block neede to be moved but ien't later it
Any
already on
top
clear, I'll back
already
Block B is already on the table.
according to the initial condition, s0 I proceed. I unstack it. Then, I it dovn the table. This clears Block A.
top
only
All the blocks are noe clear and an the table.
2. I look at the geal and stack each tover need, starting from the botten. I check each of the tovers.
which then
can it Block needs to be on top of Block C, so it can't be the base of a toxer. Block needs be top of Block it can't the base of Block needs Block A on top of it. double check the goal: this is the only condition that must top any other
block the base of tover.
In sumnary, only Black can be used as base of tover.
build the
base. don't nead Block and it be, I know it is already placed correctly.
has Block
goee top of The block that goes top of Block à is Block B. I pick up Block B. I stack it top of Block A. needs
is
pick up
top
on top
toser complate.
There are no other blocks which can be used as base of a toser, and all the blocks nentioned in conatructing.
the goal
placed,
my plan required
[PLAX]
unstack Block C from top of Block pat doan Block C
pick up Block stack Block on top of Block C pick up Block
stack Block
of Block A
[PLAX END]
Here is anothar example:
[STATEMENT]
empty, on top D, Black is on of Block c, Block in on of Block and Block D ia on the table. My geal to have that Block C is top of Block D, Block A is top of Block C Block of Block
initial
that,
clear,
plan follows:
[THOUGHTS]
take all the tovers, and all the blocks the table.
I vill check every block. Any that are already on the table can be left there. Any that are on top another block must be put the table, but only unstack then if they clear, will check that before trying to unstack any of then. If block needs to be noved but isn't
clear, I'll come
only unatack statea that Block on top Block A. Therefore, can't unstack Block until move
top
is
condition
Block don't touch it yet, and look at Block C.
Block top of Block A. I only unstack block that clear. The initial condition states that Block is on top of Block c. Therafore, can't unstack Black until move yet,
Block
Block B is on top of Block c. I can only unstack block that is clear. The initial condition clear. unatack it. Then, put down on the table. Thia clears
atatea that Block
Block
Block is on top of Block A. I can only unstack a block that is clear. Block cleared previously. unstack Then, put it down the table. This clears Block
19

---

Block la on top of Black D. I can only unatack block that is clear. Block cleared previously. unstack it. Then, put it down on the table. This cleare Block
All the blocks are noe clear and an the table.
2. I look at the goal and stack each tover need, starting Iron the botton. check each of the which of then
tovere.
Block needs be on top of Block D, so it can't the base of a tower.
Block needs be top of Block Block needs to be on top of Block A, so it can't be the base of a taver.
it can't be the base of
top
goal:
only condition that
hold for block D. It does not need to be on top of any other block. Therefore, can use it
In sumnary, only Block can be used as the base of a toxer.
build the toser that has Block as base. don't nead to nove Black D, and leave it be, as already placed correctly.
The block that goes on top of Block D is Block C. I pick up Block C. stack it on top of Block D. pack up
goes top The block that goas top of Block A is Block B. I pick Block в. stack it top of Block A. No block needs to be on top of Block B. Therefore this tover is complete.
top Block
There are no other blocks which can be used as base of tover, and all the blocks nentioned in constructing,
the goal
placed,
Now, doxn my plan required [PLAX]
unstack Block from top Black put dovn Block B
top
put dovn Block C
Block dovn Block A
top
pick up Block c
stack Block
top Block
pick Block A
top Block
pick Block B
top [PLAN END]
Hare 1s the problam to solve:
[STATEMENT]
initial conditions
clear, clear,
eapty,
Ny geal on top of Black 4, Block is on the table and Black C is the table.
the following be trues Block on top Block and Block 1a top Block
plan follows:
Response
[THOUGHTS] take spart the tovers, and put all the blocks the table. I vill check every block. that already the table can be left there. Any that top another block must be put an the table, I can only unstack then they are clear, so If
trying clear, I'll come back to it later once it is.
any
needs noved
Block la on the table but is not clear.
Block is on top of Block A. I can only unstack a block that is clear. is clear according to I proceed. I unstack it. Then, I dovn the table. This
the initial condition, clears Block A.
already
table.
the
nou clear and
look at the geal and stack each tover need, starting from the botton. check each of the mentioned blocks order determine uhich of then the bases tovers.
top of Block B needs Block A on top of it. I double check the goal: this is the only condition that must hold for block It doea not need be top of any other block. Therefore, can use it the base of tover.
needs
of toxer.
Block neede to be on top of Block so it can't be the base of a tower.
20

---

In sumnary, only Block can be used as the base of toser.
I build the
base. don't 't need Block B, and Leave it be, I knnw it already placed correctly.
that has Block
The block that goas top of Block is Block pick up Block stack top Block The block that goea on top of Block A is Block C. I pick up Block c. stack it on top of Block A. block
top
complete.
There are no other blocks which can used ae base of tower, and all the blocke nentioned done constructing.
the have been placed,
ny plan required fornat:
[PLAX]
unstack Block from top of Block
put doun Block B
pick Block
stack Block an top of Block B
pick up Block
stack Block on top of Block A 2HD]
VALID PLAN
A.6.4 Stacking Prompt
Stacking Chain of Thought Prompting and GPT-4 Response
STATENENT]]
initial conditions have that, Block A is clear, Block is clear, Block is on the table, Block B is the table and the hand enpty. By goal is to have that Block B is on top of Block A.
My plan is as follows:
[THOUGHTS]
look the goal and stack each tover need, starting from the bottom. goal nentions blocks and chack each of the nentioned blacks order determine which of then are the bases of tovere.
Block A needs to have Block
top of it. I double chack the goal: this is the only condition that must hold for block &. It does not need be on top of any other block. Therefore, can it the base of tover.
Block needs to be on top of Block so it can't be the base of a toxer.
In sunnary, only Block can be used as a base of a tover.
base. don 't need to nove Block A, and I leave it be, I know it alreedy placed correctly. The block that
build the toser that has Block
top of Block A is Block B. I pick up Block stack it of Block This satisfies the part of the goal that says that Block is on top of Block A. block needs
top Black Tharefore
camplata.
tover,
the have been placed,
done conatructing.
Now, urite domn my plan the required format:
[PLAX]
pick Block B
top Block [PLAN END]
Hore another example:
[STATEMENT]
As initial conditions have that, Block A is clear, Block B is clear, Block C is clear, Block A on the table, Block on the table, Block on the table and the hand ia eapty. is to have that Block A is top of Block and Block top Block A.
follous:
[THOUGHTS]
1. look at the geal and stack each tover need, starting from the botton.
My geal only nentions blocks A, E, and C. an free to ignore all other blocks. I check each of the nentioned blocks order determine which of then the bases tovere.
21

---

Block B, and I leave it be, Block needs Block A on top it. double check the goal; this the only condition that must hold for block C. It does not need be top of any other block. Therefere, can it the base of taver.
Block neads Block
double check goal: thin ian't only condition that must hold for Block 4, because Black A needa to be on top of Block c. Therefore Block A can't tower.
top
Block needs be top of Block it can't be the base of tower.
cunnary, only Block
used the base of block goes top of
build
it be, top Block of the geal that says that Block is on top of Block C.
don't
C,
4 I know it is already placed correctly.
is
pick up
This satiefies the
The block that goes on top of Block A is Block B. I pick up Block B. stack it on top of Block A. This satisfies the
of the goal that says that Block
top of Block A.
No black needs to be on top of Black B. Tharefore this tover is complate.
There are no other blocks which can be used ae base of tower, and all the blocke nentioned in the goal
placed,
constructing.
2. Nov, urite down my plan in the required format:
[PLAX]
pick up Block
stack Block on top of Block c pick up
stack Block
of Block A
[PLAN END] Here the problam solve:
[STATEMENT]
As initial conditions have that: Black A is clear, Block B is clear, Block C is clear, the hand empty,
table, Block is la for the following be true: Block 1s on top of Block and Block C is of Block
Block on
My plan is as follows:
CPT-44 Response
[THOUGHTS]
look the goal and stack each tover need, starting from the botton. My goal only mantions blocks A, B, and c. an free to ignora all othar blacks. I chack each of the nentioned
order deternine which of then
only condition must hold for black B. It does not need to be top of any other block. Therefore, can use it the base tover.
needs
on top
goal:
Block needs Block C on
double check the goal: this isn't the only condition that mast hold for Block 4, bacause Black A needa to be on top of Block B. Therefore Block A can't
Block C needs to be on top of Block à, so it can't be the base of tower.
In sumnary, only Black can be used as the base of taver.
I build the
that has Block I know it already placed correctly.
base. don't 't need
top Block This satisfies the part of the goal that says that Block A is on top of Block B.
block goee top of
is
pick up
that goes top This satiariea the of the goal that that Block is on top of Block A.
pick up
top
block neede be on top of Block C. Therefore this tover is complete.
There are no other blocks which can be used as base of tover, and all the blocks nentioned in placed,
the goal
constructing.
Nou, down my plan requirad fornat:
[PLAX]
pick Block A
stack Block an top of Block B
pick up Block
stack Block on of Block A END]
VALID PLAN
22

---

A.6.5 Lexicographic Stacking Prompt
Lexicographic Stacking Chain of Thought Prompting and GPT-4 Response
Here is an example problem:
[STATEMEKT]
initial conditions have that, Block is clear, Block is clear, Block ia on the table, Block B is the table and the hand is expty. My goal is to have that Block A is on top of Block
My plan is as follows:
[THOUGHTS]
look at the geal and stack the tover need, starting Irom the bottom. Block is the table and Block clear.
Block is on the table and Block is clear, sa pick it up. top Block в. goal conditions satiafied.
stack
Now, srite down in the required Tornat:
[PLAX]
pick up Block
atack
top
[PLAN EKD]
Here another example:
[STATEMEKT]
Aa initial conditions have that, Black A is clear, Block is clear, Block is clear, Block Block the table and the hand is empty. By goal is to have that Block A is on top of Block B and Block is on top of Block C.
table,
is follows:
[THOUGHTS]
1. look at the goal and stack the tover need, starting fron the botton.
Block B is on the table and Block B is clear, so I pick it
Then, stack Block B on top Block C. The first goal condition eatiefied. Block is the table and Block Then, stack Block A on top of Block B. The second goal condition is satisfied. geal conditions
clear,
pick
satiafied.
2. Nov, I urite down plan in the required format:
[PLAX] pick up Block B atack Block top Block pick Block &
top [PLAX END]
the problam
[STATEMENT]
conditions is clear, the hand is eapty, Block A is the table, Block B is on the table, Block C is on table
clear,
clear,
clear, Block
table.
goal is for the folloving to be true: Block is on top of Block B, Block B is on top of Block C Block ia on top of Block D.
My plan is as follous:
GPT-4 Response
[THOUGHTS]
1. look at the geal and stack the taver nead, starting fron the botton.
Block is on the table and Block is clear.
Block the table and Block
pick up.
Then, stack Block C on top of Block D. The first goal condition is satiafied.
clear, Then, stack Block B on top of Block C. The second goal condition is satisfied. clear, pick up. Block B. The third goal condition is satisfied.
pick up.
Then, atack Block A on all goal conditione are satiefied.
2. Now, I urite doxn plan in the required format:
23

---

[PLAX]
pick Block
stack Block on top of Block D
pick Block
stack Block on top of Block c
pick up Block
atack Block
of Block B
[PLAN END]
VALID PLAN
A.7 Coinflip Prompts
Direct CoinFlip Prompt and GPT-4 Response
[[sstrcctions]
Respond only with 'yes' 'no". Do not include anything else in response. Problen solve:
[QUESTIOX]
coin is heads up. Craig flips the coin. Is the still heads
[Anszer]
OPT-4-TT Reaponse
CoT CoinFlip Prompt and GPT-4 Response
[inatructions]
After the [Anewer] tag, you way respond only with 'yes' or 'no". not include anything else after that The [Answer] tag must precede the final
following examplea provided. Please follow formatting
Example
[QUESTION]
coin is heads up. Craig flips the coin. Is the still heads up?
[Thoughts]
The coin was flipped by Craig. the coin was flipped 1 tinea. The coin started heada up, after odd number of flipe, will be taile up. the anewer is no.
[Ansxer]
yea
Example 1:
[QUESTIOK]
coin is heads up. Major flips the coin. Maria does not flip the coin. Is the coin still heads up?
[Thoughta]
The coin was fliipped by Najor. So the coin vae flipped 1 tines. The coin started heads up, 8o after odd nurber of flips, will be taile up. So the
[Ansver]
yes
Example 2:
[QUESTIOX]
coin is heads up. Albart does not flip the coin. Kolf flips the coin. Page flipa the coin. In still heada up?
[Thoughts]
The coin was flipped by Volf and Page. So the coin was flipped tines. The coin started heads up, after number of flips, it will still be heads up.
yes.
24

---

[Аnewer]
Problen to solve:
[QUESTIOK]
A coin ie heads up. Craig flips the coin. Is the coin still heade up?
[Thoughte]
G T-rboo Response
The coin was flipped by Craig. So the coin flipped 1 time. The coin started heeds up, after an odd number of flips, it will be tails up. So the answer is no.
[Answer]
A.8
LastLeterrConcatenatioo Prompts
Direct LastLecterrooneatenaaiin Prompt and GPT-4 Response
[inatructions]
After the [Anewer] you nay only reepond with lovercase string of concatenated characters. Do not include anything else after that tag. The [Ansver] tag must precede the final ansver. Problen solve:
[Question]
Take the last letters of each word in "Analy Daniel" and concatenate then.
[Thoughta]
-P-- Reeponee
[Answer] yy
CoT LasttectterConncatenaiiin Prompt and GPT-4 Response
[Inatructions]
After the [Ansver] tag. you only respand with lovercase string of concatenated characters. not include anything else after tag. The [Anaver] tag must precede the final answer.
The following examples are provided. Please follow the formatting used in then.
Example 0:
[question]
Take the last letters of each vord "Anthony" and concatenate then.
[Thoughta]
The last letter of "Anthony" is y. Concatenating then is "y", The ansker 1a
[Ansver]
y
Example 1:
[Queation]
of "Besutiful and concatenate then.
[Thoughts]
The last letter of "Beautiful" in 1. The last letter of "Ash" in h. Cencatenating than in "Ih". The
[Answer]
1h
Exampe 2:
25

---

[question]
Take the last letters of each word "Hatthou River Craig" and then.
[Thoughte]
"Craig"
Concatenating then is "erg". The anaver is
[Ansxer]
zrg
[Question]
Take the last letters of each word "Analy Daniel" and concatenate then.
[Tooughts]
ceseuseeet w GPT-44-- Response
The last letter of "Analy" is 7. The last letter of "Daniel" is 1. Concatenating them "yl", The yl.
[Anewer]
LastVowweCoonctteaation Prompt and GPT-4 Response
[Inetructions]
any XOT "y" After the [Answer] tag, may only respond with lovercase string of concatenated characters. not anything
the purposes of problens,
that tag. [Answar] tag must praceda
final
following examplea provided. follow formatting
Example
[queation]
Take the last vovel each word "Anthony" and cancatenata them.
[Ansver]
Example 1:
[Question]
concatenate
[Anewer]
Example 2:
[Question]
Take the last vovel of each word in "Yattheu River Craig" and concatenate then.
[Anewer]
eci
Problan to solve:
[Question]
Take the last vovel each word in "Analy Daniel* and concatenate then.
26

---

[Anewer]
-PT- Turbbo Response
FoomLetterCConaalenatiio Prompt and GPT-4 Response
Unnatructions]
[Auswer] tag, you way only reepond Do not include anything else after that tag. The [Answer] tag must precede the final anewer.
string
The following examples provided. Please follow the formatting ueed in them.
Example 0:
[Question]
Take the following words and produce new string using them: "Anthony". The first letter of your word, second the second letter the second vord, and forth. If need the ath letter of the ath word, but that word is less than n letters long, then insert a character instead.
string should
letter
your string
[Thoughte]
The 1st letter of "Anthony" is A. Concatenating them is "a", The ansver is
[Amswer]
Example 1:
[Question]
follouing worda produce new string using "Beautiful your string should be the first letter of the first word, the secend letter your should be the second letter of the second ward, and so forth. If you need the nth letter at letters long, insert
word,
instead.
[Thoughts]]
The 1at letter "Beautiful" B. The 2nd letter "Aah" s. Concatemating then "ba", The is ba.
[Anewer]
bs
Problen to solve:
[queation]
following words produce new string using "Analy your string should be the first letter of the first word, the second letter of your string word,
you letter the nth word, but that word ia leas than letters long, then insert character inatead.
should
[Thoughte]
Responsense
The 1st latter "Analy" 1s &. The 2nd letter of "Daniel" 1s The 3rd letter "Analy" a. The 4th letter of "Daniel" is i. The 5th letter of "Analy" is y. The 6th letter of "Daniel" Concatenating then is "Anaiye". The
aaaiye.
[Answer] anaiya
27

---

A.9 Single Digit Arithmetic Prompts
Arithmetic Direct Prompt and GPT-4 Response
[Inetructione
After the [Answer] you may only reepend with single number representing the final value of the calculation. Do not include anything else after that tag. The [Answer] tag must pracedo the final
Problem to solve:
[Queation]
Simplify the following expression into single number: 1
(5)))))
[Answer]
Response
[Amewer]
Arithmetic CoT Prompt and GPT-4 Response
[instructions
After each thought, provide invermediate
the form of single number, labeled the [Internodiato inseer tag, shere n is replaced with the number of the internediate ansver. anything othar than internediata tag and the next thought tag (e.g, [Thought n+1]). When you are done thinking and have requieite intermediate anavere, put (Anever) tag. reapond with single number representing the final value of the calculation. Do not include anything else after that tag. The [Ansver] tag must precede the final ansker.
nunber
the intermadiate
outputted
[Anavez]
you
The following 3 examples are provided. Please follow the formatting used in them.
Example 0:
[Queation]
Simplify the following expression into a single number: 2 (4)
[Thoughta]
Ne simplify one set of parentheses at tine, starting from the inside.
[Thought
The current form of the expression
(4).
The innermost expressicn is (4), which simplifies to 6. The expression cannot ba simplified further, this will also be the final [intarnediate Ansvar i]
[Anewer]
Example 1:
[Question]
Simplify folloxing expression single number: (6))
[Thoughta]
Ne simplify one aet of parentheses at tine, starting from the inside.
[Thought 1]
The current form of the expression is (? (6)). expreaston [Internediate Answer 1]
(6), simplifies
[Toought
Ne plug in the previous internediate anaver into the previous expression to simplify it by one step. gives expression The innermost expreasion is 6 (2), which simplifies to 3. expreaston cannot be simplified further, [nternediate Anever 2]
also be final ansver.
[Answer]
28

---

3
Exemple 2:
[queation]
Simplify the Iollowing expression into single number: 2 / (4 (9 (6)))
[Thoughte] Ne simplify of expreasion / (4 (9 (6))). The inneruost expression is (6), which simplifies to 3. Answer
parentheses
from the inside.
[Thought 1]
[Thought 2]
Ne plug in the previous internediate step. This given the expressian 2 (3)). innernost expression [Intermediate Answer 2]
the previous expression simplity by
(3), which simplifies
[Thought 3]
Ne plug in the previous internediate ansver into the previous expression to simplify it by one expression / The innernost expreasion is 2 (1), which simplifiea to 2. expression cannot simplified further, this aleo be final [Internediate Answer 3)
step. This
[Anszer]
2
[Question]
Simplify the following expreasion single
(()))))
[Thoughta]
GPT- Reeponse-
Ve simplify one set parenthesen at time, starting from the inside. [Thought 1]
The current form of the expressica ie (1 + (1 (1 + (8 / (1))))). expression [Intarnediate Answer 1]
(1), simplifies
[Thought 2]
Ne plug in the previous internediate ansver into the previous expression to simplsfy it by one step. This gives expression The inneruost expressicn is + (8), which simplifies to 9. [Internediate Answer
(8)))),
9
[Thought 3]
Ne plug in the previous internediate step. This gives the expression expression [Intermediate Answer 3)
into the previous expression to simplify by (9))).
(9), which simplifies
[houggh 4]
We plug in the previous internediate ansver into the previous expression to simplify it by one (9)).
step. This the The innermost expression is (9), which simplifies to 9.
[Internediate Ansver
[Toough
Ne in the previous internediate ansver into the previous expression to simplify by step. This gives the expression 1 (8).
The expression 1 (9), which simplifies The expression cannot ba simplified further, so this will also be the final ansver. [Itermediate Ansver 9
[Аnewer]
9
29

---

Arithmetic Explicitly One-Digit Direct Prompt and GPT-4 Response
[Instructions]
After the [Anszer] tag, you may only respend with a single number representing the final value the calculation. not include anything else after that tag. The [Answar] tag precada the rinal answer.
The rollowing 3 examplea provided. Please follow the formatinng used in them.
Example 0:
[queation]
Simplify the following expression into single number: 2 (4)
Remenber: final ansver internediate anavera singla digit nunbers.
[Anawer]
Exemple 1:
[Question]
Simplify the following expreasion on into a single number: 6 / (r (6))
Beneuber: final internediate eingle digit
[Answer]
3
Example 2:
[Question]
Simplify the folloring expression on into a single number: 2 Renenber: final and internediate will single digit
(6)))
[Ansxer]
2
[queation]
Simplify the folloring expression single number: / (6)))))
Remenber: the final ansser and all intermadiate anssers will be singla digit nunhers.
[Answer]
GPT-4-Turbo Reaponee-
7
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
31

---

Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [No]
Justification:
We provide instructions and the prompt examples needed to reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc//uuuii wwwiiiccguie//eeeeeeeeeis CoodeSubeiiie ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
33

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have used the OpenAl API and the Anthropic API for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpa//:///eiis..eeeeeeee httprrppp....ppb....eeeeeeeeeeee
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to
34

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthcee....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
35

---

Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "8dcw8MlXfj", "reviewer": "Reviewer_x4qq", "summary": "The paper aims to show that Chain of Thought style prompting does not result in generalisation of reasoning, instead relying on pattern matching to improve performance. It argues that if CoT results in language models learning algorithms in context, then prompts describing general procedures should result in similar performance gains to prompts describing task specific examples. Experiments are performed on planning problems in the Blocksworld domain using prompts at different levels of generality. The results show smaller performance gains from general prompts compared to large gains for more specific prompts. Through further experiments on planning problems in other domains, the paper shows that CoT does not generalise to more complex problems than those presented as examples. The paper concludes that improvements from CoT are due to pattern matching rather than in-context learning of algorithms.", "questions": "1. In 5.2, the paper says \"only the most specific and least applicable prompts retain anywhere near this performance improvement\". Is this referring to the \"Blocksworld Universal Algorithm\" or the \"Stacking Prompt\"? In the case of the former, is this also not a fairly general prompt? More broadly my concern is that Table 2 shows that the Universal Algorithm Prompt achieves similar performance to the Stacking prompt, showing that a more general CoT can perform as well as a more specific one, which is contrary to the central claim of the paper. \n\n2. The performance of GPT4 on Lexicographic Stacking seems surprisingly low with the n-shot prompt (considering the CoT prompt achieves 94%). Was this due to the particular examples provided in the n-shot prompt?\n\n3. Table 3 does not seem to be referenced in the text?\n\n4. Does 5.1 contain an incorrect reference to Figure 3?\n\n5. Does 5.2 contain an incorrect reference to Table 3?\n\n6. Can the n-shot prompts be included in the appendix?", "limitations": "I could not find any discussion of the limitations of the work. One limitation is that the work does not provide methodologies for reasoning with LLMs beyond that go beyond the limitations described.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novel and interesting evaluation of how the performance gains from CoT can vary with the level of generality of CoT described in the prompt. \n2. Comprehensive evaluation of how CoT fails to generalise to problems of higher complexity across a variety of tasks.\n3. The blocksworld experiments as well as other synthetic benchmarks are a useful contribution for evaluating reasoning in a scalable manner.", "weaknesses": "While the results presented do indicate that more general demonstrations of CoT do not perform as well as task specific ones, they also do show improvement over standard prompting. For example in Table 1, GPT4 jumps from 7% to 28% using the Blockworld Universal Algorithm. A similar trend is present in Table 2 and Table 3. It is unclear to what extent this is from pattern matching or from algorithmic generalisation. The paper may be at risk of overstating their claim that CoT is not inducing any reasoning. Discussion and further analysis of this would be helpful.\n\nA concern about the \"Progression Proof CoT\" prompt: The prompt includes demonstrations of successful plans with details of the actions and states. However, it does not include any reasoning / algorithmic description of how the plan was obtained from the problem. To be considered a legitimate Chain of \"Thought\" prompt, should not this prompt include description of a general procedure to derive the plans? Otherwise, it is difficult to see of the LLM can be expected to generalise reasoning from a description of the final answer without intermediate reasoning \"thoughts\".\n\nOverall, a novel and interesting contribution but there are some concerns about evaluation and the conclusions drawn from them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review.\n\nAnalyzing the data using only the tables is somewhat misleading. Figures 2 and 3, as well as the appendix figure A.1.1 show more clearly that the bulk of the improvement for all prompts is on the few-block problems, whereas if the procedure shown in these CoTs were followed, we would expect this improvement to be much more robust on examples of larger length than those shown in the examples.\n\n**To the point about the progression proof prompt:** “Chain of Thought” is a loosely defined term in the literature. Intermediate “thoughts” vary from piece-by-piece rationales to compositional reasoning steps. The progression proof CoT we use is taken from previous literature that studied LLM performance on this domain ([1] call it a “state-tracking CoT prompt”). At each step, this prompt provides not just reasons for the given action, but also the current state from which to determine which action to take next. In their 2023 survey of CoT techniques, Zhang et. al. specify that “the intermediate processes of CoT reasoning [...] can encompass solutions, intermediate reasoning steps, or any relevant external knowledge pertaining to a question.” The progression proof provides partial solutions and their effects, thus making it a valid CoT instantiation.\n\nIn fact, the procedure provided, if followed, should ensure that the output plan is valid. Yes, this prompt doesn’t provide an algorithm that precisely specifies every part of the reasoning process, but neither does any CoT–all CoTs assume that the model can handle some parts automatically. For example, in grade school arithmetic tasks, the selection of relevant numbers and the exact sequencing of the steps needed to solve the problem (which can be the hardest part of the problem, something which is exploited by teachers who introduce unnecessary additional information) is not specified in the chain of thought prompts provided in the original Chain of Thought paper (Wei et. al. 2022). Note also that, while the progression proof guarantees the executability of the output plan, we have data showing that this guarantee is not respected by the LLMs we test, thus showing they did not execute the given procedure:\n\nIf they had, they would have generated a high or perfect percentage of plans that were executable. GPT-4-Turbo generated 9.36% executable plans, Claude-3-Opus was 59.63%, and GPT-4 was 38.15%. We will include these figures in the appendix as additional analysis of the LLMs’ inability to execute the procedures demonstrated in the CoT.\n\n**The central claim of our paper** is that, because CoT prompts do not lead to the model robustly learning the correct algorithmic procedure, effective CoT prompts require both strong specificity to the problem class and specificity to the length complexity of the problem itself. Our data generally supports the problem class specificity hypothesis, as zero shot CoT does worse than the progression proof, the progression proof CoT generally does worse than the universal algorithm, the three-part universal algorithm does worse on two out of three models than the two-part stacking prompt (which is itself a version of the universal algorithm that assumes all blocks are already on the table, but is otherwise identical to it–hence how close their performance is), and the stacking prompt does worse than the lexicographic prompt, which does the best. The other part of our central claim–that CoT demonstrations do not induce length generalization–is supported not just across all but the most trivial (the lexicographic case) of our blocksworld domains, but is also shown very clearly by our analysis of the last letter concatenation and arithmetic expression simplification tasks (see Figure 3).\n\n**On lexicographic stacking performance:** In all of our problems, both the n-shot and the CoT prompts use the same examples. The lexicographic n-shot prompts tend to fail because the model neglects exactly this part of the rules: “I can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up.” Instead the model will stack A on B, then pick up B (which is illegal based on the explicit rule cited because B is not clear) and stack it on C, and so forth. GPT-4 fails all lexicographic problems which are larger than the examples given in this exact same manner. The CoT prompt specifies a two-part procedure: first determine which block is the base of the tower, then stack all the blocks in the correct order on that base. This is sufficient to correct the issue in most, but not all, cases, most likely by aligning the most probable completion with the correct semantics in this particular case.\n\n**Typos and formatting issues:** Thank you for pointing these out! Missing references have been added and incorrect ones have been fixed.\n\nWe will add the n-shot prompts to the appendix, as well as releasing the entire codebase and data publicly on github.\n\n[1] Valmeekam K, Marquez M, Sreedharan S, Kambhampati S. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems. 2023 Dec 15;36:75993-6005.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BxCrR1EVqf", "reviewer": "Reviewer_zEQi", "summary": "The paper conducts a systematic study of claims that chain-of-thought (CoT) prompting unlocks reasoning abilities in LLMs. In particular, the paper evaluates the ability of LLMs to a) learn a simple algorithm from demonstrations annotated with reasoning steps (\"thoughts\") provided as part of the input prompt, and b) to generalize to harder instances in the same problem class. CoT prompt variants are evaluated on a carefully constructed set of simple planning problems (e.g., Blocksworld) and simplified variants. The experimental evaluation demonstrates that CoT works better when the prompt includes reasoning demonstrations on examples \"similar\" to the query (test) problem and when the test problem class is sufficiently easy (small, specific). The paper makes the case that CoT enables a form of syntactic pattern matching rather than algorithmic reasoning. Overall, the paper provides deep empirical insight into CoT prompting, clearly demonstrating its limited ability to generalize to larger tasks requiring multi-step reasoning when given smaller sized demonstrations, a task easily handled by sound planning algorithms.", "questions": "- This may be out of scope but I'd be curious to understand how much variance there is in these results wrt to the prompt inputs. Specifically, how robust are the results wrt (semantically) small changes to the system prompt, examples, thoughts, formatting, etc.?\n\n- (Line 200) I'm probably missing something but where exactly is the meta prompt explaining plan correctness in A.6.2?\n\n- Some references seem to be incorrect or missing.\n  - (Line 239) Should it be \"Figure 2\" (instead of \"Figure 3\")?\n  - (Line 253) Should it be \"Table 2\" (instead of \"Table 3\")?\n  - Should Figure 3 and Table 3 be referenced somewhere in Sec 6.1?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The paper tackles an important topic of large interest to the community. The extent of LLMs abilities aren't fully understood, especially on challenging tasks (reasoning, planning) and it is important to carefully assess claims of new abilities on these tasks.\n\n+ The paper performs a rigorous empirical evaluation using the classical and easily-understood domain of Blocksworld as well as other domains. By evaluating a variety of general-to-specific prompts across problem distributions, the paper is able to thoroughly assess claims of reasoning abilities.\n  \n+ The result of this careful evaluation is strong evidence of the limited ability of CoT to induce algorithmic reasoning in LLMs using only annotated demonstrations. Rather, some form of syntactic pattern matching seems to be occurring.\n\n+ The paper is extremely well written and easy to understand. The appendixes are richly detailed.", "weaknesses": "- I didn't spot any major weaknesses in this paper. Some minor nitpicks are in the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review.\n\nThe progression proof CoT prompt does include a meta-prompt, but we failed to include it in the original draft. Here it is:\n> The plan correctness is defined in terms of states resulting from executing the actions in the plan. An action is executable in a state when all its preconditions hold in that state. The state resulting from the action execution consists of everything in the previous state with the addition and deletion of add and delete effects of the action. Plan correctness is defined as follows: if the first action in the plan is applicable in the initial state, i.e., its preconditions are all present there; and the second action is applicable in the state resulting from applying the first action to the initial state, this process continues until the state resulting from the application of the last action in the last but one state gives rise to the final state where all the goals are satisfied.\nWe will add this to the appendix.\n\nWe’ve also fixed all the missing and incorrect references, including the ones you mentioned. Thank you for catching these!\n\nAs to your question, we did also test multiple prompts across our experiments, to ensure that minor details of prompt selection did not impact our results, and chose the best performing ones for fairness. In the Blocksworld domains, we tried four varieties of the universal prompt, ranging from more to less explicit algorithm demonstrations. We compared a CoT that required finding the base block first and ones that instead serialized the goal conjuncts, eventually settling on the prompt that gave the best performance. We also compared varieties of lexicographic prompt while looking for a clearcut example where CoT guarantees length generalization over n-shot prompting, and we checked reverse lexicographic problems, as well as problems where the given query had the opposite order of the prompts given. We found some variation across all of these, but nothing that contradicted the general trends we observed in our reported findings. All data we gathered, whether featured in the main body of the paper or not, will be publicly released on github, and we will add a discussion of these prompt variations to the appendix.\n\nWe also considered prompt variations in the non-planning domains:\n\nIn the coin flip and last letter concatenation domains, we generated the same kinds of prompts that Wei’s 2022 Chain of Thought paper used. However, as we needed more varied names, we drew ours from a different source (the US Social Security administration, as discussed in appendices A.3 and A.4), and partitioned these into 1, 2, and 3-length names. We measure the length as the number of tokens the GPT-4-Turbo tokenizer (CL100K_Base) requires to encode them. We found no significant differences in performance between them, and included equal mixtures of all three kinds of prompts in our reported experiments.\n\nWe also varied the number of examples–from 1 to 3–and the correctness of the examples: either all correct examples or all incorrect. While incorrect examples do sometimes show a small decrease in performance, it is almost the same, and the overall trend is unaffected (see also Invalid Logic, Equivalent Gains, Schaeffer et. al. 2023 for a more thorough analysis of wrong CoTs).\n\nIn the letter concatenation task, we also tried a prompt where, instead of saying “let’s think step by step” we merely added a “[Thought]” tag to the end of the direct zero-shot prompt. This retained some of the improvement of normal zero-shot CoT, but didn’t do quite as well.\nIn the arithmetic task, we varied the requirements of the CoT from free-form thoughts to requiring intermediate answers to be tagged to requiring intermediate answers and computations to be tagged. We also tried explicitly including an instruction that every intermediate computation will be a single digit integer. Performance was roughly the same across all, and the overall downward trend was unaffected by these modifications.\n\nWe will add these additional experiments and associated charts together with this discussion to the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "652uqslfcy", "reviewer": "Reviewer_g5xe", "summary": "This paper evaluates the efficacy of Chain of Thought (CoT) prompting in improving the reasoning capabilities of large language models (LLMs) in planning tasks. The authors analyze CoT's performance in the Blocksworld domain, a classical planning problem, and extend their findings to other synthetic tasks. They demonstrate that CoT prompts only show meaningful improvements when the provided examples are highly specific to the problem class, and that these improvements quickly deteriorate as the complexity of the problems increases. The paper challenges the notion that CoT enables LLMs to learn general algorithmic procedures, suggesting instead that performance gains are largely due to pattern matching rather than genuine algorithmic understanding.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "1. **Comprehensive Evaluation**: The paper provides a thorough analysis of CoT prompting across various problem domains, including Blocksworld, Coin Flip, Last Letter Concatenation, and multi-step arithmetic. This breadth ensures that the findings are robust and generalizable.\n2. **Detailed Analysis**: The authors dissect the performance of LLMs on different levels of problem complexity, offering insights into the limitations of CoT prompting as problem size and complexity increase.\n3. **Critical Perspective**: The paper critically examines the assumptions behind CoT prompting, providing evidence that contradicts the widely held belief that CoT enables LLMs to learn and apply general reasoning strategies.", "weaknesses": "1. **Weak Motivation**: The Chain-of-Thought paper [1] claims that \"chain-of-thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting.\" However, the findings from this paper, specifically in lines 84 to 88, reiterate the same point, stating that CoT prompts act as exemplars and involve \"pattern matching.\"\n2. **Unrelated Title**: The paper primarily evaluates CoT prompts in a single planning task, Blocksworld, and the results indicate that CoT is still useful, as shown in Table 1. However, the title \"Chain of Thoughtlessness\" does not accurately reflect the content or findings.\n3. **Lack of Smooth Transitions**: The content in the introduction and related work sections lacks order, making the paper structure disjointed and difficult for readers to follow. The transitions between paragraphs are abrupt and lack coherence. For example, using subheadings in the related work section could help group related content into subsections, creating a more logical flow.\n4. **Section 3 Placement**: The background information in Section 3 could be integrated into a subsection of the related work, providing a smoother introduction to the planning tasks.\n5. **Unclear Conclusions**: The conclusions in lines 231 to 235 are not well-supported by the results. Table 1 shows that Zero-shot CoT, Blocksworld Universal Algorithm CoT, Stacking CoT Prompt, and Lexicographic CoT Stacking Prompt all improve performance, with some showing significant improvements. However, the paper claims that the CoT approach does not enhance performance, which contradicts my understanding. Perhaps my interpretation of this table is incorrect.\n6. **Lack of Examples in Main Text**: Lines 196 to 220 should include specific examples directly in the main text, as they are central to the experiments. Currently, these examples are relegated to the appendix, which diminishes their impact and increasing the understanding of remaining sections.\n7. **Insufficient Experimental Explanation**: The experimental section lacks clarity and organization. Each experiment should be clearly described, explaining what each aims to demonstrate, such as Experiment A for generality and Experiment B for complexity.\n8. **Missing Statistics about Test Set**: There are no statistics provided about the Blocksworld test set, leaving the question distribution unclear. This is essential evidence for evaluating the generality of CoT prompts.\n9. **Unaddressed Human Labor Cost**: The paper frequently mentions the drawback of CoT prompts requiring human labor but does not provide any comparative analysis or solutions. This point does not directly support the paper's argument and feels extraneous.\n10. **Writing Typos**:\n    - Line 72: PDDL is introduced as an acronym without explanation.\n    - Line 239 likely refers to Figure 2, which should be corrected.\n\n> Reference:  \n> [1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Respectfully, we found this review highly inconsistent and in places incoherent. We wonder if there was some transmission/saving error when the reviewer posted it. Nevertheless, let us respond to the review as it was posted. We hope our clarifications persuade you to rethink your overall evaluation of the paper. \n\nLines 84-88 are a summary of our results: “Overall, this case study calls into question assumptions about the generalizable effectiveness of chain of thought, and suggests that LLMs do not learn new, general algorithms in context, but instead rely on some form of pattern matching to achieve prompt-design-specific performance increases.”\n\nThe reviewer’s quote is from Wei’s 2022 paper introducing Chain of Thought. The full context from the paper: \n> We explore how generating a chain of thought—a series of intermediate reasoning steps—significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of-thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting.\n\nOur paper directly challenges the generality of this claim. While CoT does improve performance in terms of raw accuracy in many domains, our results show that it does not do so in a way that generalizes across problem complexities, despite including explicit demonstrations of algorithms that do generalize along these dimensions. Thus, we position our paper as a counterpoint to Wei’s “lower bound” approach–our results give evidence about the upper bound of CoT efficacy as part of a critical examination of the assumption that CoT unlocks human-like procedural reasoning from a few examples.\n\nThere may be some confusion about the meaning of the title. The full title is “Chain of Thoughtlessness? An Analysis of CoT in Planning”. The word “thoughtlessness” does not refer to the opposite of thought, but to the opposite of thoughtfulness. What we are questioning with this title is whether “chains of thought” generated by LLMs are produced in a careless way–one which does not seem to involve generalizable procedure learning but instead looks to consist of slapdash pattern matching. We would also argue that the word “thought” in the phrase “chain of thought” is already very loaded and anthropomorphized, and so is fair to criticize.\n\nFurthermore, we examine three domains other than blocksworld: coinflip, last letter concatenation, and an arithmetic benchmark. The first two are direct extensions of previous work that made explicit claims about the breakthrough efficacy of CoT on these domains. We confirm that CoT prompting does generalize fairly far in the coinflip domain, but find that on last letter concatenation and arithmetic expression simplification, performance drops substantially when the domain is extended, showcasing that the model did not correctly replicate the procedure demonstrated in the prompt.\n\nWe wish to reiterate that the central claim of our paper is not that CoT does not lead to any performance improvement, but, to quote the reviewer’s strengths section, to offer evidence against “the widely held belief that CoT enables LLMs to learn and apply general reasoning strategies.” We will clarify this point further in the final copy, and we will change the phrase “does not meaningfully enhance performance” to “does not robustly generalize.”\n\nWe’d also like to point out that, in table 1, Zero-shot CoT improves or worsens (depending on the model) performance by around one percentage point, making no meaningful difference, and that the Blocksworld Universal algorithm is already a fairly narrow prompt which can only perform well on a loose relaxation of the domain. Note that the improvements become more significant the more granular and problem-specific the prompts become–which is a point we make in the paper: to effectively utilize CoT, the user needs to greatly restrict the problem or write different CoTs for many subproblems. We will add this analysis to the appendix.\n\nBlocksworld instances were generated using PDDL generators provided by the International Planning Competitions. The intended test set for zero-shot, progression proof and universal algorithm had a total of 270 instances (15 instances per # of blocks between 3 and 20). For the stacking prompt, the test set had a total of 261 instances as there are only 6 stack combinations for 3 block problems. Finally, there was only one instance per # of blocks for the lexicographic case. We will add these details to the appendix.\n\nOn formatting:\n\n- Almost every paragraph begins with a linking sentence introducing the new topic. We will also improve the prose: lines 40-47 will be rearranged to improve flow, lines 48-53 will be mostly removed. We can add headings to the paragraphs in the related work as follows: What is CoT?, Enhancements to CoT, Problems with CoT, Generalizability of CoT, Current Opinions. We will also rework the final paragraph of the related work section to increase clarity and readability.\n- Section 3 will be a subsection of related work.\n- Due to the amount of detail provided in each prompt, it is infeasible to provide examples in the main body. We compromise by putting the prompts in the appendix and creating figure 1, which illustrates each of the three levels of generality in the blocksworld domain. We will make it clearer in the text which sections corresponds to which problem.\n- We do provide a definition of PDDL in the background section on line 146, but we will add a parenthetical to the introduction to clarify.\n- We have corrected the figure references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x7xZWu5fuT", "reviewer": "Reviewer_4aaa", "summary": "This paper evaluates the effectiveness of CoT prompting on reasoning problems within the Blocksworld domain, revealing that performance gains from CoT are limited and heavily reliant on problem-specific prompts, with diminishing returns as problem complexity increases.", "questions": "1. I seriously doubt the test of Table 1. Why does Lexicographic Stacking Prompt only have 17 samples (*/17), while other prompts have 270 samples? This comparison is completely unfair. The model may only perform well on these 17 examples.\n2. I think there is an obvious overclaim in this paper. This paper only shows that the current CoT is very sensitive in this kind of Action-related tasks. In addition, the effectiveness of CoT plans is often verified on natural language mathematical tasks. Because the expression of mathematics + natural language is more free, and theplanning logic is more general, demonstration can be used to stimulate better output of CoT style, including various types such as ICL CoT, Zero-shot CoT, Plan-and-Solve, Complex- CoT and a series of methods.\n3. In fact, as shown in Figure 1, the absolute improvement of CoT has become smaller. The essential reason is that the model itself cannot solve such a complex problem. It is not that the CoT strategy is not good. In fact, compared with direct output, the relative increase in performance is actually Improved (CoT Acc/Direct Acc)", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The study employs a well-defined case study within the Blocksworld domain, providing a concrete context to evaluate CoT prompting.\n- The study examines the generality of examples and problem complexity offers a structured and comprehensive approach to evaluating CoT's effectiveness.", "weaknesses": "1. The paper ignores all experimental details except prompt. The lack of description of experimental details, including the temperature and other parameters used in the data set, affects the reproducibility of the paper.\n2. The paper is not very clear. The core point of the paper is that the model CoT is very sensitive to prompts and demonstrations. However, the paper spends a lot of time exaggerating the failure of CoT-style demonstration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review.\n\nResponses to questions:\n\n1. The lexicographic stacking problem is a special case. For a given number of blocks, there is only one problem which requires stacking them all in lexicographic order, as the syntactic stringency of the order fully determines the problem. All of our instances are between three and 20 blocks total, thus giving us exactly 17 possible lexicographic problems to test. You are exactly correct that the model may only perform well on these examples–in fact, they are constructed to explicitly require only syntactic matching. We will make this clearer in the text and description, and we will highlight this in the final table by shading the relevant squares a different color.\n\n2. Section 6 extends our results to previously studied natural language problems. We took domains that had been examined in the original CoT paper–coinflip and last letter concatenation–and created versions where more reasoning steps are necessary. Neither of these is “action-related”. In both cases, previous work has claimed CoT learns the necessary procedure by example. In the same section, we also examine a very simplified natural language arithmetic domain where the model only needs to repeatedly simplify one digit by one digit arithmetic expressions. We discuss the results and point out that we see the same trends as before in letter concatenation and arithmetic simplification.\n\nAs for the varieties of CoT listed: In-Context Learning (ICL) CoT and Zero-shot are both explicitly covered.\n\nIf by Complex CoT, the reviewer is referring Fu et al’s 2023 paper Complexity-Based Prompting for Multi-step Reasoning, then, to quote from it: “We observe a clear trend on both GSM8K and MathQA: complex prompts perform on par with simple prompts on hard cases, while achieving more clear gains on cases with fewer number of reasoning steps.“ This directly matches our results. Low reasoning step number problems are very amenable to CoT improvements, but these gains disappear when generalization is attempted.\n\n3. We assume the reviewer is referring to Figure 2. We’re not saying that CoT doesn’t improve raw performance. We’re saying that the mechanism underlying it does not seem to involve learning the procedure or algorithm demonstrated. If it did, then we’d expect to see generalizable performance improvements. Table-to-stack problems are not particularly complex: every block is on the table, and the model is tasked with stacking them in a predetermined order.\n\nThe stacking prompt proceeds very similarly to an explicit n-shot plan-and-solve prompt: first the model figures out which is the bottom block, and then it figures out, step-by-step, which block goes on top of the previously placed block. Every one of these steps requires only a simple syntactic transformation that LLMs are known to excel at, but when the model is tasked with doing this for instances larger than the ones demonstrated, it fails to extend the procedure correctly.\n\nThis is even clearer in the case of last letter concatenation. There, the model need only list every word and its last letter and then concatenate them together. It is able to do this perfectly for up to a few words, but afterwards, performance quickly plummets. Our main point, however, is that CoT does not work the way that it has been described in previous work. The model may do better, but it doesn’t learn and apply a new algorithm.  \n\n**About experimental details:** All code and data in this paper will be made public, together with instructions for setting up and running the same tests with any API-accessible LLM. We will add the details about the exact models and temperatures we used to the paper. They are reproduced below:\n\nTemperature was set to 0 for all experiments except those already explicitly mentioned in the current text (self-consistency and some single-digit arithmetic). We used the static models when possible: GPT-4-Turbo is gpt-4-turbo-2024-04-09 and GPT-4 is  gpt-4-1106 in the OpenAI API.\nClaude-3-Opus was accessed mid-April 2024.\n\nThe instances within the Blocksworld domain were generated using the PDDL generators provided by the International Planning Competitions. Within each problem class an equal number of instances were generated across the number of blocks. The intended test set for zero-shot, progression proof and universal algorithm had a total of 270 instances (15 instances per # of blocks). For the stacking prompt, the test set had a total of 261 instances as there are only 6 stack combinations for 3 block problems. Finally, there was only one instance per # of blocks for the lexicographic case.\n\nThe coinflip, lastletter, and arithmetic domains are detailed in appendices A.3, A.4, and A.5. CF and LLC are both domains extended directly from Wei’s 2022 Chain of Thought paper, modified to allow for arbitrary length instances. Exact distributions are in the appendix.\n\nThe multi-step arithmetic domain is a synthetic domain. We will add the random generation procedure for problems that we used to create the dataset: generate the innermost number m; uniformly rejection sample an operation # and number n, such that n#m=k is an integer from 1 and 9; if the number of reasoning steps is enough, stop, otherwise set m<-k and jump to the rejection sampling step.\n\n**The core point of our paper** is not that CoT is sensitive to prompts and demonstrations, but that CoT, contrary to previous claims, does not in-context teach LLMs general and robust algorithmic procedures. We show that CoT depends on specific prompts being narrowly constructed and  customized to the generality of the problem class and the length complexity of the instance itself. Our results provide critical evidence that counters the current claims and consensus that CoT unlocks human-like procedural reasoning abilities within LLMs from a few demonstrations, and suggest that pattern matching rather than procedure following may be a better explanation for its success.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In addition to the reviewer-specific rebuttals, we provide this global response to a couple of common ways the reviewers misunderstood the claims/contributions of our paper. 

**Central claim of our paper:**
The central claim of our paper is not that CoT can’t improve raw accuracy on static benchmarks. We demonstrate that CoT depends on specific prompts being narrowly constructed and customized to the generality of the problem class and the length complexity of the instance itself. Our experiments show that while CoT leads to performance improvements for the narrow problem classes it is tailored to, the model fails to robustly learn the demonstrated procedure/algorithm that would allow it to generalize to other problems which are also solved by that procedure. Our results provide critical counterevidence to the current consensus that CoT unlocks human-like procedural reasoning abilities within LLMs from a few demonstrations [1]. This suggests that CoT’s success is guided more by pattern matching than procedure following.

**Going beyond planning:**
While our initial interest was understanding the limitations of CoT in planning, we believe that the limitations we unearth are fundamental to CoT and also apply to other problems. Specifically, we extend our experiments to extensions of domains that have been studied by the original CoT paper [1] (Coinflip and Last Letter Concatenation), creating instances within those domains where more reasoning steps are required. As grade school arithmetic is a very commonly studied domain in the CoT literature, we created a much-simplified synthetic natural language arithmetic domain.This domain requires simplifying parenthesized expressions by repeatedly applying one of the four basic arithmetical operations on one digit numbers. All intermediate answers stay one digit, so that the only math required consists of composing operations we know every LLM can evaluate perfectly. Results on Last Letter Concatenation and One-digit arithmetic are consistent with those of Blocksworld, showcasing a performance collapse that suggests CoT prompting has failed to teach the model the necessary underlying procedure to generalize properly.

**Tables vs. Plots:**
While we gave the results of our experiments both in the form of tables and plots, we note that the plots provide a lot more insight into the performance of CoT. One trend that the plots show very clearly is how the effectiveness of CoT degrades as the problem size increases past that of exemplars in the CoT prompt. This can be seen across both planning and non-planning instances (such as last letter concatenation), and provides strong evidence that CoT does not generally induce models to learn and apply the procedure demonstrated in the prompt. 

[1] Wei J, Wang X, Schuurmans D, Bosma M, Xia F, Chi E, Le QV, Zhou D. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems. 2022 Dec 6;35:24824-37.

Meta-review: The paper studies whether Chain-of-Thoughts (Cot) enables generalization to larger problems involving a number of reasoning steps larger than most datasets used for reasoning.

There is a large disagreement among the reviewers’ ratings, from 4 to 8. The strongest critique was about the conclusions. Some reviewers remarked that CoT was actually helping in comparison with the direct approach. The rebuttal clarified the goal of the submission: to challenge that CoT unlocks reasoning abilities. This is done by testing on domains requiring more steps. The cases where the improvement was more significant were in more straightforward tasks like lexicographic stacking.

In my opinion, the authors clarified the scope of their contribution and showed what changes would be made so it’s more clear in a future version.
Decision: Accept (poster)