Chain of Thoughtlessness? An Analysis of CoT in Planning

Authors: kaya  stechly
karthik  valmeekam
subbarao  kambhampati
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:37:32

Abstract: Large language model (LLM) performance on reasoning problems typically does not generalize out of distribution. Previous work has claimed that this can be mitigated with chain of thought prompting--a method of demonstrating solution procedures--with the intuition that it is possible to in-context teach an LLM an algorithm for solving the problem.
This paper presents a case study of chain of thought on problems from Blocksworld, a classical planning domain, and examines the performance of two state-of-the-art LLMs across two axes: generality of examples given in prompt, and complexity of problems queried with each prompt. While our problems are very simple, we only find meaningful performance improvements from chain of thought prompts when those prompts are exceedingly specific to their problem class, and that those improvements quickly deteriorate as the size n of the query-specified stack grows past the size of stacks shown in the examples.
We also create scalable variants of three domains commonly studied in previous CoT papers and demonstrate the existence of similar failure modes.
Our results hint that, contrary to previous claims in the literature, CoT's performance improvements do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts. This spotlights drawbacks of chain of thought, especially the sharp tradeoff between possible performance gains and the amount of human labor necessary to generate examples with correct reasoning traces.

Main paper: 
 introduction: 
 While originally designed for text completion, Large Language Models (LLMs) have shown promise on a diverse set of unrelated tasks. While initial anecdotal results were unexpectedly impressive [8] , followup systematic studies showed that-outside of limited, non-generalizable classes of problemsthese models generally perform poorly on basic, multi-hop reasoning tasks [17] ranging from arithmetic [35] and logic puzzles [14] to constraint satisfaction [42, 2] and classical planning [47] .At the same time, the subfield of prompt engineering [36] has grown rapidly, promising improvements in performance without retraining. A core tenet of this subfield is that LLMs are capable of powerful in-context learning [12, 56] , that is, capable of intelligently using additional context provided in a prompt to correctly respond to queries that would otherwise be answered incorrectly. Generally, this requires operationalizing algorithmic/procedural advice, and, in principle, learning such procedures includes being able to effectively apply them beyond syntactically similar instances.The foundational method for inducing in-context learning is the chain of thought approach, which has been claimed to "unlock the reasoning abilities of LLMs" [50] . To create a chain of thought (CoT) prompt, a user annotates similar problems with intermediate reasoning steps and prepends them to the standard prompt. These annotations are meant as demonstrations, intended to teach a procedure applicable to both the examples and the new query. When prompted like this, the LLM is expected to output a similar series of reasoning steps prior to the new answer. Numerous studies have claimed that this procedure significantly enhances LLM performance in complex reasoning tasks [49, 54, 39, 56, 52, 43] . However, in general it is unclear how "similar" the examples need to be to the problem, how broadly any given chain of thought prompt will apply, and-most importantly-how much human effort is necessary to craft prompts specific to each problem subclasses. Followup work has claimed that merely adding magic phrases ("let's think step by step") to every prompt is sufficient for some improvement [26] . While in some domains, this technique has proven to be even more brittle than manual CoT, it has achieved the same performance increases in others, hinting that improvements observed with CoT may not indicate as much about LLMs' general in-context learning abilities as previously thought.We are interested in the tradeoff between possible performance gains from chain of thought prompt engineering and the amount of human labor necessary to generate examples with useful reasoning traces. Ideally, a properly constructed prompt should teach the LLM how to robustly generalize a basic algorithmic procedure in order to increase performance on a large class of problems, thereby converting a modest amount of human teaching effort into a significant capability boost. Unfortunately, this only seems to be possible to a very limited extent [14] .In the current work, we examine the limits of chain of thought in solving classical planning problems. Test domains commonly used in previous chain of thought studies (e.g. GSM8K [10] , CommonSense QA [44] ) present two significant issues: (a) they lack a systematic method to scale instances, which is essential for evaluating whether LLMs can extend provided procedures to larger instances of the same type, and (b) due to their static nature, are more likely to be well-represented on the web [51] , increasing the chance that they were part of LLM training data, a factor which could obscure the true reasoning capabilities of LLMs. Planning is a well-studied kind of sequential decision-making which tasks an agent with devising a plan that takes a given initial state to a pre-specified goal state. New, diverse, and unique problem instances are easy to generate, but potentially hard to solve.We focus on Blocksworld, a simple commonsense domain widely recognized and utilized in International Planning Competitions [23] , where a set of blocks in an initial configuration must be rearranged step-by-step into a goal configuration. For a subset of our results, we simplify even further, and only consider problem instances where every block starts on the table and the goal is a single stack of blocks. These instances require very minimal reasoning: one need only figure out which block is on the bottom, and then stack the remaining blocks in the sequence directly defined in the goal. For 3 ≤ n ≤ 20, we generate a variety of instances where the goal requires a specific n height stack, while providing examples of how to solve 2 and 3 height instances.We consider different chain of thought prompts, where each is more specific-and provides more problem-specific knowledge-than the last: a zero-shot variant, a general progression proof, a suboptimal algorithm specific to Blocksworld, a table-to-stack specific simplification of that algorithm, and a lexicographic version of the simplification. The most general could be applied to any problem, while the least is specific to an easier version of the stacking problem. The three human-crafted prompts all teach algorithms which could, in principle, solve any of the instances they are tested on. We test on three state of the art models: GPT-4 [3] , Claude-3-Opus, [5] and GPT-4-Turbo.Our results reconfirm that LLMs are generally incapable of solving simple planning problems [47] , and demonstrate that chain of thought approaches only improve performance when the hand-annotated examples and the query are sufficiently similar to the current query. As goal stack size increases, accuracy drops drastically, regardless of the specificity of the chain of thought prompt. As generality of the prompt increases, performance on even the smallest goal stacks also decreases, and often falls short of standard prompting. Even state of the art extensions of CoT (like self-consistency [49] ), show similar or sometimes even worse performance. Overall, this case study calls into question assumptions about the generalizable effectiveness of chain of thought, and suggests that LLMs do not learn new, general algorithms in context, but instead rely on some form of pattern matching to achieve prompt-design-specific performance increases. This in turn increases the burden on humans giving advice.To better compare to previous work, we construct scalable versions of three previously studied synthetic problems-Coin Flip, Last Letter Concatenation, and multi-step arithmetic [49, 50, 26, 48] and replicate reported chain of thought prompts. While these domains do not have a corresponding notion of prompt granularity, they do cover a range of difficulties. When testing on GPT-4-Turbo, We see a similar lack of generalization on these problem sets as we saw in Blocksworld.In the rest of this paper, we first review related work, then describe the chain of thought approaches we have developed in the context of planning, analyze the overall effectiveness of chain of thought prompting on Blocksworld problems, and extend our results to three synthetic tasks well-represented in the CoT literature. 
related work: 
 Modifying text prompts to elicit intermediate problem-solving steps from LLMs originally took the form of scratchpads [33] . [50] proposed a similar prompt style in natural language, dubbing this approach chain of thought (CoT), and claiming that-with some human hand-annotation of examples-this not only boosts performance without retraining, but "allows reasoning abilities to emerge naturally". They argued that by merely interspersing intermediate reasoning steps in natural language into examples, they were inducing the LLM to "learn via a few examples", motivating this idea with anthropomorphizations ("Consider one's own thought process when solving a complicated reasoning task such as a multi-step math word problem"). [26] argued that some of the performance of CoT could be retained without providing any examples, and instead just appending the magic phrase "let's think step by step" to the end of a prompt. This has been called zero-shot CoT.However, CoT has long been known to be imperfect and incomplete. Previous work has investigated improving the consistency of CoT through self-consistency [49] , multi-agent debate [13] , least-tomost prompting [55] , deductive verification [28] , and other approaches. Unfortunately, many of these involve prompting the LLM multiple times for a single problem, which can balloon the cost of inference. Other work has examined the possibility of reducing or removing the need for human annotation of examples by using LLMs to generate their own examples automatically [54, 9] . To avoid well-known issues with the brittleness of LLM self-verification and self-teaching [42, 22, 20, 19, 24] , we restrict this paper's scope to manually written chains of thought.Previous papers have analyzed CoT from multiple perspectives [15, 37] , finding that there is only a loose relationship between the presented chain and the final answer [6] , and that the correctness of provided annotations has little effect on resultant performance [38] . LLM-produced chains of thought are also known to be unfaithful to the underlying reasoning process [29, 25, 11] . In particular, the way the examples are presented can bias a model into giving some answer (e.g. if all the example answers are A, the model will be more likely to output A), but its CoT will not reflect this [45] .Motivated by claims that CoT prompts allow models to learn in context how to reason-that is, to learn how to execute human-specified algorithms-we focus on CoT prompting's out-of-domain generalization. [14] previously showcased a lack of generalization in multiplication, puzzles, and a number sequence problem, even when the model was fine-tuned on CoT examples. However, they only examined one set of prompts, did not experiment with levels of prompt specificity, and were much more interested in local failures of compositionality arising from cumulating error. More broadly, previous work has examined generalization limits of LLMs in arithmetic tasks [35] , formula simplification [34] , and theorem proving [4] .While early accounts claimed LLMs, despite not being trained for it, were capable of reasoning and planning [8] , later work showcased serious brittleness across these domains [47] . [50] claims that "standard prompting only provides a lower bound on the capabilities of large language models", with proper prompting allowing reasoning to "emerge naturally." Recent work seems to maintain this optimism [7] . In this paper, we examine the effectiveness of CoT in the context of classical planning problems, which have well-defined and algorithmically checkable ground truths, can be generated with arbitrary size and difficulty, and are unlikely to be in the training data. If CoT induces more than just pattern matching, and can in fact teach LLMs to perform generalizable, compositional reasoning, then we should expect that to be reflected in robust and maintainable improvements on a simple commonsense benchmark set like Blocksworld, and we should expect these results to hold for scaled variants of the very benchmarks tested in [50] and later CoT work. 
background: 
 Classical planning problems task a planner with finding a sequence of actions that, when executed, will take an agent from a pre-specified initial state to a desired goal state. STRIPS planning is a discrete, deterministic formalism that encompasses this class. Problems are represented using the Planning Domain and Definition Language (PDDL) [30] and have long featured in various planning challenges and competitions. Our main experiments are all on the Blocksworld PDDL domain.A PDDL specification consists of three components. The domain doesn't change between problems and consists of a set of predicates-whose truth values describe the state of the world-and a set of actions-defined by their preconditions and effects-that the agent is allowed to take. The initial state is a list of predicates that are true at the outset of the specific problem (an example predicate: "Block A is on the table "). The goal is a boolean expression of predicates (a goal: "Block A is on Block B.").A plan is a sequence of actions. The solution to a PDDL problem is a plan in which the preconditions of every action are satisfied at execution time, and which arrives at a goal-satisfying final state. To verify a plan, follow the actions in order and check that these two desiderata are achieved. In this work, we convert natural language responses into PDDL [46] and evaluate them with VAL [21] . We examine the influence of prompt selection on LLM performance within subsets of the Blocksworld domain. 
chain of thought setups for planning: 
 A formally specified problem instance can be translated into many possible prompts. The most basic of these is input/output (I/O) prompting: the problem is translated directly from PDDL into natural language and provided to the LLM [47] . While this directly tests the LLM's ability to solve the problem, it is not always the most effective strategy for maximizing performance.Drawing on metaphors of human learning, recent literature has claimed that LLMs are capable of in-context learning. The basic idea is that-by first presenting the model with examples of similar problems-it is possible to cause an LLM to acquire relevant new skills within the current context window. n-shot prompts operationalize this by prepending a number of relevant examples. Chain of thought [50] approaches take this further, presenting human-crafted "thoughts" which the LLM is intended to imitate in its response. Practitioners argue that, intuitively, these augmented examples teach the LLM how to solve problems in the given set.However, this method relies on human labor [53] to provide task-specific knowledge and an (at least rough) algorithmic or procedural approach to the problem. The more general the provided knowledge is, the more problems it can be applied to, and the less human prompt-crafting it requires. On the other hand, the more granular and specific it is, the more performance can be expected.In our experiments, we consider subsets of Blocksworld problems. We follow a prompt structure similar to that described in [47] ,2 but include "thoughts" in our n-shot prompts. These thoughts are written to follow an algorithmic procedure for solving the example problem.Not every procedure is applicable to every problem. From the point of view of a human handcrafting a chain of thought prompt, there is intuitively an expected target distribution on which the demonstrated algorithm generally works. For instance, a prompt designer detailing how to stack C on top of B on top of A will expect that a model that learns this procedure will also be capable of stacking B on top of A on top of C, but may not expect it to know how to first properly dismantle an existing tower of blocks to access a necessary block. However, this distribution often differs from the effective target distribution-that is, the actual set of problems on which the prompt gives robust improvements in performance. We explicitly describe the gap between these two distributions.Zero-Shot Chain of Thought (Universal): This is the most general approach, and involves merely appending "let's think step by step" to the end of the prompt [26] .Progression Proof (Specific to PDDL): Versions of this CoT could, in principle, be prepended to any PDDL problem prompt, as the generation of annotated examples is easy to automate without knowledge of the specific PDDL domain. [47] This prompt includes (1) a meta-prompt explaining plan correctness and (2) an example where each action is annotated with the state prior to the action, the reason why the action is applicable in that state, and the resulting state after the action is applied. Examples start from an arbitrary block configuration and construct a single stack of blocks from it.Blocksworld Universal Algorithm (Specific to the Domain): In Blocksworld, it is possible to reach any goal state from any initial state by simply unstacking all the blocks, placing them on the table, and then reassembling them into the required stacks. Resulting plans are not only executable and goal-reaching, but will never exceed twice the length of the optimal plan for any given instance [40] . This prompt demonstrates an annotated version of this approach, explaining and performing both the deconstruction and reconstruction steps of the algorithm. The same examples are used as in the previous prompt. The expected target distribution encompasses all Blocksworld problems.Stacking Prompt (Specific to a Narrow Problem Class): Every example is a table-to-stack problem: every block starts on the table, and the goal is to create a single specific stack of blocks. This specificity simplifies the problem greatly, and allows near-direct pattern matching between the examples and the LLM's output; however, it is infeasible to specify prompts with this level of detail for every problem class. The expected target distribution is table-to-stack Blocksworld problem, as they are the only problems that can be solved by the described algorithm.Lexicographic Stacking (Specific to Particular Syntactic Sequences): We simplify the problem further by focusing on a particular syntactic form of the goal. This prompt is very similar to the stacking prompt, but is specific to a subset of the target distribution: the goal state is always a lexicographic prefix (e.g., A, AB, ABC, etc.). 
blocksworld results: 
 We perform two parallel studies. The first tests each chain of thought prompt on its intended problem distribution, as explained in the previous section. Then, we focus on a specific subclass of Blocksworld problems and test every prompt on just that subclass. Together, we expect these two studies to give us a good picture of how effective LLMs are in applying advice beyond the specific instances.We evaluate the performance of GPT-4 and Claude-3-Opus on Blocksworld problems with both standard 2-shot prompts and chain of thought prompts of varying granularity. Each prompt is tested on its intended problem class, as discussed in the previous section. fix these paragraphs send lexico to a subsection after rest of experiments Chain of thought does not meaningfully enhance performance except on the narrowest problem distributions. While providing this chain of thought advice becomes significantly harder as the level of specificity increases, it is necessary, as the LLM succeeds only when the problem is reduced to a level where basic pattern matching suffices: at each stage, stack the next letter on top; if that letter does not exist on the table, then stop.A key advantage of planning domains is that they provide the ability to easily and systematically generate larger test sets, including arbitrarily more challenging instances. The difficulty of a Blocksworld instance scales with the number of blocks involved, allowing us to clearly assess the out-of-domain generalization achievable with and without chain of thought. As shown in Figure 2 , chain of thought does not generalize beyond a handful of blocks. Note that sound planning systems (such as Fast Downward) have a 100% accuracy on all problems tested. As mentioned before, a table-to-stack problem is any problem in the intended target distribution of the stacking prompt. The initial state has every block on the table, with a goal of arranging all the blocks into a single, pre-specified stack. While a simple problem, GPT-4's zero-shot performance over 261 instances is 3.8%. With the stacking CoT prompt, performance improves to 59.3%. Is this a result of the model learning in-context how to reason correctly over this type of problem? If so, we might expect it to perform the same when presented with a more general CoT prompt that demonstrates the same procedure, but is applicable to a greater variety of problems.To check this, we evaluate performance of our prompts on table-to-stack problems with prompts of varying granularity: standard I/O prompting, general n-shot (drawn from arbitrary Blocksworld problems), goal-specific n-shot (drawn from table-to-stack problems), and three levels of CoT specificity. Table 1 shows the results: only the most specific and least applicable prompt retains anywhere near this performance improvement. Figure A.1.1 in the appendix further illustrates that none of the prompts provide robust stack-height generalizability. We also tested self-consistency [49] on these prompts, but found that performance dropped. Details can be found in Appendix A. our results only show a modest improvement in performance on some domains, with specific enough prompting strategies, which quickly deteriorates when the problems shown become slightly larger. 
extension to scalable synthetic benchmarks: 
 Previous work on CoT mainly constrained its evaluations to static test sets ranging from commonsense domains (Sports Understanding [41] , StrategyQA [18] , CommonSenseQA [44] ), few-hop math word problems (AsDiv [31] , GSM8k [10] , MAWPS [27] ), to a number of basic "symbolic reasoning" tasks (CoinFlip [26] , LastLetterConcatenation [26] , Shuffled Objects [41] ). [26, 50, 55, 6] . Many of these benchmarks are difficult to scale, but a number of them can be modified to allow for the generation of arbitrary new instances which nevertheless have clear ground truths. We examine CoinFlip, LastLetterConcatenation, and a synthetic proxy for multi-step arithmetical reasoning. Exact prompt details can be found in the appendices A.7, A.8, and A.9. When possible we used the manual CoT prompts found in [50] and the zero-shot CoT prompt described in [26] . Number of examples ranges from 0 to 3 for both CoT and direct prompts. Results for all three domains are in Table 2 and Figure 3 .CoinFlip: Parity tests have a long history in machine learning [32] . CoinFlip is a natural language version of this task introduced in [50] to showcase the performance of CoT, though that paper only studies up to four flip problems. An example prompt is "A coin is heads up. Craig flips the coin. Alice does not flip the coin. Is the coin still heads up?". The correct answer is "no". Note that chance performance on this domain is 50%, as there are only two possible answers. Our extension to the domain is detailed in A.3LastLetterConcatenation: Also introduced in [50] , the LastLetterConcatenation task is a simple text processing task that asks for the concatenation of the last letters of a series of words. An example prompt is "Take the last letters of each word in 'Craig Alice' and concatenate them." for which the correct answer is "ge". The set of possible answers on this task is much larger than in CoinFlip, but previous work has claimed significant performance increases on this kind of task with CoT. Modeling something similar to our Blocksworld granularity experiments, we create two other test sets, using the same underlying words in the same distribution, but which differ in what they ask the model to do. LastVowelConcatenation requires using only the last vowels of words. FoomLetterConcatenation requires using the first letter of the first word, the second letter of the second word, and so forth. If the nth word does not have an nth letter, the problem specifies that a 0 should be concatenated to the string instead.Multi-step Arithmetic on Single-Digit Numbers: CoT is often tested on math word problems. However, many of these test sets only include problems which require very small numbers of reasoning steps. GSM8k was designed partly so that its problems would "require more steps to solve", but its problems only range 2 to 8 steps [10] , and, in fact, previous analyses have found that only 10% of those problems require more than five steps-the majority is 2, 3, or 4. [16] To sidestep this issue, we construct a synthetic dataset that involves linearly simplifying parenthesized expressions that consist of repeated applications of the four basic arithmetical operations on one digit numbers. An example prompt is "Simplify the following expression into a single number: 3 where the correct answer is 1. We filter our problems so that no operation ever results in a number that isn't in the range 1 to 9. 3 This can be seen as a deeply simplified variant of the arithmetical expression simplification dataset presented in [34] where no modular arithmetic, negative numbers, or non-linear nesting is required. However, we extend our maximum number of required reasoning steps much further and we construct prompts which are more specific and spell out every single step explicitly. More details on the dataset can be found in A.5.Length Generalization The only synthetic domain that shows any hints of generalization is CoinFlip. Using [50] 's prompt, performance is perfect for 1 through 4 step problems, starts to show the occasional mistake after, and only dips below 90% at 31-step problems (as shown in Figure 3 ). However, the problems in this domain are very simple. Parallel to the lexicographic stacking case of Blocksworld, it does not require much reasoning beyond counting up to an average of half a given problem's step count.LastLetterConcatenation and multi-step arithmetic show behavior almost identical to our main experiments. While sufficiently specific CoT prompts do increase performance on small instances, this performance increase quickly degrades as the number of steps necessary increases. Notably, the string-based nature of the LastLetterConcatenation problem does allow us to examine what exact improvement CoT is inducing. We examine the data with different metrics and find that the only properties that do generalize with CoT are syntactic. In particular, while overall accuracy plummets back to that of direct prompting, CoT consistently improves the Levenshtein distance to the correct answer and ensures that the final response string contains exactly the right letters, just not in the right order or number. We take this as further evidence that CoT, rather than teaching algorithms or procedures, modifies the syntactic style of the LLM's output, and that this pattern matching is what leads to observed increases in performance on smaller instances.Prompt Granularity and Problem Variation Because of the simplicity of these problems, prompt granularity is much harder to examine than in Blocksworld. There isn't enough variation in possible problems. However, across the three types of letter concatenation and two types of arithmetic expression simplification that we test, we see very similar patterns as before: CoT's performance improvements are maintained much longer in easier cases, and take longer to collapse back to direct performance. There still seems to be a "sweet spot" where the problem is just barely hard enough that CoT makes a difference, but not so hard that this difference doesn't matter.Examining Intermediate Reasoning The construction of our synthetic arithmetic task gives some hints as to what part of CoT may be failing. [14] argues that compositional reasoning fails because LLMs perform linearized subgraph matching and act as noisy estimators of intermediate functions (see e.g. proposition 4.2 in [14] ) and that performance collapses follow from the fact that repeated application of any error-prone function estimator leads to exponentially accumulating error.In our problem, it is possible to exhaustively check whether this is the case. There are exactly 118 possible 1-digit binary arithmetic problems which result in a 1-digit number. We tested GPT-4-Turbo, GPT-4, GPT-3.5-Turbo, Llama3-70b, and Llama3-8b on this dataset at various temperatures and every single model scored 100%. However, despite perfect performance on application of the required intermediate function, CoT still does not lead to robust generalization to arbitrary length problems. Therefore, at least on this problem set, the issue isn't due to accumulating error. The problem must be with the LLM's inability to learn the correct algorithm from contextual demonstrations, rather than with its inability to execute that algorithm.Overall, we see that our results on planning are not a fluke. These three synthetic domains showcase similar generalization failures, but these failures only become clear when the problems tested on require sufficiently many reasoning steps or when the minor modifications of the domain are studied. This illustrates the need for testing on benchmarks which can generate arbitrary new instances of increasing difficulty. Without such testing, conclusions drawn from static test sets of limited size are unlikely to be robust. We implore the community at large to adopt more rigorous evaluation mechanisms, especially when making claims about the poorly-understood yet much-hyped algorithmic reasoning abilities of black box models. 
conclusion: 
 In this paper, we conducted a systematic evaluation of the effectiveness of chain of thought in large language models on a specific classical planning problem. Our case study indicates that, contrary to previous claims in the literature, providing examples of procedural reasoning does not induce the general ability to apply that procedure to novel instances in current state-of-the-art large language models. In fact, the performance improvements seen when prompting LLMs in this manner quickly vanish when queries differ in generality from the examples, despite the fact that the same algorithmic procedure applies to the larger or more general instance. Chain of Thought (CoT) has become one of the most widely adopted ideas for improving planning and reasoning abilities of LLMs. Almost every system routinely, and uncritically, uses some prompting strategy attributed to CoT. On the flip side, whenever LLMs are shown to have limitations in any sphere, practitioners tend to question those studies by attributing it to unskilled use of CoT methodology. Our study, based on both in planning and other more standard tasks, calls into question the prevalent belief that LLMs are capable of operationalizing and generalizing the CoT advice effectively. It instead suggests that CoT is effective only when the LLM can do straightforward pattern matching between the example and the problem. We believe that the lessons of this study will be helpful in mitigating the applications of LLMs to tasks requring planning and reasoning with false confidence. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: NA Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: NA Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We evaluated self consistency [49] , a state-of-the-art extension of CoT, on table to stack problems. We sampled 5 different reasoning paths (with temperature 0.7) and chose the most frequent plan breaking ties randomly. As our results show (in Table 3 and Figure A.2.1), self-consistency does not lead to a generalization breakthrough, and in fact is generally worse than the original results in Table 1 . This is likely because the solution space for planning problems is much larger than that studied in previous (often multiple choice) benchmarks. In fact, most queries led to five unique responses, forcing us to choose the final answer from them at random.Given a list of names, generating new instances is just a matter of filling in a template. We source our list of names from the U.S. Social Security Administration [1] , and only keep names with at least 50 occurrences. We scale our problems with the number of names (potentially repeated) mentioned in the prompt. The main test set consists of 1120 instances, with 40 instances per number of names and 28 different numbers of names, ranging from 1 to 28. We also tested [50] 's prompt on an extended We use the same database as in CoinFlip to generate words. [1] To scale instances, we simply increase the number of words whose last letters must be concatenated. Our problems range from 1 to 20 words, with 40 instances per word, giving a total of 800 problems.The number of reasoning steps in this domain corresponds directly to the number of operations that need to be performed to simplify a given expression. Our test set consists of 1160 total problems, spread 1 to 29 operations, with 40 instances per number of operations. Again mirroring The plan correctness is defined in terms of states resulting from executing the actions in the plan. An action is executable in a state when all its preconditions hold in that state. The state resulting from the action execution consists of everything in the previous state with the addition and deletion of add and delete effects of the action. Plan correctness is defined as follows: if the first action in the plan is applicable in the initial state, i.e., its preconditions are all present there; and the second action is applicable in the state resulting from applying the first action to the initial state, this process continues until the state resulting from the application of the last action in the last but one state gives rise to the final state where all the goals are satisfied.→ → → → → → → →Here is an example problem: [STATEMENT] As All the blocks are now clear and on the table.2. I look at the goal and stack each tower I need, starting from the bottom. I check each of the mentioned blocks in order to determine which of them are the bases of towers.Block A needs to be on top of Block B, so it can't be the base of a tower. Block B needs Block A on top of it. I double check the goal: this is the only condition that must hold for block B. It does not need to be on top of any other block. Therefore, I can use it as the base of a tower.Block C needs to be on top of Block A, so it can't be the base of a tower.In summary, only Block B can be used as the base of a tower.I build the tower that has Block B as a base. I don't need to move Block B, and I leave it be, as I know it is already placed correctly. My plan is as follows:[THOUGHTS] 1. I look at the goal and stack each tower I need, starting from the bottom. My goal only mentions blocks B and A. I check each of the mentioned blocks in order to determine which of them are the bases of towers.Block A needs to have Block B on top of it. I double check the goal: this is the only condition that must hold for block A. It does not need to be on top of any other block. Therefore, I can use it as the base of a tower.Block B needs to be on top of Block A, so it can't be the base of a tower.In summary, only Block A can be used as a base of a tower.I build the tower that has Block A as a base. I don't need to move Block A, and I leave it be, as I know it is already placed correctly.The block that goes on top of Block A is Block B. I pick up Block B. I stack it on top of Block A. This satisfies the part of the goal that says that Block B is on top of Block A.No block needs to be on top of Block B. Therefore this tower is complete.There are no other blocks which can be used as a base of a tower, and all the blocks mentioned in the goal have been placed, so I am done constructing. My plan is as follows:[THOUGHTS] 1. I look at the goal and stack each tower I need, starting from the bottom. My goal only mentions blocks A, B, and C. I am free to ignore all other blocks. I check each of the mentioned blocks in order to determine which of them are the bases of towers.Block C needs Block A on top of it. I double check the goal: this is the only condition that must hold for block C. It does not need to be on top of any other block. Therefore, I can use it as the base of a tower.Block A needs Block B on top of it. I double check the goal: this isn't the only condition that must hold for Block A, because Block A needs to be on top of Block C. Therefore Block A can't be the base of a tower.Block B needs to be on top of Block A, so it can't be the base of a tower.In summary, only Block C can be used as the base of a tower.I build the tower that has Block C as a base. I don't need to move Block C, and I leave it be, as I know it is already placed correctly. Block A needs Block C on top of it. I double check the goal: this isn't the only condition that must hold for Block A, because Block A needs to be on top of Block B. Therefore Block A can't be the base of a tower.Block C needs to be on top of Block A, so it can't be the base of a tower.In summary, only Block B can be used as the base of a tower.I build the tower that has Block B as a base. I don't need to move Block B, and I leave it be, as I know it is already placed correctly. Example 0:[QUESTION] A coin is heads up. Craig flips the coin. Is the coin still heads up?[Thoughts] The coin was flipped by Craig. So the coin was flipped 1 times. The coin started heads up, so after an odd number of flips, it will be tails up. So the answer is no.Example 1:[QUESTION] A coin is heads up. Major flips the coin. Maria does not flip the coin. Is the coin still heads up?→[Thoughts] The coin was flipped by Major. So the coin was flipped 1 times. The coin started heads up, so after an odd number of flips, it will be tails up. So the answer is no. The coin was flipped by Wolf and Page. So the coin was flipped 2 times. The coin started heads up, so after an even number of flips, it will still be heads up. So the answer is yes.Problem to solve:[QUESTION] A coin is heads up. Craig flips the coin. Is the coin still heads up?[Thoughts] ==================GPT-4-Turbo Response==================== The coin was flipped by Craig. So the coin was flipped 1 time. The coin started heads up, so after an odd number of flips, it will be tails up. So the answer is no. Problem to solve:[Question] Take the last letters of each word in "Analy Daniel" and concatenate them.[Thoughts] ==================GPT-4-Turbo Response==================== [Answer] yyCoT LastLetterConcatenation Prompt and GPT-4 ResponseAfter the [Answer] tag, you may only respond with a lowercase string of concatenated characters. Do not include anything else after that tag. The [Answer] tag must precede the final answer.The following 3 examples are provided. Please follow the formatting used in them.Example 0:[Question] Take the last letters of each word in "Anthony" and concatenate them.[Thoughts] The last letter of "Anthony" is y. Concatenating them is "y". The answer is y.[Answer] y Example 1:[Question] Take the last letters of each word in "Beautiful Ash" and concatenate them.[Thoughts] The last letter of "Beautiful" is l. The last letter of "Ash" is h. Concatenating them is "lh".The answer is lh.Example 2:[Question] Take the last letters of each word in "Matthew River Craig" and concatenate them.[Thoughts] The last letter of "Matthew" is w. The last letter of "River" is r. The last letter of "Craig" is g. Concatenating them is "wrg". The answer is wrg.Problem to solve:[Question] Take the last letters of each word in "Analy Daniel" and concatenate them.[Thoughts] ==================GPT-4-Turbo Response==================== The last letter of "Analy" is y. The last letter of "Daniel" is l. Concatenating them is "yl". The answer is yl.LastVowelConcatenation Prompt and GPT-4 Response[Instructions] For the purposes of these problems, a vowel is any one of the letters "a","e","i","o", or "u", but NOT "y". After the [Answer] tag, you may only respond with a lowercase string of concatenated characters. Do not include anything else after that tag. The [Answer] tag must precede the final answer.The following 3 examples are provided. Please follow the formatting used in them.Example 0:[Question] Take the last vowel of each word in "Anthony" and concatenate them.[Answer] o Example 1:[Question] Take the last vowel of each word in "Beautiful Ash" and concatenate them.[Answer] ua Example 2:[Question] Take the last vowel of each word in "Matthew River Craig" and concatenate them.[Answer] eei Problem to solve:[Question] Take the last vowel of each word in "Analy Daniel" and concatenate them.[Answer]==================GPT-4-Turbo Response==================== ya FoomLetterConcatenation Prompt and GPT-4 Response[Instructions] After the [Answer] tag, you may only respond with a lowercase string of concatenated characters. Do not include anything else after that tag. The [Answer] tag must precede the final answer.The following 2 examples are provided. Please follow the formatting used in them.Example 0:[Question] Take the following words and produce a new string using them: "Anthony". The first letter of your string should be the first letter of the first word, the second letter of your string should be the second letter of the second word, and so forth. If you need the nth letter of the nth word, but that word is less than n letters long, then insert a 0 character instead.[Thoughts] The 1st letter of "Anthony" is A. Concatenating them is "a". The answer is a.[Answer] a Example 1:[Question] Take the following words and produce a new string using them: "Beautiful Ash". The first letter of your string should be the first letter of the first word, the second letter of your string should be the second letter of the second word, and so forth. If you need the nth letter of the nth word, but that word is less than n letters long, then insert a 0 character instead.[Thoughts] The 1st letter of "Beautiful" is B. The 2nd letter of "Ash" is s. Concatenating them is "bs". The answer is bs.Problem to solve:[Question] Take the following words and produce a new string using them: "Analy Daniel". The first letter of your string should be the first letter of the first word, the second letter of your string should be the second letter of the second word, and so forth. If you need the nth letter of the nth word, but that word is less than n letters long, then insert a 0 character instead.[Thoughts]==================GPT-4-Turbo Response==================== The 1st letter of "Analy" is A. The 2nd letter of "Daniel" is a. The 3rd letter of "Analy" is a. The 4th letter of "Daniel" is i. The 5th letter of "Analy" is y. The 6th letter of "Daniel" is e. Concatenating them is "Aaaiye". The answer is aaaiye.→ →NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: NA Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Prompt and response examples for each approach can be found in the Appendix.We exclude 0, since any number multiplied by zero is zero, and this would quickly lead to zero representing around 50% of correct answers for larger numbers of reasoning steps. 


Decision: The paper studies whether Chain-of-Thoughts (Cot) enables generalization to larger problems involving a number of reasoning steps larger than most datasets used for reasoning.

There is a large disagreement among the reviewers’ ratings, from 4 to 8. The strongest critique was about the conclusions. Some reviewers remarked that CoT was actually helping in comparison with the direct approach. The rebuttal clarified the goal of the submission: to challenge that CoT unlocks reasoning abilities. This is done by testing on domains requiring more steps. The cases where the improvement was more significant were in more straightforward tasks like lexicographic stacking.

In my opinion, the authors clarified the scope of their contribution and showed what changes would be made so it’s more clear in a future version.

Reviews: review: paper_summary: paper_summary: The paper aims to show that Chain of Thought style prompting does not result in generalisation of reasoning, instead relying on pattern matching to improve performance. It argues that if CoT results in language models learning algorithms in context, then prompts describing general procedures should result in similar performance gains to prompts describing task specific examples. Experiments are performed on planning problems in the Blocksworld domain using prompts at different levels of generality. The results show smaller performance gains from general prompts compared to large gains for more specific prompts. Through further experiments on planning problems in other domains, the paper shows that CoT does not generalise to more complex problems than those presented as examples. The paper concludes that improvements from CoT are due to pattern matching rather than in-context learning of algorithms.
strength_weakness: strength_weakness: While the results presented do indicate that more general demonstrations of CoT do not perform as well as task specific ones, they also do show improvement over standard prompting. For example in Table 1, GPT4 jumps from 7% to 28% using the Blockworld Universal Algorithm. A similar trend is present in Table 2 and Table 3. It is unclear to what extent this is from pattern matching or from algorithmic generalisation. The paper may be at risk of overstating their claim that CoT is not inducing any reasoning. Discussion and further analysis of this would be helpful.

A concern about the "Progression Proof CoT" prompt: The prompt includes demonstrations of successful plans with details of the actions and states. However, it does not include any reasoning / algorithmic description of how the plan was obtained from the problem. To be considered a legitimate Chain of "Thought" prompt, should not this prompt include description of a general procedure to derive the plans? Otherwise, it is difficult to see of the LLM can be expected to generalise reasoning from a description of the final answer without intermediate reasoning "thoughts".

Overall, a novel and interesting contribution but there are some concerns about evaluation and the conclusions drawn from them.
questions: questions: 1. In 5.2, the paper says "only the most specific and least applicable prompts retain anywhere near this performance improvement". Is this referring to the "Blocksworld Universal Algorithm" or the "Stacking Prompt"? In the case of the former, is this also not a fairly general prompt? More broadly my concern is that Table 2 shows that the Universal Algorithm Prompt achieves similar performance to the Stacking prompt, showing that a more general CoT can perform as well as a more specific one, which is contrary to the central claim of the paper. 

2. The performance of GPT4 on Lexicographic Stacking seems surprisingly low with the n-shot prompt (considering the CoT prompt achieves 94%). Was this due to the particular examples provided in the n-shot prompt?

3. Table 3 does not seem to be referenced in the text?

4. Does 5.1 contain an incorrect reference to Figure 3?

5. Does 5.2 contain an incorrect reference to Table 3?

6. Can the n-shot prompts be included in the appendix?
limitations: limitations: I could not find any discussion of the limitations of the work. One limitation is that the work does not provide methodologies for reasoning with LLMs beyond that go beyond the limitations described.
paper_summary: paper_summary: The paper conducts a systematic study of claims that chain-of-thought (CoT) prompting unlocks reasoning abilities in LLMs. In particular, the paper evaluates the ability of LLMs to a) learn a simple algorithm from demonstrations annotated with reasoning steps ("thoughts") provided as part of the input prompt, and b) to generalize to harder instances in the same problem class. CoT prompt variants are evaluated on a carefully constructed set of simple planning problems (e.g., Blocksworld) and simplified variants. The experimental evaluation demonstrates that CoT works better when the prompt includes reasoning demonstrations on examples "similar" to the query (test) problem and when the test problem class is sufficiently easy (small, specific). The paper makes the case that CoT enables a form of syntactic pattern matching rather than algorithmic reasoning. Overall, the paper provides deep empirical insight into CoT prompting, clearly demonstrating its limited ability to generalize to larger tasks requiring multi-step reasoning when given smaller sized demonstrations, a task easily handled by sound planning algorithms.
strength_weakness: strength_weakness: - I didn't spot any major weaknesses in this paper. Some minor nitpicks are in the questions.
questions: questions: - This may be out of scope but I'd be curious to understand how much variance there is in these results wrt to the prompt inputs. Specifically, how robust are the results wrt (semantically) small changes to the system prompt, examples, thoughts, formatting, etc.?

- (Line 200) I'm probably missing something but where exactly is the meta prompt explaining plan correctness in A.6.2?

- Some references seem to be incorrect or missing.
  - (Line 239) Should it be "Figure 2" (instead of "Figure 3")?
  - (Line 253) Should it be "Table 2" (instead of "Table 3")?
  - Should Figure 3 and Table 3 be referenced somewhere in Sec 6.1?
limitations: limitations: Yes
paper_summary: paper_summary: This paper evaluates the efficacy of Chain of Thought (CoT) prompting in improving the reasoning capabilities of large language models (LLMs) in planning tasks. The authors analyze CoT's performance in the Blocksworld domain, a classical planning problem, and extend their findings to other synthetic tasks. They demonstrate that CoT prompts only show meaningful improvements when the provided examples are highly specific to the problem class, and that these improvements quickly deteriorate as the complexity of the problems increases. The paper challenges the notion that CoT enables LLMs to learn general algorithmic procedures, suggesting instead that performance gains are largely due to pattern matching rather than genuine algorithmic understanding.
strength_weakness: strength_weakness: 1. **Weak Motivation**: The Chain-of-Thought paper [1] claims that "chain-of-thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting." However, the findings from this paper, specifically in lines 84 to 88, reiterate the same point, stating that CoT prompts act as exemplars and involve "pattern matching."
2. **Unrelated Title**: The paper primarily evaluates CoT prompts in a single planning task, Blocksworld, and the results indicate that CoT is still useful, as shown in Table 1. However, the title "Chain of Thoughtlessness" does not accurately reflect the content or findings.
3. **Lack of Smooth Transitions**: The content in the introduction and related work sections lacks order, making the paper structure disjointed and difficult for readers to follow. The transitions between paragraphs are abrupt and lack coherence. For example, using subheadings in the related work section could help group related content into subsections, creating a more logical flow.
4. **Section 3 Placement**: The background information in Section 3 could be integrated into a subsection of the related work, providing a smoother introduction to the planning tasks.
5. **Unclear Conclusions**: The conclusions in lines 231 to 235 are not well-supported by the results. Table 1 shows that Zero-shot CoT, Blocksworld Universal Algorithm CoT, Stacking CoT Prompt, and Lexicographic CoT Stacking Prompt all improve performance, with some showing significant improvements. However, the paper claims that the CoT approach does not enhance performance, which contradicts my understanding. Perhaps my interpretation of this table is incorrect.
6. **Lack of Examples in Main Text**: Lines 196 to 220 should include specific examples directly in the main text, as they are central to the experiments. Currently, these examples are relegated to the appendix, which diminishes their impact and increasing the understanding of remaining sections.
7. **Insufficient Experimental Explanation**: The experimental section lacks clarity and organization. Each experiment should be clearly described, explaining what each aims to demonstrate, such as Experiment A for generality and Experiment B for complexity.
8. **Missing Statistics about Test Set**: There are no statistics provided about the Blocksworld test set, leaving the question distribution unclear. This is essential evidence for evaluating the generality of CoT prompts.
9. **Unaddressed Human Labor Cost**: The paper frequently mentions the drawback of CoT prompts requiring human labor but does not provide any comparative analysis or solutions. This point does not directly support the paper's argument and feels extraneous.
10. **Writing Typos**:
    - Line 72: PDDL is introduced as an acronym without explanation.
    - Line 239 likely refers to Figure 2, which should be corrected.

> Reference:  
> [1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.
questions: questions: N/A
limitations: limitations: N/A
paper_summary: paper_summary: This paper evaluates the effectiveness of CoT prompting on reasoning problems within the Blocksworld domain, revealing that performance gains from CoT are limited and heavily reliant on problem-specific prompts, with diminishing returns as problem complexity increases.
strength_weakness: strength_weakness: 1. The paper ignores all experimental details except prompt. The lack of description of experimental details, including the temperature and other parameters used in the data set, affects the reproducibility of the paper.
2. The paper is not very clear. The core point of the paper is that the model CoT is very sensitive to prompts and demonstrations. However, the paper spends a lot of time exaggerating the failure of CoT-style demonstration.
questions: questions: 1. I seriously doubt the test of Table 1. Why does Lexicographic Stacking Prompt only have 17 samples (*/17), while other prompts have 270 samples? This comparison is completely unfair. The model may only perform well on these 17 examples.
2. I think there is an obvious overclaim in this paper. This paper only shows that the current CoT is very sensitive in this kind of Action-related tasks. In addition, the effectiveness of CoT plans is often verified on natural language mathematical tasks. Because the expression of mathematics + natural language is more free, and theplanning logic is more general, demonstration can be used to stimulate better output of CoT style, including various types such as ICL CoT, Zero-shot CoT, Plan-and-Solve, Complex- CoT and a series of methods.
3. In fact, as shown in Figure 1, the absolute improvement of CoT has become smaller. The essential reason is that the model itself cannot solve such a complex problem. It is not that the CoT strategy is not good. In fact, compared with direct output, the relative increase in performance is actually Improved (CoT Acc/Direct Acc)
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: Re. author response
comment: Thanks to the authors for their response.

After considering the clarifications in the rebuttal, I better understand the paper's claim that CoT does not robustly generalize underlying algorithms from prompts. I believe this has been demonstrated through sufficient emperical analysis and is a valuable contribution. I have decided to increase my score given these considerations.
I thank the authors for their detailed response to all reviewers. After reading the other reviews and the comments, I'm now more positive about the paper. In my opinion, the authors have been clear about their central claims both in the paper and comments ("we are only claiming that CoT doesn’t robustly learn the algorithmic procedure demonstrated in prompt, as shown by the performance deteriorating as the number of reasoning steps increases"), and proceed to demonstrate its empirical validity rigorously. I think this paper adds valuable empirical insight into the limitations of current LLMs, which is sufficient for me to recommend acceptance. Beyond that, I think this paper would be a valuable addition to the growing body of work in LLM-based planning.
Thanks you for your detailed response. I have no additional questions.
Thank you for your reply.

We reiterate that our global response as well as the response to you clearly points out that we are only claiming that CoT doesn’t robustly learn the algorithmic procedure demonstrated in prompt, as shown by the performance deteriorating as the number of reasoning steps increases (despite the fact that the given procedure does solve the problem). This is not contradicted by raw improvements on problems similar to the given CoT exemplars. As we pointed out in our individual response to you, your original review seems to have misread this.

We would also like to draw your attention again to the fact that in addition to planning tasks, we have done experiments on extensions of standard CoT benchmarks–including last letter concatenation (Section 6)--and those results too are in agreement with our claim about Cot not leading to general procedure learning. 

Finally, we note–as you yourself can readily see–that all the other reviewers have had significantly more positive assessment of the paper. 

We look forward to your reconsideration of your assessment.
Thank you for your reply.

1. Perhaps we are misunderstanding, but this response seems contradictory: robustly learning and applying a correct algorithmic procedure and learning a “step-by-step thought” are the same thing. Our claim is that, while the improvements seen do exist, they are brittle in ways that robustly learning the requisite step-by-step process wouldn’t be–raw improvement does not contradict this. The evidence necessary for our claim is the rapid deterioration of this improvement on instances that require more reasoning steps, a phenomenon we demonstrate across multiple diverse multi-step reasoning domains, from variants of Blocksworld to last letter concatenation and arithmetic expression simplification. In the camera ready version, we will make much clearer both this distinction and how we are using the evidence we gather to show the claim we make.

2. Outside of zero-shot methods (“let’s think step by step”), manual CoT construction is always domain-specific, and requires demonstrating an in-context procedure for each exemplar. We do not claim anything about CoT’s sensitivity to particular prompts. Consider the Last Letter Concatenation domain: performance is perfect for smaller instances, yet falls quickly on larger ones, despite the prompt and procedure being the same in all cases. Our contribution is to provide critical evidence that the intuitions underlying previous work–that CoT demonstrations of procedures allow LLMs to learn those procedures and apply them in-context, an anthropomorphization arising from the response format engendered by the technique and the popular name for it–miss the mark. We constrain our claims and evaluations to CoTs that are designed for and applicable to the problems they’re presented with.

3. In its current form, our paper does explicitly discuss GSM8k, CommonsenseQA, MAWPS, AsDiv, and others. We go further in depth on arithmetical reasoning benchmarks and construct a simplified natural language synthetic benchmark in section 6, under the heading “Multi-step Arithmetic on Single Digit Numbers”. In particular, we discuss a fundamental limitation of these benchmarks: they generally require only a couple of reasoning steps. Only ten percent of all GSM8k problems (a benchmark which explicitly attempted to increase the number of reasoning steps necessary) require more than five steps, and none are over eight. This sort of narrow problem distribution is insufficient to properly evaluate whether the model has learned and applied the correct procedure or is merely pattern matching to syntactically similar templates.

Note also the quote we brought up in our rebuttal from the Complex CoT paper, which we will include together with a deeper discussion in our camera ready version: “We observe a clear trend on both GSM8K and MathQA: complex prompts perform on par with simple prompts on hard cases, while achieving more clear gains on cases with fewer number of reasoning steps.” Robust generalization is still not seen even with additional prompt engineering or manipulation. 

Furthermore, as mentioned in our paper, CommonsenseQA and similar domains do not explicitly require multi-step reasoning. In fact, the CoT exemplars given in Wei’s 2022 Chain of Thought paper for CSQA are all exactly two steps. Because CSQA questions require broad knowledge, and are very amenable to memorization, there is no way to scale the dataset to instances that necessarily require more steps–these are problems that mainly test what the model knows rather than whether the model can reason in a generalizable manner.
Thank you for your response. Your rebuttal indeed clarified some of my misunderstandings. I have decided to improve my overall score.

1. **It seems that over-claiming still exists.** As you mentioned, the CoT prompt does not make the model robustly learn the correct algorithmic procedures. However, in most of your tasks, CoT indeed shows an improvement compared to Direct, which precisely proves that LLMs can learn a step-by-step thought.
2. **Prompt sensitivity:** Additionally, I still believe that the contribution of this paper is limited to the fact that CoT is only sensitive to specific prompting or relies on a certain way of thought. In fact, to solve certain types of problems, demonstration or prompt strategy is inherently domain-specific. The clearer and more relevant the logical format for some problems, the faster and better the reasoning performance can be achieved. This kind of discussion seems unnecessary. For example, if I demonstrate or guide anyone through a Program-of-Thought logic, as humans, we cannot completely follow the Program-of-Thought logic to solve commonsense reasoning problems that cannot be strictly expressed.
3. **Insufficient consideration:** Based on the previous point, using a specific logic to solve specific tasks is inherently a shortcut for humans, while general logic is often inefficient for uncommon tasks with special detailed settings (including so-called blocksworld, letter concatenation, and coin flipping). I believe this is beyond doubt. Therefore, I still think it is insufficient that this paper does not discuss the benchmark, like GSM8K, MATH, CommensenseQA, which is a natural language problem requiring general logic.
Thank you for the authors' response.   
First, I apologize for the unstructured review. My main concern is that your paper seems to overclaim, and the experiments are not convincing, which has also been mentioned by other reviewers. I will wait for the responses from the other reviewers before deciding whether to raise the score.