Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse

## Abstract

Abstract Chain-of-thought (CoT) prompting has become a widely used strategy for working with large language and multimodal models. While CoT has been shown to improve performance across many tasks,
determining the settings in which it is effective remains an ongoing effort. In particular, it is still an open question in what settings CoT systematically reduces model performance.
In this paper, we seek to identify the characteristics of tasks where CoT reduces performance by drawing inspiration from cognitive psychology, looking at cases where (i) verbal thinking or deliberation hurts performance in humans, and (ii) the constraints governing human performance generalize to language models.
Three such cases are implicit statistical learning, visual recognition, and classifying with patterns containing exceptions. In extensive experiments across all three settings, we find that a diverse collection of state-of-the-art models exhibit significant drop-offs in performance (e.g., up to 36.3% absolute accuracy for OpenAI o1-preview compared to GPT-4o) when using inference-time reasoning compared to zero-shot counterparts. We also identify three tasks that satisfy condition (i) but not (ii), and find that while verbal thinking reduces human performance in these tasks, CoT retains or increases model performance. Overall, our results show that while there is not an exact parallel between the cognitive processes of models and those of humans, considering cases where thinking has negative consequences for human performance can help us identify settings where it negatively impacts models. By connecting the literature on human deliberation with evaluations of CoT, we offer a new tool that can be used in understanding the impact of prompt choices and inference-time reasoning.

## 1 Introduction

Figure: Figure 1: Tasks evaluated for reductions in performance from CoT prompting. Implicit Statistal Learning (ISL): Classification of strings generated by an artificial grammar. Face Recognition (FR): Recognition of a face from a set that shares similar descriptions. Classification of Data with Exceptions (CDE): Learning labels in the presence of exceptions. Natural Language Inference (NLI): Recognizing a logical inconsistency. Spatial intuitions (SI): Tilting water glasses. Working Memory (WM): Aggregating features for a decision. Humans show reductions in performance when engaging in verbal thinking in all tasks, we show that the first three have similar effects on LLMs and VLMs, while the last three differ between humans and models in meaningful ways.
Refer to caption: extracted/5987228/figures/main_fig.png

Chain-of-thought (Wei et al., ; Nye et al., ) is a widely used prompting technique for large language and multimodal models (LLMs and LMMs), instructing models to “think step-by-step” or providing other structure that should be incorporated into their response. Large meta-studies have shown that this technique improves the performance of models on many tasks, particularly those involving symbolic reasoning (Sprague et al., ). More generally, inference-time reasoning has become a default component of the newest LLMs and LMMs such as OpenAI o1-preview (OpenAI, ) and Claude’s web interface and mobile apps (Anthropic, ). However, there also exist cases where CoT decreases performance, but there have not been any identified patterns as to when this happens. With the increasing use of inference-time reasoning in deployed models, it is imperative to understand and predict when CoT has a negative effect on model performance.

A key challenge for determining the limits of CoT is the sheer variety of tasks for which LLMs and LMMs are used. While the machine learning community has dedicated great efforts towards developing a large set of benchmarks for these models (e.g., Hendrycks et al., ; Suzgun et al., ), applications of models extend beyond benchmarks to diverse contexts and variations of tasks that could all potentially affect performance. Exploring this enormous space to identify settings where CoT has negative effects is a daunting problem. This motivates the need to develop heuristics to help us identify risky cases that could pose real and salient challenges for model inference-time reasoning.

To narrow down the set of tasks to explore, we draw a parallel between CoT prompting and humans engaging in verbal thought (Lombrozo, ). Specifically, we explore the heuristic that tasks for which thinking or deliberation decrease human performance can also be ones for which CoT harms model performance.
An important caveat is that models and humans have fundamentally different capabilities and consequently have different constraints affecting their performance (Griffiths, ; Shiffrin & Mitchell, ; McCoy et al., ). For example, LLMs have long context lengths that far exceed human memory limitations. Despite this, we believe that for some failure cases the tasks themselves, in conjunction with traits shared between humans and models, are responsible for thinking having a negative effect on performance. Thus, we anticipate that CoT will hurt model performance in cases where (i) deliberation hurts performance in humans, and (ii) the constraints governing human performance on the task generalize to LLMs.

To explore this approach, we draw on the cognitive psychology literature on cases where engaging in verbal thinking hurts human task performance (Schooler & Engstler-Schooler, ; Dijksterhuis, ; Van den Bos & Poletiek, , inter alia). We chose six well-studied types of tasks from psychology that satisfy this, selected the most representative task of each type, and adapted them to properly evaluate LLMs and LMMs (see Figure 1). In particular, three of these task types satisfy (ii): those which involve implicit statistical learning, those where language is ill-suited for representing stimuli, and those that involve learning labels that contain exceptions to generalizable rules.

We identify another three types of tasks where (ii) is not satisfied, i.e., the constraints governing humans and models in these tasks differ. For these, we hypothesize that model performance will not necessarily reduce: tasks involving reasoning where models still significantly underperform humans without CoT, those where we expect that human and model priors differ substantially, and those that involve limits of working memory. For each of these six types, based on whether conditions (i) and (ii) hold, we develop testable hypotheses about whether CoT will have a negative effect on model performance and evaluate these hypotheses on a set of state-of-the-art models.

In representative tasks for each of the three categories where (i) and (ii) are satisfied, we find that CoT drastically decreases model performance across models. For implicit statistical learning, we observe an absolute performance drop of 36.3% in the performance of OpenAI o1-preview compared to GPT-4o zero-shot, as well as consistent reductions in accuracy across eight other state-of-the-art models. For tasks involving visual stimuli that are ill-represented by language, we find reductions in performance across all six vision-language models tested. And when learning data that contain exceptions to generalizable rules, CoT increased the number of iterations it took to learn the labels of data points by up to 331%.

In contrast, for the three categories of tasks for which (i) is satisfied but (ii) is not, we observed no negative effects caused by chain-of-thought.
A basic prerequisite for seeing a negative impact from CoT is that zero-shot prompting produces reasonable performance. Thus, a logical reasoning task where human judgments are worse after deliberation was not a candidate for a negative effect of CoT because zero-shot prompting resulted in models that were unable to score above chance. In this case, performance improved using chain-of-thought, matching existing findings showing an advantage on tasks involving logic and mathematical reasoning (Sprague et al., ). When models lacked access to relevant priors, such as in a task where motor simulation was responsible for improved performance (relative to verbal thinking) in humans, performance was roughly equal between conditions. On the other hand, having access to longer context windows than human working memory synergized with CoT to improve model performance in a preference task involving aggregating a lot of features described in text. whether psychological results translate to effects for model inference-time reasoning. Our findings confirm that CoT drastically harms LLM/LMM performance in cases where (i) overthinking hurts human performance and (ii) humans and models face similar constraints when performing the task.

The remainder of the paper is structured as follows: We cover related work surrounding CoT and intersections between LLM/LMMs and psychology in Section 2. We ground our work within the psychology literature and identify six categories of tasks where thinking reduces human performance in Section 3. In Section 4, we cover the implementations of each task, how we adapt them to test models, and their corresponding results. We then discuss the potential limitations and implications of our work in Section 5.

## 2 Related work

### 2.1 Inference-time reasoning

Chain-of-thought prompting aims to improve the performance of language-based models by encouraging them to generate an intervening string of tokens that increases the probability of producing the correct answer (Wei et al., ; Nye et al., ). This approach can result in significant performance improvements in language (Zhang et al., ) and vision (Zhang et al., ) tasks, hypothesized to be a consequence of exploiting local structure in language (Prystawski et al., ). However, a recent metastudy suggests that the gains from using CoT are primarily in mathematical and symbolic reasoning tasks, and that it is otherwise unclear when it improves performance (Sprague et al., ). In related settings such as planning, there is little benefit from CoT prompting (Kambhampati et al., ), and CoT can also increase harmful outputs (Shaikh et al., ). Despite these results, the default expectation seems to be that CoT improves performance. For example, a recent update to a language-understanding benchmark cited the fact that CoT results in an improvement on the new benchmark but decreased performance on the original benchmark as an indicator that the new benchmark is better (Wang et al., ). This expectation seems to have driven the tendency towards the default use of CoT in the latest LLMs and LMMs.

### 2.2 Psychological methods as a tool for studying LLMs and VLMs

Since the introduction of LLMs, there has been growing interest in understanding the connections between models and human minds (Hardy et al., ).
Human cognition is often studied using well-controlled tasks involving carefully curated datasets designed to test specific hypotheses. The availability of these datasets, and the fact that they often consist mainly of text and/or images, have led to these tasks from the psychology literature quickly becoming popular methods for evaluating and understanding LLMs and LMMs (e.g., Binz & Schulz, ; Coda-Forno et al., ).
For example, recent studies that leverage insights or datasets from psychology have evaluated the representational capacity of LLMs (Frank, ), explored how RLHF and CoT lead to different outcomes when trying to make models both helpful and honest (Liu et al., ),
compared human and machine representations via similarity judgments (Peterson et al., ; Marjieh et al., ; ; ), determined that LLMs over-estimate human rationality in decisions (Liu et al., ), identified incoherence in LLM probability judgments (Zhu & Griffiths, ), identified susceptibility to linguistic illusions in LLMs (Marjieh et al., ), uncovered LLMs’ underlying social biases (Bai et al., ), used storytelling to understand episodic memory in LLMs (Cornell et al., ), constructed prompts using psychological theories of metaphor (Prystawski et al., ), discovered cross-linguistic variability in LLM representations (Niedermann et al., ), and probed the roles of language and vision for in-context learning in VLMs (Chen et al., ).
Many of these studies start with a well-studied phenomenon in human cognition and then explore whether there is an isomorphic analog to it in LLMs or LMMs. Our work follows this established line of literature by associating the well-studied impact of deliberation on human performance to effects that occur in models using CoT.

## 3 Approach: When thinking reduces human performance

A large body of psychological research has investigated effects of verbal thinking (often explicit “deliberation”) on memory, learning, judgment, and decision-making. Very often these effects are positive. For example, people who spend more time deliberating are
more likely to respond correctly on questions that initially trigger an intuitive but incorrect response (Travers et al., ). However, there are also cases in which deliberation can impair performance, often involving a mismatch between the representations or types of processing induced by deliberation and those that best support task performance (Schooler, ).

A classic setting for such effects is in the domain of implicit statistical learning. For example, in studies of artificial grammar learning participants are presented with sequences of letters or phonemes that conform to some structure (such as a finite state grammar) and asked to recognize well-formed sequences. Studies often find that participants can differentiate well-formed sequences from those that are not well-formed, but cannot verbalize the basis for their judgments (Aslin & Newport, ; Romberg & Saffran, ). Some (but not all) studies further find that receiving explicit instructions to identify rules in verbal form impairs performance (Reber, ).

Another class of cases concerns a phenomenon termed verbal overshadowing. In a classic demonstration, instructions to verbalize a face led to impaired facial recognition relative to a condition in which participants did not verbalize (Schooler & Engstler-Schooler, ). Such effects have been found for other perceptual stimuli (Fiore & Schooler, ; Melcher & Schooler, ), but do not extend to stimuli that are easy to verbalize (such as a spoken statement) (Schooler & Engstler-Schooler, ) or to logical problem solving (Schooler et al., ).

As a third example, studies find that asking people to generate verbal explanations for their observations supports the discovery of broad and simple patterns (Edwards et al., ; Walker et al., ; Williams & Lombrozo, ; ). But when the stimuli are designed such that these broad and simple patterns contain exceptions, participants who were prompted to explain learned more slowly and made more errors (Williams et al., ).
These effects are thought to arise from the mismatch between the representations or processes induced by a form of thinking (in this case, explaining) and the representations or processes that best support task performance (Lombrozo, ).

The effects reviewed so far plausibly concern impairments that arise from the representational limitations of language and the generalization of patterns found in language: language is not well-suited to encoding fine-grained perceptual discriminations (as required for face recognition), and language readily encodes some kinds of relationships (such as deductive entailment, or simple and broad patterns) but is less well-suited or frequently employed for others (such as complex finite state grammars, or patterns with arbitrary exceptions). Given that LLMs are likely to share limitations that arise from language and generalization, we might expect LLMs to exhibit patterns of impairment that mirror those found for humans on these tasks. We test these predictions in Section 4.

Prior work has documented additional impairments in humans from verbal reasoning, but for some it is less clear if they should generalize to LLMs. For example, explaining how inconsistent statements could be true makes participants less likely to recognize a logical inconsistency (Khemlani & Johnson-Laird, ). However, this assumes a reasonable baserate in recognizing logical inconsistencies – something that can be a challenge for LLMs with zero-shot prompting. Prior work has also found that verbal reasoning can be less accurate than visual or motor simulation (Schwartz & Black ; see also Aronowitz & Lombrozo ; Lombrozo ), but this is a consequence of information encoded in visual and motor representations that are likely not available to models. Finally, humans sometimes make poor choices that they regret when they deliberate over complex, multi-dimensional problems (Dijksterhuis, ) – plausibly a consequence of memory limitations that are not faced by LLMs. We anticipate that for these tasks CoT is less likely to reduce performance.

## 4 Experiments

Following the different failure settings of human deliberation described in Section 3, we select six tasks, each representative of a class of failure settings from the psychological literature, and conduct an experiment to test the effect of CoT on LLMs and LMMs.
Based on the considerations outlined at the end of Section 3, we identify two criteria under which we anticipate a task might result in CoT reducing performance:

Verbal thinking or deliberation hurts performance in humans.

The constraints governing human performance are the same as those governing AI models.

We use these criteria to develop a hypothesis for each task. We first cover three cases where both (i) and (ii) are satisfied, before discussing three cases where (ii) is not satisfied. In each, we scale up the classic psychology study that tested humans, while also adapting the task towards modern use-cases of large language or multimodal models.

### 4.1 Implicit statistical learning

##### Task.

The first class of tasks we examine are those involving implicit statistical learning. As described in Section 3, psychology studies have found that data that contain statistical patterns can be better generalized by humans when those patterns are not linguistically described. We explore this for LLMs by replicating the task of learning artificial grammars (Reber & Lewis, ; Whittlesea & Dorken, ; Van den Bos & Poletiek, ). In the task, artificial “words” are constructed using finite-state grammars (FSGs) and participants are tasked with identifying which words belong to the same category (i.e., are generated by the same FSG). In total, we constructed 4400 classification problems corresponding to 100 randomly sampled unique FSGs that were structurally similar to those used to test humans in Fallshore & Schooler (). Each classification problem consisted of 15 training examples generated from the grammar, and the model was given a new example and asked to classify it. Models were asked to classify 44 words per FSG, where 22 words belonged to the FSG and 22 did not. Words not belonging to the grammar were generated by replacing one letter from an existing word in the grammar. Details on problem generation are provided in Appendix A.1.

##### Hypothesis.

The artificial grammar learning task satisfies (i) based on the findings of Fallshore & Schooler (), where humans who conducted verbal thinking did worse on the problem. The task also satisfies (ii): verbal reasoning is thought to impair human performance due to the constraints of using explicit language-based processing, which is something shared by humans and LLMs using CoT. Thus, following our hypothesis, we predict that CoT will reduce LLM performance on the artificial grammar learning task.

##### Models and prompts.

We evaluate this task on several open- and closed-source models: OpenAI o1-preview, GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 70B & 8B Instruct, and Llama 3 70B & 8B Instruct. We considered two prompts, zero-shot and CoT (see Appendix A.2).

##### Results.

Consistent with our hypothesis, we find large reductions in performance when using CoT prompting compared to zero-shot prompting, displayed in Table 1. We find that when run on a randomly selected subset of 440 problems, OpenAI o1-preview, which has a form of CoT built into its responses, has a 36.3% absolute accuracy decrease compared to GPT-4o zero-shot on the same subset. Similarly, while there is limited performance change between conditions for Claude 3.5 Sonnet, we see that its performance is lower than the zero-shot accuracy of Claude 3 Opus. Across the other models, we find consistent decreases in performance when performing CoT: 23.1% in GPT-4o, 8.00% in Claude 3 Opus, 6.05% in Gemini 1.5 Pro, and 8.80% in Llama 3.1 70B Instruct. Weaker models such as Llama 3.1 8B Instruct and Llama 3 8B Instruct perform closer to chance (50%), but the reduction in performance caused by CoT remains statistically significant.

**Table 1: Results contrasting zero-shot and CoT for artificial grammar learning.**
|  | Zero-shot | CoT | Performance<br>decrease | $p$-value |
| --- | --- | --- | --- | --- |
| GPT-4o (subset) | 94.00% | - | $36.30\%$ | $<0.0001$ |
| OpenAI o1-preview (subset) | - | 57.70% |  |  |
| GPT-4o | 87.50% | 64.40% | 23.10% | $<0.0001$ |
| Claude 3 Opus | 70.70% | 62.70% | 8.00% | $<0.0001$ |
| Claude 3.5 Sonnet | 65.90% | 67.70% | -1.80% | $0.969$ |
| Gemini 1.5 Pro | 68.00% | 61.95% | 6.05% | $<0.0001$ |
| Llama 3 8B Instruct | 59.70% | 57.90% | 1.80% | $<0.05$ |
| Llama 3 70B Instruct | 60.50% | 58.30% | 2.20% | $<0.05$ |
| Llama 3.1 8B Instruct | 53.52% | 51.54% | 1.98% | $<0.0001$ |
| Llama 3.1 70B Instruct | 65.90% | 57.10% | 8.80% | $<0.0001$ |

### 4.2 Facial recognition

##### Task.

Another class of tasks we identify in Section 3 where verbal thinking reduces performance involves verbal overshadowing, where verbal thinking affects interactions with perceptual stimuli. We study this case using a classic human face recognition task, where participants are first shown a face of a person and then asked to select an image of the same person out of a list of candidates (Schooler & Engstler-Schooler, ). While psychological studies often include a distractor task between the initial person and the candidates to increase the difficulty, we did not use these for LMMs due to their weak performance. We scale this task from one recognition problem to a novel synthetic dataset of 500 problems across 2500 unique faces. For each problem, all faces were given the same described attributes for seven features: race, gender, age group, eye color, hair length, hair color, and hair type. We then generated a pair of images of the same person and four images of other people matching this description using stable-image-ultra (StabilityAI, ). We adjusted the generation process to ensure that the pair clearly consisted of the same person, while the others clearly did not (see Appendix B.1 for further details). One of the pair was selected to be the initial stimulus, while the other was shuffled with the four images of other people to create five candidate answers.
Models were prompted to identify which candidate matched the person from the initial stimulus.

##### Hypothesis.

The facial recognition task satisfies (i), as people who verbally reasoned about faces performed worse in Schooler & Engstler-Schooler (). (ii) is satisfied as human failure is attributed to the lack of granularity and inadequacy of language in describing the visual stimuli, a constraint relating to verbal thinking rather than people. Following our hypothesis, we predict that CoT will reduce performance on our facial recognition task in LMMs.

##### Models and prompts.

We evaluated this task on several open- and closed-source state-of-the-art LMMs: GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, InternVL2 26B, and InternVL2 Llama3 76B. Other models such as Llama 3.2 90B Vision and Molmo 72B were not considered as they do not support multiple image input. We considered two prompts, zero-shot and CoT, available in Appendix B.2.

##### Results.

We find results consistent with our hypothesis — every LMM tested shows a drop in performance when asked to perform CoT (see Table 2). Weaker models often answered that “all images are of the same person”, resulting in accuracies below the random chance rate of 20%. However, even under these conditions, we observe decreases in performance due to CoT.

**Table 2: Comparison of zero-shot and CoT prompts for facial recognition.**
|  | Zero-shot | CoT | Performance decrease<br>(absolute) | Performance decrease<br>(relative) | $p$-value |
| --- | --- | --- | --- | --- | --- |
| GPT-4o | 64.00% | 51.20% | 12.80% | 20.00% | $<0.01$ |
| Claude 3 Opus | 44.00% | 29.60% | 14.40% | 32.73% | $<0.0001$ |
| Claude 3.5 Sonnet | 97.80% | 94.80% | 3.00% | 3.07% | $<0.05$ |
| Gemini 1.5 Pro | 66.00% | 54.60% | 11.40% | 17.27% | $<0.05$ |
| InternVL2 26B | 9.20% | 6.00% | 3.20% | 34.78% | $<0.05$ |
| InternVL2 Llama3 76B | 15.77% | 13.77% | 2.00% | 12.68% | $0.44$ |

### 4.3 Classifying data with patterns that contain exceptions

##### Task.

A third class of tasks we identify where CoT may harm performance is learning to classify exemplars when there are exceptions to generalizable rules. As mentioned in Section 3, when humans try to explain the category membership of exemplars, they tend to hypothesize simple classification rules, which can lead to inefficient learning when data contain arbitrary exceptions to these rules.

To study if this phenomenon extends to CoT, we replicate a multi-turn vehicle classification task from Williams et al. (), where participants try to correctly assign binary labels to a list of vehicles. Participants are given feedback after each prediction, and conduct multiple passes over the list until they label all vehicles correctly in a single pass or exceed the maximum number of tries.
Vehicles in the task contained one feature that was almost fully correlated (80%) with the classification label, three features with no relation to the label, and one feature (the unique color) that individually identified the vehicle. Thus, participants could either try to learn the generalizable rule from the highly correlated feature and fail due to the exceptions, or they could learn the individual mappings from the identifying feature to the corresponding label. Human participants who were prompted to explain the classification of exemplars performed worse because they tended to attempt the former strategy.

Participants in the original study performed this explanation after receiving feedback. To more explicitly include inference-time reasoning, we modify the point at which verbal thinking is performed, asking the LLM to perform CoT before making each prediction instead.
In total, we constructed $2400$ vehicles — split into $240$ lists of ten vehicles each — and measured LLMs’ abilities to learn the labels of each list across up to $15$ passes (see Appendix C.1 for details).
Memory was implemented by including previous problems, guesses, and feedback in context.

##### Hypothesis.

This learning with exceptions task fulfills (i) as people tended to reason about generalizable rules when conducting verbal thinking, which increased the amount of time needed to learn the labels for the entire list (Williams et al., ). (ii) is satisfied as we expect models to replicate the tendency that humans have to focus on generalizable features when performing such thinking.

##### Models and prompts.

We evaluated this task on GPT-4o, Claude 3.5 Sonnet, and Claude 3 Opus. We only evaluated these models as the task requires long-context conversation capabilities, which other models such as Llama 3.1 70B Instruct were not sufficiently good at. We varied the prompt between direct and CoT, where the former corresponded to directly asking the model to classify with the memory in context (see Appendix C.2 for details).

**Table 3: Average number of rounds for models to learn labels using either direct or CoT prompting.**
|  | Direct | CoT | # Rounds increase<br>(absolute) | # Rounds increase<br>(relative) | $p$-value |
| --- | --- | --- | --- | --- | --- |
| GPT-4o | 2.9 | 12.5 | 9.6 | 331% | $<0.0001$ |
| Claude 3.5 Sonnet | 2.3 | 6.4 | 4.1 | 178% | $<0.0001$ |
| Claude 3 Opus | 2.4 | 5.5 | 3.1 | 129% | $<0.05$ |

##### Results.

In our experiments we find that CoT drastically increases the number of passes needed for the model to learn all the labels correctly. Averaged across the 240 lists, GPT-4o with CoT needed more than four times the number of passes to learn the labels compared to direct prompting, while Claude 3.5 Sonnet and 3 Opus both needed more than double (see Table 3).

We also investigated the per-round accuracy of GPT-4o and found that direct prompting consistently resulted in the model attaining perfect classification on the second or third iteration, while with CoT, the model was only able to correctly classify around 8/10 objects after 4 to 5 iterations (see Appendix C.3). The model was unable to surpass this degree of accuracy over the long run, likely due to CoT biasing the model to rely on the seemingly generalizable rules from the exemplars, while in the process, significantly down-weighing the usefulness of contextual tokens that explicitly contained all of the correct answers.

### 4.4 Tasks with a mismatch between human and model abilities

In the next three tasks, we consider cases where (i) is satisfied but (ii) is not. In other words, humans perform worse on the task after deliberation or verbal thinking, but we do not expect this to generalize to model CoT due to differences between humans and models. Reasons for this include models producing poor performance with zero-shot prompting — providing no opportunity for a decrease in performance, or humans and models possessing different limitations for task-relevant abilities, such as access to different kinds of information or memory resources.

##### Explaining a logical inconsistency.

Studies have found that when human participants are shown a pair of logically inconsistent statements and asked to explain their coexistence, they become worse at judging whether the statements are indeed logically inconsistent (Khemlani & Johnson-Laird, ). In the task, participants are provided with two sentences following the template: “If $A$ then it is always the case that $B$”, and either “$A$, but it is not the case that $B$” or “It is not the case that $B$”. The former corresponds to a logical inconsistency between the statements, while the latter does not. In one condition humans were first asked to explain why an inconsistent pair could coexist before providing a judgement on their inconsistency, while in another they conducted the same explanation after providing a judgement. The former condition saw its classification performance drop significantly compared to the latter, indicating that deliberation towards the incorrect class reduced human performance.

The original human experiment contained 12 unique $\{A,B\}$ pairs. To scale this task to evaluate LLMs, we leverage existing entailment pairs in natural language inference tasks, which we use to fill in $A$ and $B$ to form the sentences. We used a combination of three datasets: The Stanford Natural Language Inference (SNLI) dataset, the Multi-Genre Natural Language Inference (MNLI) dataset, and a synthetic LLM-generated dataset of 100 entailment pairs. We filtered the datasets for pairs which were labeled “entailment” (i.e., $A$ entails $B$). In addition, we limit the maximum length of $A$ and $B$ such that the template forms coherent sentences. In total, we evaluate on 1608 pairs of $\{A,B\}$ pairs: 675 from SNLI, 833 from MNLI, and 100 synthetic. Each pair was used to construct two classification problems, one consistent and one inconsistent, for a total of 3216 problems which we use to evaluate LLMs. For more details on problem generation see Appendix D.1.

We evaluated a suite of state-of-the-art LLMs on this task: OpenAI o1-preview (on a subset of 30 synthetic questions), GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 70B Instruct. We used zero-shot prompting and two conditions of CoT: one where the model is simply asked to reason before answering, and another that follows the original experiment by asking the model to explain the inconsistency directly (see Appendix D.2 for details). Results were very similar across the two CoT conditions, so we report an average over both.

Zero-shot prompting resulted in poor performance on this task, with most models performing close to chance (see Table 4). CoT often improved this performance, attributable to both the low base performance and the logical reasoning component for which CoT is typically helpful. This was especially pronounced in GPT-4o, where CoT improved performance by over 40% on pairs from MNLI and SNLI. However, in the model that performed best with zero-shot prompting, Gemini 1.5 Pro, we did see a decrease in performance with CoT. These results suggest that a reasonable level of performance with zero-shot prompting is a prerequisite for CoT to reduce performance.

**Table 4: Results comparing zero-shot and CoT across the logical inconsistency task using stimuli from MNLI, SNLI, and synthetic LLM generation.**
|  | MNLI | SNLI | Synthetic |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Zero-shot | CoT | Zero-shot | CoT | Zero-shot | CoT |
| OpenAI o1-preview (subset) | - | - | - | - | - | 86.5% |
| GPT-4o | 53.2% | 93.9% | 51.4% | 94.3% | 51.0% | 74.0% |
| Claude 3.5 Sonnet | 65.2% | 67.5% | 67.4% | 69.8% | 56.7% | 57.8% |
| Claude 3 Opus | 62.7% | 58.8% | 66.2% | 58.7% | 54.5% | 51.8% |
| Gemini 1.5 Pro | 73.2% | 68.2% | 68.8% | 63.9% | 60.5% | 61.5% |
| Llama 3.1 70B Instruct | 55.6% | 81.6% | 50.4% | 82.3% | 50.0% | 65.8% |

##### Spatial intuitions.

Psychologists have documented cases involving spatial reasoning in which humans generate more accurate responses after visual or motor simulation compared to verbal reasoning. To investigate whether this applies to models, we replicate a cup-tilting task from Schwartz & Black (). In the task, participants are shown an image of two rectangles with varying height and width, representing two cups — one empty and one that contains some water. Participants are then asked to estimate the water that should be added to the empty cup so that when tilting both cups, water will reach the rim at the same angle (see Figure 1, SI). While the original task had participants draw the water level on the empty cup, LMMs were unable to do this consistently. Thus, we turned the task into a multiple choice question by adding markings $A$ – $D$ to the side of the empty cup and asking the model to choose one. Incorrect answer options were generated by adding Gaussian noise to the correct answer while satisfying the constraint that answer options must be a certain distance apart. We scale up this task by varying the dimensions of cup sizes and water height, creating a total of 100 problems, each with a code-drawn image containing the cups and multiple choice answers (see Appendix E.1 for details). We evaluated with zero-shot and CoT prompts on several open- and closed-source large multimodal models: GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and InternVL2 Llama3 76B.

In this setting, it is unlikely that large multimodal models would share the same motor simulation capabilities as humans due to lack of representations built from motor experience. As the improved performance in the non-verbal thinking condition requires spatial or motor intuition, (ii) is not satisfied. Consistent with this, we did not observe statistically significant differences between zero-shot and CoT prompts (see Table 5). Generally, we expect this to extend to other tasks where models lack task-relevant priors that humans possess.

**Table 5: Results comparing zero-shot and CoT on the spatial intuition task.**
|  | Zero-shot | CoT | Performance change<br>(absolute) | Performance change<br>(relative) | $p$-value |
| --- | --- | --- | --- | --- | --- |
| GPT-4o | 38% | 40% | +2% | +5.00% | $0.61$ |
| Claude 3.5 Sonnet | 42% | 38% | -4% | -10.53% | $0.28$ |
| Claude 3 Opus | 42% | 38% | -4% | -10.53% | $0.28$ |
| Gemini 1.5 Pro | 35% | 36% | +1% | +2.78% | $0.99$ |
| InternVL2 Llama3 76B | 39% | 31% | -8% | -25.81% | $0.67$ |

##### Aggregating features for a decision.

The final category of tasks we consider are complex, multidimensional tasks that exceed human working memory capacity. A study conducted by Dijksterhuis () showed that humans made poor choices when deliberating over apartment options when provided with a large amount of information about various decision features.
In the study, participants were shown 48 statements for one second each, where the statements described either a positive, negative, or neutral aspect of one of four apartment choices. Afterwards, they were asked to select the best apartment after either deliberating or completing a distractor task. The authors found that the distractor task condition actually improved performance over deliberating based on memory.

To scale this task up to evaluate LLMs, we generated 80 unique apartment features with four statements per feature: One positive, one negative, and two in between. We then asked GPT-4o to rate the impact each statement would have on the impression of an average tenant from -5 to 5. We randomly sampled apartments by choosing one statement per feature, and constructed sets of four where the best apartment had a per-feature average score $\Delta\in\{[0.1,0.3],[0.3,0.5],[0.5,1]\}$ higher than the next-best option. We sampled 300 such sets of four apartments (100 per $\Delta$ range) to form choice tasks (see Appendix F.1).
We tested several open- and closed-source LLMs with zero-shot and CoT prompts: GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus, and Llama 3.1 70B Instruct. Llama 3.1 70B Instruct was often unable to return an answer after deliberating in the CoT condition, resulting in lower performance.

In this setting, the differences in working memory between humans and models led to (ii) being unsatisfied. Humans performing the task were forced to rely on their aggregate impressions of each apartment due to the large amount of information. However, even after scaling up the number of contextually relevant statements over six-fold, models were simply able to access all feature statements in-context. Consistent with this, we observed somewhat positive effects from CoT (see Table 6). Essentially, the availability of context turns the problem into summing up the importances of the features, which the model is able to leverage additional inference-time reasoning to conduct. This highlights fundamental differences in capabilities between models and humans for solving certain tasks, highlighting the need for (ii) — to consider whether the constraints governing human performance in these failure cases generalize to LLMs.

**Table 6: Results for apartment selection task across four models and three ranges of $\Delta$.**
| $\Delta$ | [0.1, 0.3] | [0.3, 0.5] | [0.5, 1] |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Zero-shot | CoT | Zero-shot | CoT | Zero-shot | CoT |
| GPT-4o | 47% | 45% | 57% | 56% | 80% | 87% |
| Claude 3.5 Sonnet | 50% | 62% | 62% | 72% | 81% | 95% |
| Claude 3 Opus | 35% | 50% | 57% | 58% | 72% | 84% |
| Llama 3.1 70B Instruct | 42% | 6% | 44% | 5% | 43% | 20% |

## 5 Discussion

Chain-of-thought prompting is an effective way to expand the capacities of large language and multimodal models. However, knowing that CoT significantly decreases performance in specific settings is important in considering when it should be deployed, and in particular whether it should be deployed by default. By using cases where verbal thinking decreases human performance as a heuristic, we successfully identify three such settings where CoT results in large decreases in model performance, contributing towards discourse regarding both these discussions.

While we make a connection between human cognition and large language and multimodal models, we do not claim that these systems operate in the same way or that models should be anthropomorphized. Rather, we see this connection as a tool for identifying settings where either the structure of the task or shared limitations result in negative effects of verbal thinking for both humans and models. Our exploration was guided by considering not just (i) that deliberation reduces human performance but also (ii) that there are no other constraints at work that result in a meaningful difference between humans and models.

Across experiments on six diverse tasks, we find overwhelming evidence that CoT results in a decrease in performance when both (i) and (ii) are satisfied, but not when (ii) is not satisfied. More generally, these results suggest that we can successfully leverage the existing literature in cognitive psychology on deliberation and thinking to find cases that are informative about the performance of CoT.

In the remainder of this section, we consider the limitations of this work and some future directions.

### 5.1 Limitations

##### Types of inference-time reasoning.

Since the invention of chain-of-thought prompting, researchers have developed various prompts that are specifically suited to target application domains, as well as more elaborate general-purpose prompts with multiple forward passes such as tree-of-thought (Yao et al., ) and self-consistency (Wang et al., ). We test the effectiveness of tree-of-thought on GPT-4o for the implicit statistical learning task (see Appendix A.3), and find that while it improves classification accuracy (64.55%  vs. 62.52%) , this is still far from the zero-shot performance of 94.00%, suggesting that our approach extends across inference-time reasoning techniques. However, future work is still required to determine whether this generalizes to other task domains and methods of eliciting verbal thinking in models.

##### Scope of application.

While our psychology-based heuristic offers a strategy for identifying failure cases of CoT, it is unlikely to cover all cases where CoT decreases performance.
Existing psychological research has been guided by a variety of theoretical and practical considerations towards studying humans, but this does not offer an exhaustive or representative sample of all tasks, and will miss cases that are uniquely interesting to study in models but not humans. Thus, we envision our contribution to be complementary to existing evaluation methods in natural language processing.

As we’ve seen across our six sets of experiments, knowledge of what drives the decrease of performance in humans can be leveraged to generate predictions about the effects of CoT for different tasks, but this remains an inferential step that requires careful reasoning and an understanding of model capabilities. Despite these limitations, our method can be used to identify large and consequential failures of CoT, as documented in our three failure tasks. It also offers valuable cross-domain insight that can help build intuition and contribute to our overall understanding of inference-time reasoning.
On the flipside, the existence of capable LLM/LMM systems also allows us to reflect upon the reasons why human performance can be degraded by deliberation. By considering when CoT’s effects mirror humans and when they do not, we can distinguish when the task or mechanisms shared by humans and models are responsible for failures, versus when the issues are with uniquely human strategies or limitations.

##### Alternative explanation for why CoT does not replicate human results.

There exists an alternative explanation for why we do not see drops in performance in the latter three tasks — that how we implemented the tasks for LLMs removed the failure effect. It’s possible that with different implementations we might in fact see decreased performance, mirroring what we see in humans. While we explored a large number of variations for the latter three tasks, these explorations were not exhaustive due to the endless changes that one could make to the prompts. In other words, because the tasks were inevitably changed to scale up the evaluation and match more realistic applications of models, it’s also possible that these changes are what explain the human-model mismatch.

### 5.2 Future directions & broader implications

While we have focused on CoT reasoning, the framework presented here suggests a more general strategy for leveraging empirical work on humans to characterize the performance of models: jointly considering (i) psychological results concerning humans and (ii) whether the relevant constraints that shape human performance extend to those models. For example, it could be fruitful to investigate effects of comparison or analogical prompting (Lombrozo, ) through this lens.

Altogether, we visualize studying how to evaluate and improve models as a collaborative effort between natural language processing methods (e.g., benchmarks), psychological insights, and the burgeoning literature of evaluations that compare human and model performance. By sharing knowledge and building a strong collaboration between these different disciplines, we can utilize rich insights from decades of literature that studied humans to advance the domain’s intuitions about these models and analyze an even broader array of tasks and applications for AI.

#### Acknowledgments

Experiments were supported by Azure credits from a Microsoft AFMR grant. This work and related results were made possible with the support of the NOMIS Foundation. Special thanks to Howard Chen and Mengzhou Xia for providing invaluable feedback on this project.

## Appendix A Implicit Statistical Learning Task

To study cases involving implicit statistical learning, we consider an artificial grammar learning task. In the task, LLMs are provided with letter string train examples that belong to the same category, and are tasked to classify whether a new string belongs to the same category.

### A.1 Generation of artificial grammar learning dataset

In the original psychology experiments (Fallshore & Schooler, ; Reber & Lewis, ), participants performed the classification task on strings generated by a fixed finite state grammar (FSG) constructed by the researchers (see Figure 2). A string is generated by the FSG if it corresponds to a valid path along the directed edges from the source node $s$ to the sink node $t$, where the letters on the path are appended together.

Figure: Figure 2: The FSG used in Fallshore & Schooler () and Reber & Lewis (), two classic studies on artificial grammar learning. This FSG was used the generate strings for all participants in both studies. We form our dataset using FSGs that follow a similar structure.
Refer to caption: extracted/5987228/figures/fsg_original.png

In our experiments, we expand the experiments massively to 100 randomly sampled FSGs that follow the same rough structure of those used in the experiment.
To scale up the dataset, we construct and sample from all possible FSGs that obey the following rules. For a visual representation please see Figure 3.

- •
6 nodes total, including source $s$, sink $t$, and four nodes $x_{1},\dots,x_{4}$.
- •
Edges $(s,x_{1}),(s,x_{3}),(x_{2},t)$ and $(x_{4},t)$ are always present.
- •
Edge $(x_{1},x_{2})$ is always present to avoid isomorphisms and the null case where no paths exist from $s$ to $t$.
- •
The remaining middle edges $\{(x_{1},x_{3})$, $(x_{1},x_{4})$, $(x_{2},x_{1})$, $(x_{2},x_{3})$, $(x_{2},x_{4})$, $(x_{3},x_{1})$, $(x_{3},x_{2})$, $(x_{3},x_{4})$, $(x_{4},x_{1})$, $(x_{4},x_{2})$, $(x_{4},x_{3})\}$ can either exist or not, for a total of $2^{11}$ combinations.
- •
Each $x_{i}$ can have self-loops, e.g., $(x_{1},x_{1})$, for a total of $2^{4}$ combinations.
- •
Letters on each edge are randomly selected from the capital alphabet, for a total of $26^{8}$ combinations.
- •
Each FSG should be able to generate at least $37$ unique strings with length $\leq 8$.
- •
The construction of the FSG is unique with respect to the three graphical isomorphisms that each FSG satisfying the rules could have.

Figure: Figure 3: The potential FSGs used in our dataset. Directed edges that always exist are in black, while the others that could exist are dashed and in gray. Bi-directional arrows denote two potential directed arrows. The letters on each edge represent a random sample.
Refer to caption: extracted/5987228/figures/fsg_ours.png

For each FSG, we sampled paths of up to length 8 and used them as stimuli for the experiment. Following Fallshore & Schooler (), we sampled $37$ to use in the experiment, assigning 15 to be training examples and $22$ to be positive test examples. We also constructed $22$ negative test examples by sampling a random string from the FSG, perturbing one letter in a randomly selected position to another letter that exists on some edge of the FSG. We ensured that the negative examples did not belong to the FSG.

In total, this yielded 4400 individual questions asked to the large language models. Each question was asked individually after the 15 training examples. See the next section for the specific prompts.

### A.2 Prompts

For our experiments, we prompted the models using one zero-shot prompt and one CoT prompt. The zero shot prompt is shown in Table 7.

**Table 7: Example prompt for artificial grammar learning task, zero shot.**
| Prompt: |
| --- |
| These strings were generated according to a certain set of rules. Does the following string also<br>follow the same set of rules?<br>$[$test example$]$<br>Please ONLY answer “Yes” or “No”. |

The CoT prompt uses one of the most common prompting methods for chain-of-thought, replacing the last line with, ‘Please reason about your answer before answering “Yes” or “No”.’

When conducting pilot experiments, we also tried a version of the prompts where we asked models to “memorize the following letter strings” in the first line of the prompt as this was more in-line with the original human experiment. We found that results were extremely similar to the more general version shown above, and thus discarded this more specialized case.

### A.3 Tree-of-thought experiments

**Table 8: Results comparing zero-shot, CoT, and ToT on a subset of the artificial grammar learning task.**
|  | Zero-shot | CoT | ToT | Performance<br>decrease (CoT) | Performance<br>decrease (ToT) | $p$-value<br>(CoT) | $p$-value<br>(ToT) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4o | 94.00% | 62.52% | 64.55% | $31.48\%$ | $29.45\%$ | $<0.0001$ | $<0.0001$ |

To analyze whether our hypotheses about model chain-of-thought extend to other types of inference-time reasoning, we evaluated the performance of GPT-4o with tree-of-thought (ToT) (Yao et al., ) on a subset of 10 artificial grammars, totaling 440 examples. Given the input prompt, we asked the LLM to generate five different thoughts to explain whether the string also followed the same set of rules as the in-context examples. Then, five votes were conducted to select the best thought, which we parsed and compared to the ground truth label. We found that ToT resulted in a small improvement over the task (64.55% vs. 62.52% accuracy, see Table 8), but this performance was still much worse compared to GPT-4o zero-shot accuracy (94.00%). This suggests that the reduction in performance is not only associated with CoT, but also other types of inference-time reasoning.

## Appendix B Facial recognition task

To study tasks where language impairs the recognition of visual stimuli, we focus on a facial recognition task, where VLMs are asked to select one of five candidate images that matches the face of a provided image. The original experiment in Schooler & Engstler-Schooler () had participants view a 30-second video of an individual robbing a bank and then perform a 20-minute distractor task, before either writing down descriptions of the robber’s face or doing a distractor task for 5 minutes. Participants were then provided with 8 verbally similar faces to choose from, and those who performed the written description performed much worse (38% vs. 64% accuracy) at identifying the robber.

### B.1 Generation of Facial Recognition Dataset

To adapt this task to testing models, we made a few design decisions. First, we chose to replace the initial video stimuli with an image of the person’s face to allow for the testing of vision language models. Next, we chose to remove the distractor tasks. This decision was based on pilot results indicating that common psychology distractor tasks such as the n-back task (Kirchner, ) resulted in large amounts of noise in model outputs, while other distractors were of limited effect on the model due to it being able to retrieve the earlier stimuli in-context. Furthermore, even without the distractor, models already showed a large difference in performance across zero-shot and CoT conditions.
Thus, our task was simplified to a facial matching task, where a model was given a human face as input and responded with the index of the matching face image as its output.

To generate the faces for the facial recognition dataset, we use stable-image-ultra (StabilityAI, ). We experimented with other models such as DALL-E 2 (OpenAI, ) and DALL-E 3, but found generation capabilities were significantly less realistic than stable-image-ultra. This difference was especially pronounced in generating realistic facial images of people in racial minorities.

To cover a diverse set of human faces, we prompt models to generate faces with features age {young, middle-aged, old}, race/ethnicity {asian, black, hispanic, white}, gender {man, woman}, eye color {brown, blue, green}, hair color {brown, black, blonde, red, gray}, hair length {long, short}, and hair type {curly, wavy, straight}. We removed some low-probability combinations such as red hair with asian ethnicity due to poorer quality of image generation. Then, we randomly sampled combinations of features to form a descriptor set.

One issue with stable-image-ultra is that when asked naively to generate an image of the same person as another image, it would alter some details such as ear shape, nose shape, or other facial ratios that would make it impossible to be the exact same person. We addressed this issue by prompting the stable-image-ultra image generation model to

> “Generate two realistic images of the same person, one on the left and one on the right. The person should have the following description: [description]”.

After doing so, we were able to manually check and verify that the faces shown in the two images is clearly the same to the naked eye. One of these images was assigned to be the initial stimuli shown, while the other would be shuffled into the list of answers.

We also ensured that the other remaining images were 1) clearly not of the same person as the image, and 2) the pose of the person, which was often similar between the pair of generated images, was also replicated in the other fake answers.
This was achieved using the following prompt with the edit structure task in the set of image control API calls from StabilityAI:

> “Generate an image of a unique person with the same pose and style as the image provided. The person should have the following description: description”. image input: [correct answer image]

Once all the answers were generated, we manually verified the quality of generated images, and ensured that each of 1) and 2) were satisfied. An example of the images generated for a problem are shown in Figure 4.

Figure: Figure 4: An example of the six images generated for a problem. The first row contains one of the pair of generated images. The first image in the second row contains the other image in the pair, and the remaining four images are incorrect answers generated from this image.
Refer to caption: extracted/5987228/figures/face_example.png

### B.2 Prompts

To evaluate models on the facial recognition task, we used one zero-shot prompt and one CoT prompt. The zero shot prompt is shown in Table 9.

**Table 9: Example prompt for facial recognition task, zero shot.**
| Prompt: |
| --- |
| Here is an image of a person.<br>$[$image of initial person$]$<br>Select the image that contains the same person as the person in the first image.<br>$[$five images of possible matching faces$]$ |

The CoT prompt uses the most original chain-of-thought prompting method by appending “Let’s think step by step” to the end of the zero-shot prompt, with no other changes.

## Appendix C Data with exceptions task

In this task, we analyze the effect that CoT prompting has on the ability of LLMs to learn a classification of objects that appear to follow a pattern, but with exceptions. In these types of settings, Williams et al. () reveal that when humans are given opportunities to deliberate after receiving feedback, they learn more slowly and make more errors compared to those who do not deliberate. The active form of thought mentally ingrains incorrect patterns that shift when exposed to successive unexpected answers, altogether leading to the creation of many deceptively incoherent lines of reasoning throughout the learning process that hinder the ability to directly keep track of the correct labels even after multiple passes.

### C.1 Vehicle dataset generation

We build off of the experimental set-up in Williams et al. () where in each trial, we first create a list of objects (vehicles) that are either warm- or cold-climate, which is the label which we want models to learn. Based on this label, we generate one feature that correlates with this target label completely (see Column 2 of Table 10), and flip this 20% of the time to create exceptions in the data.

In addition to this discriminating feature, following Williams et al. (), we also include 1) one unique feature which is different for each object and 2) three additional features whose values are randomized and have no connection with the object class. The unique feature in the original experiment was vehicle color, which we replaced with the license plate for realism. An example setup is depicted in Table 10.

**Table 10: Sample vehicle classification list. Boldened features indicate flipped labels that break the initial classification pattern.**
| Unique features | Pattern-related features | Irrelevant features |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | License Plate | ‘Cold’ (Class A)/‘Warm’ (Class B) climate |  | Transmission | Seat covers | Doors |
|  | A23BCD | Drives on glaciers |  | Manual | Cloth | Two |
|  | B34EFG | Made in Norway |  | Automatic | Vinyl | Two |
|  | C45HIJ | Used in mountain climbing |  | Automatic | Vinyl | Four |
|  | D56KLM | Drives in jungles |  | Manual | Vinyl | Four |
|  | E67NOP | Has treads |  | Manual | Cloth | Two |
|  | F78QRS | Heavily insulated |  | Manual | Vinyl | Four |
|  | G89TUV | Made in Africa |  | Manual | Cloth | Four |
|  | H90WXY | Has wheels |  | Automatic | Cloth | Two |
|  | J12ZAB | Lightly insulated |  | Manual | Vinyl | Two |
|  | K23CDE | Used on safaris |  | Automatic | Vinyl | Two |

We sampled 240 sets of 10 vehicles each and prompt the model to learn the labels of the vehicles in a multi-turn setting, which we detail below.

### C.2 Prompts

Models are provided with text descriptions of a vehicle’s features one vehicle at a time, iterating through the full set of ten vehicles repeatedly up to 15 times. Each time the model is given a set of features, it predicts the corresponding label and subsequently receives feedback for its answer. In contrast to previous experiments, the problems, the model’s previous guesses, and the feedback given to the model are all stored in-context and provided to the model in its next prediction.

In each iteration, the vehicles’ order shown to the participant is shuffled. Prompting stopped when the model correctly classified all of the vehicles in one iteration, or reached 15 iterations without performing this successfully. We used one zero-shot prompt and one CoT prompt. The zero-shot prompt was as follows in Table 11.

**Table 11: Example prompt for vehicle classification task, zero shot.**
| [Chat history including previous prompts, model predictions, and feedback] |
| --- |
| Prompt: |
| The vehicle description is as follows:<br>License plate: [license plate]<br>Descriptor: [descriptor]<br>Transmission: [transmission]<br>Seat Cover: [seat cover]<br>Doors: [doors]<br>Is this vehicle more likely to be a Class A or Class B vehicle? Only answer with ‘A’ or ‘B’. |

In the CoT condition, instead of replicating the human study and asking the model to deliberate after each piece of feedback, we modify the prompt asking the model to make a prediction. Specifically, we replace “Only answer with ‘A’ or ‘B’.” with
“Let’s think step by step and answer with either ‘A’ or ‘B’. If you are unsure, feel free to guess and explain your reasoning”.

We append the last sentence because we observed that sometimes the model would refuse to answer based on lack of information.
While we could have also implemented deliberations after each feedback to stay more faithful to the human experiment, our ultimate goal is to inform chain-of-thought, and CoT is most often applied during the process of asking questions to the model rather than having it reflect by itself.
Furthermore, we believe that these settings are approximately equivalent: Deliberation in human experiments would focus on explaining the feedback provided, but this is also the case in this paradigm because the model would perform reasoning on the previous feedback provided when performing CoT during the prediction of the next label.

### C.3 Per-round accuracy analysis

Figure 5 depicts the aggregate accuracy (correctly predicted examples out of 10) of GPT-4o with direct and CoT prompts over 15 iterations through the list. Although CoT performs better than direct on the first iteration of the list, direct prompting quickly surpasses the performance of CoT by attaining perfect classification ability on the third iteration. Chain-of-thought prompting stagnates in performance at an accuracy level equivalent to the percentage of exemplars whose class designation adheres to the corresponding first-glance generalizable rule (80%). This suggests that the verbal thinking of CoT biases the model towards predicting via generalizable rules, even when there are more useful features that map exactly to correct answers in context.

It is worth noting that CoT’s tendency towards generalizable rules is often very helpful in other settings. For example, CoT does benefit from this tendency in the predictions of the first pass when all stimuli are previously unseen. This is in line with our conclusion that different strategies for prompting should be chosen based on the task, and neither is always better than the other.

Figure: Figure 5: Aggregate learning curve (number of correct objects classified out of 10) for GPT-4o prompted via direct prompting and chain-of-thought over 15 iterations. Direct prompting attains perfection very quickly, whereas chain-of-thought prompting results in stagnation.
Refer to caption: x1.png

## Appendix D Logical inconsistency task

Here, participants were tasked to evaluate whether a set of two statements were logically inconsistent. Statement pairs followed two forms: The first statement was always of the form $A\rightarrow B$, where $\rightarrow$ denotes implication, and the second statement was either of the form $A\wedge\neg B$ or $\neg B$, where $\wedge$ denotes the boolean AND operation, and $\neg$ denotes boolean negation. If the second statement was of the form $A\wedge\neg B$, the pair is inconsistent, whereas if the second statement was of the form $\neg B$, the pair is consistent. Khemlani & Johnson-Laird () found that if you ask humans to deliberate specifically as to why $A\wedge\neg B$ was plausible, they would subsequently be less accurate at identifying logical inconsistencies between the statements.

### D.1 Logic dataset generation

To construct the dataset for the task, we first assigned claims to $A$ and $B$, and then filled in the template to construct the actual statements. To do the first part, we took statements where $A\rightarrow B$ made logical sense following Khemlani & Johnson-Laird (). While the original authors simply hand-constructed 12 pairs of claims, we use a combination of natural language inference (NLI) datasets where pairs of statements are filtered to be of the “entailment” condition: MNLI, SNLI, and a synthetic datset generated by prompting GPT-4o using the prompt:

> Generate a list of 100 true statements of the format “if A then B”. For each statement generate the result in JSON format with separate fields for index, A and B.

To construct the actual statements, we fit $A$ and $B$ into the templates in Table 12.

**Table 12: Sentence template for logical inconsistency task.**
| Statement 1: If $[A]$, then it is always the case that $[B]$.<br>Statement 2 (conflict): $[A]$, but it is not the case that $[B]$.<br>Statement 3 (no conflict): It is not the case that $[B]$. |
| --- |

In addition, to avoid having entailment pairs where statements are more than one sentence long or contain multiple clauses, we limited the maximum amount of words per claim ($A$ or $B$) to seven. This allowed the sentences in the problem to flow smoothly, while still maintaining a large population of entailment pairs. In total, we conducted experiments on 675 pairs from SNLI, 833 pairs from MNLI, and 100 pairs of claims that were synthetically generated, for a final sum of 1608 pairs of $\{A,B\}$. This corresponded to 3216 questions asked per model, over which we calculated model accuracy.

### D.2 Prompts

We prompted models using one zero-shot prompt and two CoT prompts.
The prompt in the zero-shot condition was as follows:

**Table 13: Example prompt for logical inconsistency task, zero shot.**
| Prompt: |
| --- |
| The following are two statements:<br>1. $[$Statement 1$]$<br>2. $[$Statement 2$]$<br>Can both of these statements, as explicitly stated, be true at the same time? Please ONLY<br>answer with “Yes” or “No”. |

The two chain-of-thought prompts altered the last line in the prompt to the following two sentences, respectively:

- •
Can both of these statements, as explicitly stated, be true at the same time? Please reason about your answer and then answer “Yes” or “No”.
- •
Can both of these statements, as explicitly stated, be true at the same time? Please first explain why statement 2 could be true and then answer “Yes” or “No”.

Here, the first prompt follows the standard “reason about your answer before answering” CoT request, whereas the latter is a more specific request aimed at more closely replicating the human study.

## Appendix E Spatial intuition task

In this task, participants were given drawings of two drinking glasses, one filled with water and one empty. They were asked to estimate the level of water that the second glass would need to be filled to such that the two glasses, when tilted to a certain degree, would have the water they contain reach the rim of the glass at the same angle (Schwartz & Black, ).

To simplify the task for the model, we changed the task from drawing a line (image manipulation) to multiple choice (text output) by marking four separate heights on the side of the empty glass, labeling them $A$ through $D$, and asking the model to select a letter.

### E.1 Motor simulation task dataset generation

To scale up our dataset, instead of fixing the dimensions of the glass that contains water, we varied the width and height in {2, 3, 4} and {4, 5, 6} respectively (units are per 100 pixels). Then, following Schwartz & Black (), we created scenarios where the width and height of the empty cup was {wider, less wide, same width} and {taller, less tall, same height}. We also varied the amount of water that was in the original glass between {$\frac{1}{4},\frac{1}{2},\frac{3}{4}$} of its total height. Altogether, this resulted in $243$ unique combinations of problems compared to the original $9$.

For each problem, we computed the exact height $h$ that the empty cup would need to be filled with water to in order to get the water to the rim at the desired angle. Then, we sampled from Gaussian noise

$$ $x_{i}\sim\mathcal{N}(0,\sigma^{2})$ $$

in order to generate the other answer choices $\{a_{i}=h+x_{i},i\in\{1,2,3\}\}$, where $\sigma^{2}$ is half the distance from the correct answer to the maximum height of the glass. Furthermore, we ensured that none of the answer choices $a_{i}$ provided were above the maximum height of the cup, below zero, or within distance $\epsilon$ of each other. $\epsilon$ was an empirically determined parameter that controlled the difficulty of the problem, while also having a lower bound due to a limit for how closely the multiple choice letter options could be to each other on the graphical representation of the empty glass. A visual representation of the final problem setup is in Figure 6.

Figure: Figure 6: An example of the water problem presented to large multimodal models. The glass on the left is filled with water, and the task is to determine which letter choice the empty glass should be filled to such that when the two glasses tilt to the same angle, water reaches each of their rims at the same time.
Refer to caption: extracted/5987228/figures/water_example_real.png

### E.2 Prompts

We use one zero-shot prompt and one CoT prompt. The zero-shot prompt is shown in Table 14. For the CoT prompt, we replaced “Do not include anything else” with “Let’s think step by step”.

**Table 14: Example prompt for spatial intuition task, zero shot.**
| Prompt: |
| --- |
| On the left of the figure provided, there is a rectangular glass with its water level indicated<br>with the blue line.<br>On the right, there is a target glass. The target glass has four height<br>markings (A, B, C, D), each indicating a different water level. Which marking should the<br>target glass be filled to so that both glasses pour at the same angle?<br>Provide your answer using the letter that matches the correct height marking. Do not<br>include anything else.<br>$[$Prompt image$]$ |

## Appendix F Working memory preference task

In this task, participants were shown individual statements about one of four apartments frequently in succession. Each statement describes a different aspect of one apartment, and participants’ tasks were to determine which apartment was overall most favorable. However, due to limits in human working memory, their performance in identifying the most beneficial apartment decreased when they tried to reason about the features of each apartment.

### F.1 Apartment dataset generation

To extend this task to LLMs, we scaled up the number of stimuli to hopefully induce an increased pressure on the long-context capabilities of the model. Towards this effort, we first tested the limit of the amount of different features an apartment could have with the help of GPT-4o. We found that the model started repeating aspects of apartments after around 80 unique features. Then, we asked the model to generate positive, negative, and more neutral versions of statements regarding these features:

- •
“Generate 80 positive statements about different aspects of an apartment. None of the statements should be about the same aspect.”
- •
“The following are 80 positive statements about aspects of an apartment. For each, generate a corresponding negative statement that is the exact opposite. Make sure that all of the negative statements can coexist with positive statements that are not its direct correspondent.
[positive statements]”
- •
“The following are 80 positive [$\dots$] For each, generate a corresponding neutral statement that is about the same aspect, is worse than the positive version, but is not negative. Make sure that all of the neutral statements can coexist with positive statements that are not its direct correspondent.
[positive statements]”
- •
“The following are 80 negative [$\dots$] For each, generate a corresponding neutral statement that is about the same aspect, is better than the negative version, but is not positive. Make sure that all of the neutral statements can coexist with negative statements that are not its direct correspondent.
[negative statements]”

We then manually considered conflicts between pairs of statements that were not of the same feature, and manually replaced the only feature statement that had a conflict with another feature. Thus, cohesive descriptions of apartments could be sampled by randomly selecting one of the four statements for each of the 80 features.

Next, we asked GPT-4o to rate the importance of each statement based on how much the “statement affects the desirability of the apartment for the average tenant, from -5 to 5, with 5 being most desirable”. Based on this, we could estimate the ground truth quality of each apartment by making the assumption that the features’ utilities sum up linearly.(^1^11Note that this is sometimes untrue; e.g., close proximity to a grocery store is much more meaningful when an apartment has a kitchen.) We then randomly sampled apartments with one statement per feature, and computed the score of an apartment as the mean of the feature scores. We then constructed sets of four apartments where the best apartment had at least an average score $\Delta\in\{[0.1,0.3],[0.3,0.5],[0.5,1]\}$ higher than the next-best option. This was to ensure that there is a clear best apartment for the average tenant while not making the task too simple, which were also requirements in the original human study (Dijksterhuis, ). Intuitively, $\Delta$ can be considered as a difficulty level, where apartments are closer in rating for lower $\Delta$ problems and are thus harder to get correct.

Sampling randomly, this led to a total of three datasets corresponding to three ranges of $\Delta$, each containing 100 sets of four apartments.

### F.2 Prompts

For this task, we used one zero-shot and one CoT prompt in our evaluations. The zero-shot prompt is shown in Table 15. The CoT prompt replaces “Respond with only the number of the apartment, do not include anything else.” with “Let’s think step by step”.

In our pilot experiments, we also tried a variety of prompts such as replicating the distractor task using a verbal n-back task, setting a time limit for the model (i.e., “you have three minutes to think about the problem”) or using phrases such as “very carefully think” that were present in the original experiment, but the first resulted in too much noise whereas the latter two did not change the results.

**Table 15: Example prompt for working memory apartments task, zero shot.**
| Prompt: |
| --- |
| You are an AI assistant designed to evaluate the desirability of four apartments<br>for a potential tenant. You will be given a list of statements about the apartment<br>candidates and how much the tenant likes or dislikes an apartment with the<br>quality described by the statement. Your task is to determine which apartment<br>is the most desirable based on the given criteria.<br>The statements are as follows:<br>$[$statements$]$<br>Which apartment is most desirable to the tenant? Respond with only the number<br>of the apartment, do not include anything else. |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: What do the authors think of the following pattern -- CoT hurts when zero-shot performance is already good?

There seems to be a pattern where CoT helps on harder tasks (where zero-shot performance is low) and hurts on easier tasks (where zero-shot is already good): e.g., CoT hurts on grammar where GPT-4o gets 87.5% zero-shot, but helps on logic where zero-shot is ~50%. While this pattern doesn't hold universally (e.g. on the grammar task, models that start out performing badly still lose out from CoT) so it's not a full-on alternative explanation, it might be worth discussing. (There is also the fact that the difficulty is determined by both task & model, not just the task, and that small models are generally bad at CoT so a different mechanism might be hurting their performance compared to one hurting larger models.)

The authors even note something similar for the logic task ("reasonable zero-shot performance is prerequisite for CoT to reduce performance"). I think it'd be helpful to discuss this more generally, and/or perhaps try experiments controlling for baseline performance (e.g. find tasks with similar zero-shot performance but different CoT effects, or test whether making tasks harder/easier changes the CoT effect).
soundness: 3
strengths: - I liked this paper, and think it makes a valuable contribution in understanding LLM cognition: it's clearly important to know when CoT might decrease model performance.
- Timely topic of interest to the ICLR community.
- Solid methodology supported by fairly broad experimental results with many different models.
summary: Inspired by studies of tasks where explicit thinking hurts human performance, this paper studies when chain of thought (CoT) can hurt performance in LLMs and VLMs. Authors present three tasks where machine learning models and humans are similar (CoT / explicit reasoning are detrimental for both), and three tasks where CoT helps LLMs but explicit reasoning is detrimental for human performance.
weaknesses: ## Psychology-inspired hypothesis framing feels very post-hoc

I believe the main weakness is in how the authors frame and explain their results rather than in the empirical work itself.

The paper's story of testing psychology-inspired hypotheses feels rather post-hoc. The authors present their expectations for each task's results before showing the empirical findings, but many of these expectations seem very unintuitive. These expectations for each task are based on two criteria, and I believe criterion (ii), "human constraints generalize to AI", has a massive amount of room for interpretation and post-hoc result fitting, since it's very unclear how to tell in advance if a given task satisfies this criterion.

I'd suggest the authors either switch to a more empirical story (here's where LLMs are similar to humans, here's where they differ, here are our ideas for why this happens), or better articulate intuitions for why we should expect given results before presenting them.

>In particular, three of these tasks satisfy (ii): tasks involving implicit statistical learning, tasks where language is ill-suited for representing differences in stimuli, and tasks with data that contains no simple generalizable rules

As an example, the quote above feels post-hoc -- perhaps the authors should say that they FIND that three tasks satisfy criterion (ii)?


## Other weaknesses

1. The main text would benefit from more detailed task descriptions & exposition beyond Figure 1. E.g. for the grammar task, I was initially quite surprised that CoT resulted in worse performance there, but got a lot less surprised once I saw task examples & prompts in the appendix.

2. While presented results with standard CoT prompts support the thesis that naive CoT can be harmful, additional prompting variations would make the findings more robust. I'd be especially keen to see more extensive attempts to increase performance via CoT on the first three tasks, where naive CoT appears to hurt performance.

3. For all tasks where CoT hurts performance, I would have appreciated examples of models failing (ideally with some analysis, but even just including a few representative CoT trajectories in the appendix would be helpful).

4. The final task (aggregating features for a decision) feels unfair to humans, particularly this part: *>Each statement was provided for only 1 second before disappearing*. Allowing humans see all statements at the same time would have been a very relevant baseline -- do authors know of human studies utilizing a setting like that? Overall I am glad this task is included, but I think the paper should caveat that this comparison is somewhat unfair.


5. [Inverse scaling (McKenzie et al., 2023)](https://arxiv.org/abs/2306.09479) is extremely related to this work and should be discussed.

## Nits
- I'd recommend splitting up the incredibly long sentence in section 2.2.
- Line 356: open-source -> closed source.
- Line 406: close to task -> close to chance?

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 4
questions: Comments: I am conflicted about the work. I do find the topic highly important, but if the paper is published as is it might introduce more confusion than value to the community (if we conclude that CoT often limits LLMs in ways similar to verbal reasoning in humans while in fact the mechanism might be different). Hence, unfortunately, I can not recommend it to acceptance. I am, of course, open to rebuttals and discussion.

I recognize the hard work already put into this paper, and I see two potential paths for this paper going forward: 1) expand the experiments to include more controls and tasks or 2) substantially scale down the claims. 

I believe that condition "B" needs to be either removed from the theory (it can still be used as a retrospective explanation for why condition "A" is not perfectly predictive). Or there needs to be a very thorough investigation of how exactly it can be applied to an arbitrary task.

Extra suggestions:
One (arguably less important) control to consider is this: same experiments, but with the reversed order of requests. E.g. "answer only A or B, then provide an explanation," instead of "provide an explanation, then answer only A or B". This would help see if changes in prompt formulation alone might affect model answers, even before the explanation is given. I did not include it in the "Weaknesses" as not as crucial for the central claims that the authors are making.

Typos/minor:

"We only evaluated on open-source models for this task because it requires long-context conversation understanding capabilities, especially for the CoT setting," - none of the listed models are open-source. Anthropic and Openai models are closed-source and commercial. Also, it's worth noting that there are open-source models that support long contexts. Nevertheless, I understand the decision to go with SOTA commercial options. This did not affect my evaluation.
soundness: 2
strengths: Clarity: the paper is clearly written and is a pleasure to read. The logic of the experiment is clearly described. Moreover, the supplementary information in the appendix helps to understand the details of the study.

Significance:
The paper addresses a highly important question. It's now more important than ever to understand conditions affecting LLM performance, and the mechanisms behind them, especially whether and when LLM and human performance are governed by similar limitations.

Originality:
The authors draw original insights from Cognitive Science literature and propose a novel way to look at LLM performance under CoT.
summary: The paper studies the cases when CoT (Chain of Thought) prompting hurts LLM performance. The authors hypothesize that there are two key conditions: A) human performance on the task drops when explicit verbal reasoning is required B) the conditions/limitations determining human performance generalize to LLMs. The authors report LLM performance on a range of tasks, all satisfying A, but some failing B, and show that when A and B are satisfied, there is a noticeable drop in LLM performance when CoT is used.
weaknesses: **********Soundness**********

In many ways, the paper uses a Cognitive Science approach to studying LLMs, which is justly becoming more and more popular. I believe, however, that it's necessary to apply the same standards of rigor to such studies as is usually done in Cognitive Science.

Unfortunately, there are a few issues with experimental design that I find concerning.

** On the second criterion **
Fundamentally, the claim of the paper is that CoT hurts LLM performance when
A) Explicit language-based deliberation hurts human performance on the task.
B) The processes needed to solve the task are similarly constrained between humans and LLMs.

Condition "B", however, is very vague and hard to apply. For example, when a multimodal LLM is presented with an image, it is fed image tokens one by one, which is vastly different to how humans perceive images. It is not unreasonable to expect that such a process will be governed by substantially different rules to those of humans. The authors claim that the task does satisfy condition "B" because in case of humans, the drop in performance is attributed to the limitations of using language to describe images, but how do we know in advance whether such limitations would generalize when language and images are fundamentally represented in a much more aligned way?

For another example, consider this justification for why the implicit statistical reasoning task satisfies "B": "verbal reasoning is thought to impair human performance due to the constraints of explicit language-based processing, which are potentially mirrored by LLMs." Whether such constraints are mirrored by LLMs is, again, a separate research question. It's especially moot since LLMs are known to essentially retrofit their explanations to the answer, so it's not clear to what extent and in which conditions LLM explanations dictate the actual flow of their reasoning (see https://arxiv.org/abs/2205.03401, for example).

I believe that many other criterion B assignments in the paper are similarly tenuous.

Overall, I find the second criterion arbitrary and difficult to apply to any new given task. What's more concerning, it can be retrospectively used to explain any result, since for almost any task, LLMs are both substantially similar and substantially different from humans. Even the working memory aspect is not as clear-cut as authors claim: long-context LLMs do not actually have full mastery over their context (see https://arxiv.org/abs/2307.03172 or https://aclanthology.org/2023.findings-emnlp.1005/, for example), so it's not entirely fair to assume that their working memory is only limited by the length of their context.

** Controls for condition A **

Another major limitation is the insufficient amount of control and comparison conditions. The first such omission is that the authors did not systematically investigate or discuss cases where LLMs are known to show worse performance under CoT, while it's not natural to expect the same for humans. For example Sprague et. al 2024 (https://arxiv.org/pdf/2409.12183, cited in the paper, but not discussed in depth), shows that CoT can, on average, hurt performance on such tasks as text classification, encyclopedic knowledge, or context-aware QA. I am not aware of human studies that would show that verbal reasoning hurts encyclopedic knowledge tests or question answering.

In short, I believe that condition "A" that authors proposed ignores a large number of tasks where CoT can hurt performance in LLMs, but (most likely) not in humans. I understand that not everything can fit into one study, but it seems that if we want to claim that A is informative/predictive of LLM performance, we need to carefully address why do we also often see performance drops under CoT prompting on tasks where A is (most likely) not satisfied.

** Controls for prompt length & distractors **

If the LLMs context is seen as analogous to human working memory, it's crucial to add distractor tasks to understand whether it's indeed CoT-related verbal reasoning that hurts model performance. It might be that including any irrelevant or vaguely related text after the task presentation and before the final answer would hurt performance. This is especially clear in the CDE task, where the CoT condition requires the LLM to integrate information over a much larger span of text, but can be an issue in other conditions also.

In short, I believe it's necessary to add distractor tasks to normalize the number of tokens between the question and the answer for CoT and Zero-Shot conditions.

If I understood correctly, the authors themselves acknowledge that in their pilot experiments, distractors generated a lot of noise in the answers. It is another reason to suspect that the presence of irrelevant or unnecessary text between the question and the answer might hurt model performance regardless of whether this information is in the form of unnecessary CoT-style reasoning or something else entirely.

It is a big problem for this study because the authors attempt a mechanistic explanation of the reasons behind performance drops that they observed. E.g. in the case of facial recognition, the claim is that it's the inadequacy of language as a means of solving the task that hurts performance. If the performance is also hurt by irrelevant text, it'd severely undermine the claim.

Overall, unfortunately, I believe that these weaknesses severely undermine the reliability of the reported results.

**********Novelty**********
The novelty of the research is, unfortunately, indirectly undermined by the considerations above. In the sense that right now, I can't confidently say that LLM performance on CoT tasks is governed by the theory proposed by the authors. Such theory, if properly substantiated, would be extremely novel and valuable. But unless the experimental limitations are resolved, the presented results only show that CoT sometimes helps LLM performance and sometimes not, and that is not sufficiently novel.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could the poor CoT performance in multimodal settings be due to insufficient training of the multimodal model itself, rather than the conclusions presented in the paper?
soundness: 2
strengths: 1. The study successfully bridges human cognitive research and machine learning, using psychological findings to predict model performance, which adds depth to understanding CoT's effects.
2. The paper covers multiple settings, carefully categorizing tasks that are likely to benefit or suffer from CoT prompting.
3. The results highlight the potential limitations of CoT in practical applications, urging caution in its default implementation.
4. The insights into CoT failures are valuable for researchers and practitioners, as they guide future CoT applications in complex, varied tasks.
summary: This paper investigates the effects of chain-of-thought (CoT) prompting on large language models (LLMs) and large multimodal models (LMMs). The authors argue that while CoT can often enhance model performance, it may reduce effectiveness in tasks where "overthinking" similarly hampers human performance. Drawing from cognitive psychology, they analyze specific task types, such as implicit statistical learning and visual recognition, to understand where CoT might negatively impact LLM/LMM outcomes. By leveraging insights from human cognition, the authors seek to identify scenarios where CoT harms model accuracy and efficiency.
weaknesses: 1. The paper lacks a detailed analysis of error cases, specifically comparing instances where CoT fails, while direct prompting does not. This would help in understanding why CoT fails in certain conditions.
2.  No code or replication setup is provided.
3.  The paper does not reference recent works from the past two years that discuss CoT's robustness, nor does it clarify its distinctions from those studies. Addressing these would contextualize the current findings within the larger body of literature and clarify whether performance declines are due to fundamental limitations of CoT or specific task-related issues.
(e.g:https://arxiv.org/abs/2410.05229)
4. Could you provide scores from commonly used benchmarks to better support and contextualize your conclusions?
5. The experimental design lacks depth in exploring CoT robustness. Without detailed ablation studies and a coherent analysis, the work appears overly focused on isolated task performance and misses opportunities to offer a deeper, unified understanding of CoT's limitations and potential.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I want to know the real CoT outputs of models for each task. I think you should provide a case study in the Appendix.
2. I'm a little bit confused about Table 4. I don't think this result can lead to the conclusion that "These results suggest that a reasonable level of performance with zero-shot prompting is a prerequisite for CoT to reduce performance."  for example Claude models. So my question is: what is the **reasonable level** of performance with zero-shot prompting? How to define it.
3. In Table 6, why Llama-3.1-70B so bad in CoT? Your reason is not clear. What are your hyperparameter settings? like decode strategy, max_new_token. I would like to see the real CoT output of Llama-3.1-70B, does it output tokens repeatedly?
soundness: 3
strengths: 1. The idea of investigating scenarios where CoT may reduce model performance is novel and interesting.
2. The paper is well-structured and the flow of the framework proposed in this work is detailed and sufficiently clarified.
summary: The paper explores the effect of CoT in LLMs and LMMs, while CoT is often advantageous, there are specific settings in which it reduces performance. Inspired by cognitive psychology,  the authors focus on tasks where deliberation impairs human performance, hypothesizing that similar tasks could reveal limitations in model performance under CoT. Three task types are identified where CoT negatively impacts models.  They find significant accuracy drops across state-of-the-art models in these areas. Additionally, they find certain tasks where human constraints don’t generalize to models that do not exhibit the same performance decreases.

This study offers insights for selectively deploying CoT to avoid unintended performance declines in LLMs and LMMs.
weaknesses: 1. The experimental settings are not clear (especially the hyperparameter settings of the model), which is not conducive to reproduction.
2. There are inconsistent definitions, such as LMMs and VLMs.

Meta-review: This paper investigates when CoT prompting reduces LLM performance by drawing parallels to cases where verbal thinking impairs human performance. While the core idea is interesting and results show some significant performance drops with CoT, reviewers raised valid concerns about the theoretical framing, statistical significance, and experimental controls. The authors made good faith efforts to address these through reframing and additional statistical analyses during rebuttal, but key limitations remain around proving their proposed heuristic is meaningfully better than chance at identifying CoT failure cases. The paper would benefit from more rigorous empirical validation before the claims can be fully substantiated.

Decision: Reject