The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning
Authors: Tianshi Zheng
Yixiang Chen
Chengxi Li
Chunyang Li
Qing Zong
Haochen Shi
Baixuan Xu
Yangqiu Song
Ginny Wong
Simon See
Venue: Accepted by TMLR ()

Abstract: Chain-of-Thought (CoT) prompting has been widely recognized for its ability to enhance reasoning capabilities in large language models (LLMs). However, our study reveals a surprising contradiction to this prevailing perspective within the fundamental domain of pattern-based in-context learning (ICL). Through extensive experiments involving 16 state-of-the-art LLMs and nine diverse pattern-based ICL datasets, we demonstrate that CoT and its reasoning variants consistently underperform direct answering across varying model scales and benchmark complexities. To systematically investigate this unexpected phenomenon, we designed extensive experiments to validate several hypothetical explanations. Our analysis uncovers a fundamental hybrid mechanism of explicit-implicit reasoning driving CoT’s performance in pattern-based ICL: while explicit reasoning falters due to LLMs’ struggles to infer underlying patterns from demonstrations, implicit reasoning—disrupted by the increased contextual distance of CoT rationales—often compensates, delivering correct answers despite flawed rationales. This hybrid mechanism explains CoT’s relative underperformance, as noise from weak explicit inference undermines the process, even as implicit mechanisms partially salvage outcomes. Notably, even long-CoT reasoning models, which excel in abstract and symbolic reasoning, fail to fully overcome these limitations despite higher computational costs. Our findings challenge existing assumptions regarding the universal efficacy of CoT, yielding novel insights into its limitations and guiding future research toward more nuanced and effective reasoning methodologies for LLMs.

Published in Transactions on Machine Learning Research (11/2025)
The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning
Tianshi Zheng",, Yixiang Chen",, Chengxi Li Chunyang Li', Qing Zong' Haochen Shi', Baixuan Xu', Yangqiu Song', Ginny Y. Wongg, Simon See 'The Hong Kong University of Science and Technology, 2NVIDIA (tzhengad, ychenla, clidu) ust.uk, yqsongdese ust.hk
Reviewed on OpenRevicw:
Abstract
Chain-of-Thought (CoT) prompting has been widely recognized for its ability enhance reasoning capabilities in large language models (LLMs). However, our study reveals surprising contradiction to this prevailing perspective within the fundamental domain of pattern-based in-context learning (ICL). Through extensive experiments involving 16 state- of-the-art LLMs and nine diverse pattern-based ICL datasets, we demonstrate that CoT and its reasoning variants consistently underperform direct answering across varying model scales and benchmark complexities. To systematically investigate this unexpected phenomenon, we designed extensive experiments to validate several hypothetical explanations. Our analysis uncovers fundamental hybrid mechanism of explicit-impliées reasoning driving CoT's performance in pattern-based ICL: while explicit reasoning falters due to LLMs' struggles to infer underlying patterns from demonstrations, implicit reasoning disrupted by the increased contextual distance of CoT rationales- compensates, delivering correct answers despite flawed rationales. This hybrid mechanism explains CoT's relative underperformance, as noise from weak explicit inference undermines the process, even as implicit. mechanisms partially salvage outcomes. Notably, even long-CoT reasoning models, which excel in abstract and symbolic reasoning, fail fully overcome these limitations despite higher computational costs. Our findings challenge existing assumptions regarding the universal efficacy of CoT, yielding novel insights into its limitations and guiding future research toward more nuanced and effective reasoning methodologies for LLMs.
(a)
2C-ACC
function
ARC
SCAN
navberr of Deeventeent
MHHARC
WHOCAN
Meaneyy
Direct
Cof
Role
Us Function
TST
Numbar Domonutations
Figure 1: (a) Performance direct answering, CoT, ReAct, and ToT across 9 ICL benchmarks, averaged over 16 LLMs. (b) Performance gaps between direct answering and CoT with varying numbers of demonstrations.
"Equal Contribution.

---

Published in Transactions on Machine Learning Research (11/2025)
1 Introduction
Chain-of-Thought (CoT) prompting (Wei et al., 2022) has emerged as pivotal technique in advancing modern large language models (LLMs). By encouraging models to generale explanatory rationales (i.e., intermediate reasoning steps) prior to producing the final answer, CoT significantly improves the reasoning capabilities of LLMs, enabling them to achieve more accurate and interpretable outcomes. Extensive evidence has demonstrated that CoT is particularly effective in tasks involving mathematical, symbolic, or code-based data, and also leads to substantial improvements in general natural language reasoning and factual reasoning (Sprague al., 2024; Zheng et al., 2024; Yu al., 2024). Building upon the foundation of CoT,, numerous advanced reasoning Trameworkssndee and Staph-offThoouhe (GoT) (Besta et al., 2024)-harl been proposed to facilitate problem-solving in more sophisticated scenarios. Furthermore, the emerged ability of generating long-Col CoT reasoning steps has become a driving factor behind advanced reasoning models such as OpenAl ol (OpenAI, 2024), o3-mini (OpenAI, 2025), and Deepseek-FF R1 (DeepSeek-A) et al., 2025). Beyond empirical improvements, recent theoretical analyses also indicate that CoT enables transformers to perform inherently serial computations and thus overcome their intrinsic limitations in parallel computation (Li al., 2024).
ReAct (Yao et al., 2023b), Tree-of-Thhee Tree-of-Thought (ToT) (Yao et al., 2023a),
Despite the well-established effectiveness of CoT, several studies have also explored its limitations. For instance, Ye & Durrett (2022) conducted experiments on carlier LLMs such as GPT-3 (Brown et al., 2020) and OPT (Zhang et al., 2022), demonstrating that these models may generate unreliable explanations in few-shot textual reasoning scenarios. Additionally, Stechly et al. (2025) highlighted CoT's reliance on problem-specific prompts and its limited scalability in plamning tasks. Furthermore, Zhang et al. (2025) showed that although CoT effectively improves performance, it still faces inherent limitations stemming from the complexity involved in navigating the prompt and answer spaces. Nonetheless, CoT remains widely recognized in current LLM literature a broadly effective approach to LLM problem-solving, consistently outperforming direct answering.
In this paper, we reveal a strikingly counterintuitive finding: Chain-of Thought prompting unexpectedly degrades LLM performances in certain problem-solving contexts. We investigate in-context learning (ICL) tasks, in which LLMs learn to predict the output of a test instance by extrapolating beyond demonstrations in the form of input-output pairs. Specifically, our analysis focuses on pattern- based ICL benchmarks where the relationships (e.g., patterns, rules, functions) between inputs and outputs are explicitly definable. Through extensive experiments involving 16 modern LLMs and diverse ICL benchmarks (spanning textual, numerical, and symbolic data), we demonstrate that CoT and its reasoning variants (e.g., ToT, ReAct) consistently underperform direct answering by a significant margin (Figure la). Furthermore, we observe that this performance gap widens as the number of in-context demonstrations increases (Figure 1b). Our findings challenge the prevailing assumption that CoT is universally effective across various reasoning tasks.
To systematically investigate the underlying causes of this unexpected 'curse" effect, we formulate and evaluate three core hypotheses through extensive tailored experiments:
Hypothesis 1. The CoT rationale increases the contextual distance between demonstrations and answers, disrupting the few-shot learning structure and thereby degrading performance.
Hypothesis 2. LLMs struggle to infer underlying patterns from demonstrations when using CoT.
Hypothesis 3. LLMs struggle to apply inferred patterns to test instances when using CoT.
The experimental results empirically validate Hypotheses 1 and 2, providing valuable insights into the limitations of Chain-of-Thouuht prompting in in-context learning scenarios.
Interestingly, we observed that LLMs employing CoT often achieve correct answers even when the inferred patterns are incorrect. This observation points to a hybrid mechanism in the CoT mechanism for ICL (Hypothesis 4): the final prediction arises from an interplay between explicit reasoning (articulatod through CoT rationales) and implicit reasoning (similar to direct answering). where both processes contribute to pattern inference and execution. However, LLMs limited ability to infer accurate patterns explicitly
2

---

Published in Transactions on Machine Learning Research (11/2025)
(as validated by Hypothesis 2) introduces noise into the reasoning process, as flawed rationales disrupt the prediction pipeline. Compounding this issue, the increased contextual distance caused by CoT's inserted rationales further diminishes the efficacy of implicit reasoning (as validated by Hypothesis 1). Consequently, CoT prompting underperforms direct answering, which relies exclusively on robust implicit mechanisms. Further experiments reveal that, even long-CoT reasoning models despite consuming 40x more inference tokens-achieve only comparable or inferior performance standard LLMs using direct answering.
In summary, our findings advocate for more muanced perspective Chain-of- Chain-of-Thought prompting. Although CoT has demonstrated considerable success in enhancing the reasoning capabilities of large language models, our analysis has revealed critical limitations, especially within pattern-based, in-context learning scenarios. By providing deeper insights into the underlying mechanisms behind these limitations, we highlight that the benefits of CoT rationales are not universally applicable, emphasizing the need for adaptive and context- aware reasoning approaches. Consequently, this work contributes a more balanced and comprchensive undorstanding of CoT, informing the development of more robust and fiexible reasoning methodologies, and paving the way for future innovations aimed al optimizing large language model performance.
2 Preliminaries
Our investigation focuses in-context learning tasks characterized by explicitly defined input- output functions. Specifically, consistent and verbalized pattern governs the relationship between each input-output pair within the demonstrations. In this section, we provide formal definition of pattern- based in-context learning and describe model inference under both direct answering and Chain-of-Thought prompting.
2.1 Pattern-based in-context learning
In pattern-basd based in-context learning, LLMs are provided with a limited number of demonstration pairs, each comprising an input and its correspouding output. These pairs adhere to an explicit, consistent, and verbalizable pattern or rule. Formally, the task can be defined as follows:
Given set of demonstration examples D {(01, vn), (I22, v2),..., (Irk, yk)), where cach input-output pair (s;, Mi) conforms to specific pattern rule f, the goal is to predict the output Vlest for a new input Frests where (Itest, Vtest) also adheres to the same underlying pattern f. Formally, we have:
y: = f(x;) for all (z,, Wi) DU {(Itests )teet))
The pattern-based ICL lasks examined in this paper span various types of data, including textual, numerical, and symbolic data, and involve explicit rules such as arithmetic progressions, logical relationships, string manipulations, or symbolic transformations. Pattern-based ICL is best understood from a task of pattern induction, where the model identifies and applies a unified rule from a set of inpul-oulput, pairs. Therefore, further tasks such as complex mathematical reasoning requiring explicit multi-step solutions is not within our scope, though such tasks may be discussed in the broader context of general ICL.
2.2 Direct answering chain-of-thought prompting
In this subsection, define and compare two prompting paradigms central to our analysis Direct Answering and Chain-of-Thought Prompting.
Direct Answering In the Direct Answering paradigm, the LLM generates the test output Vrest based solely on the provided instructions, in-context demonstration examples D, and the test input Ptest- Formally, the problem-solving process can be modeled as:
P(ycest
Ztest D, Instructions)
Here, the model explicitly required to produce the final output directly, without generating intermediate reasoning steps explanatory rationales.
3

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: In-context learning datasets in our experiments.
Dataset
# Demos
Modality
Size
ARC-AGI (Chollet, 2019)
2~-10
Symbolic
835
MiniARC (Kim et al., 2022)
2~~8
Symbolic
149
ID-ARC (Xu et al., 2024)
3
Symbolic
901
SCAN (Lake & Baroni, 2018)
5m-8
Textual
1,000
MiniSCAN (Nye et al., 2020)
14
Textual
1,000
COGS (Kim & Linzen, 2020)
10
Textual
1,000
SALT (Zheng et al., 2025a)
4
Textual
1,200
List Function (Rule, 2020)
3
Numerical
1,250
RAVEN (Zhang al., 2019)
Numerical / Symbolic
1,259
Total
8,594
Chain-of-Thought Prompting In contrast, Shain-of-Thoughee Prompting involves a two-stage response process. First, the LLM generates explicit intermediate reasoning or rationale conditioned on the instructions, demonstrations D, and test input Frest Second, it produces the final output Ihest. based on this rationale, alongside the original context (instructions, D, and Ttest). This process is formally expressed as:
p(rationale Ttest; D, Instructions) p(yeest rationale, Ftest, D, Instructions)
Notably, the demonstration examples D are identical in both paradigms and do not include explicit reasoning steps. Consequently, our targeted task formulation differs from the few-shot CoT approaches commonly employed in standard QA tasks, where demonstrations explicitly illustrating CoT reasoning steps are provided. Additionally, we experimented with advanced reasoning frameworks, including ReAct and Troc-of-Tiie Troooh--Though prompting, in which explicit reasoning guidance is provided prior to the task instructions. The detailed prompting template is presented in Appendix D.
3 Datasets and models
Datasets We conduct experiments diverse selection of pattern-based in-context learning datasets spanning multiple modalities 1) Symbolic: Pattern-based transformations between symbolic matrices, e.g., ARC-AGT and MiniARC. 2) Textual:
Rule-based translations between natural language and artificial languages, e.g., SCAN and COGS. 3) Numerical: Pattern-based or function-based projections between numerical vectors or matricos, e.g., List Functions and RAVEN.
All selected datasets are reasoning-intensive, stress-testing the abstract and inductive reasoning abilities of LLMs. Details of datasets are provided in Table We include further data processing details in Appendix C.
Models We evaluated 16 open-source and proprietary LLMs with varying parameter sizes, with details in Appendix A. Note that long-CoT reasoning models, such as ol and Deepseek-R1, are analyzed separately from the main group of LLMs due to their limited compatibility with direct answering. As these models are specifically optimized for multi-step explicit, reasoning, they provide a crucial test case for our hypothesis and are therefore given a dedicated analysis in Section 6.
4 Main results
The main experimental results are illustrated in Figure 2 (full results in Appendix E). Across nine ICL benchmarks, LLMs employing direct answering substantially outperform CoT, achieving a relative improvement of 20.42% (absolute 5.10%). Compared to ReAct and ToT, direct answering yields relative improvements of 36.34% and 47.17% (absolute 8.02% and 9.64%), respectively. In terms of task modality,
4

---

Published in Transactions on Machine Learning Research (11/2025)
ARC-AGI
Direct Answering
CoT
ReAct
ToT
Accurary (%6
MiniARC
Accurary (%6
20
1D-ARC
30
Accurary PS3
40
SCAN
100
Accuracy 186
MiniSCAN
50
Aocuraty (86)
COGS
Accurary (%)
10
Jrewsn LINAA4 QNR22-1 OWALING DeGIHoI
Garmaa SALT
30
Accuracy (%)
20
List Function
30
Accuraty (5)
20-
RAVEN
Accuracy (N6
20
10
QMP25-1
Gareal4)
6%4s
Gew4l5ee (ome-)5pp
Figure 2: Detailed benchmark performance of LLMs with direct answering, C,Т, ReAct, and ToT. Gemma2 models were excluded from ARC-AGI experiments due to limited context length.
the performance gap between direct answering and CoT is most significant on symbolic ICL tasks (i.c., ARC-AGI, MinIARC, ID-ARC, and RAVEN), with a relative improvement of 41. 41.88%: in contrast, this advantage decreases 10.42% on textual ICL tasks (i.e., SCAN, MiniSCAN, COGS, and SALT). Regarding model size, since most benchmarks used in study be considered relatively out-of-distributiion

---

Published in Transactions on Machine Learning Research (11/2025)
compared to the LLM training corpora", smaller LLMs (e.g., Llama3.1-8B, Qwen2.5-7B) tend to exhibit lower overall performance, as well as more pronounced limitations when utilizing CoT other reasoning variants. In contrast, larger models (e.g., GPT-4o, Deepseek-V3. achieved better overall performances, in which reasoning frameworks occasionally achieve performance comparable to direct answering. Novertheloss, there are only a few entries in which reasoning frameworks yield positive oulcome from the additional consumption of inference tokens.
Moreover, for two benchmarks that allow flexibility in the number of demonstrations (COGS and List Function), conduct experiments by varying the demonstration count in the context, ranging from 3 to 15. As illustrated in Figure 1b, the performance gap between direct answering and CoT widens as the number of shots increases. This further substantiates the limitations of CoT under different contextual configurations.
These experimental findings reveal a surprising 'curse' of CoT, where reasoning frameworks consistently underperform direct answering in pattern-based ICL lasks with more sophisticated variants (ReAct, ToT) performing even worse. This counterintuitive phenomenon challenges conventional assumptions about the benefits of explicit reasoning in LLMs and motivates our systematic investigation into the underlying mechanisms behind this performance degradation.
5 Why chain-of-thought fails in in-context learning?
In this section, we systematically diagnose the root causes of CoT's inefficacy through a hypothesize- aadee--ndd-ttttete methodology. We design targeted experiments to validate or refute potential explanations for this limitation. Details of all four experiments are in Appendix B.
5.1 The Contextual Distance Curse: how CoT disrupts few-shot learning
In-context learning, as delineated by Brown et al. (2020), assumes that few-shot demonstrations are presented as a coherent, uninterrupted sequence, enabling the model to process them as a unified contextual signal for learning. However, under Chain-ofThhuuht prompting, the insertion of intermediate rationales between demonstrations and the final answer prediction may disrupt this continuity. We thus propose our first hypothesis:
Hypothesis 1. The CoT rationale increases the contextual distance between demonstrations and answers, disrupting the few-shot learning structure and thereby degrading performance.
Formally, we define contextual distance as the number of tokens separating the end of the in-context demonstrations (i.e., the input-output pairs provided in the prompt) from the position in the sequence where the model begins generating the final answer output. In the context of Chain-of- -hoo-gh prompting, this distance generally equals the token length of the generated reasoning rationale, which is inserted between the demonstrations and the answer.
To Lest this hypothesis, we designed two controlled experiments to isolate and evaluate the effect of contextual distance and CoT:
Dummy Rationale Experiment To disentangle the semantic content of CoT from its structural impact, we instructed LLMs to generate a semantically neutral 'dummy" rationale prior to predicting the final answer, thereby preserving the contextual distance while climinating reasoning-speciffc effects. We controlled two variables: modality and length. For modality, we considered textual and symbolic data. In the textual condition, LLMs recited excerpts from Shakespeare's Sonnets; in the symbolic condition, they generated countdown list from specified integer to one. These tasks were chosen to minimize generation variance and prevent unbounded outputs. For length, we varied the dummy rationale size: reciting 1, 2, 4, or 8 sonnets (approximately 150 tokens per sonnet) and counting down from 50, 100, 200, or 400 (approximately 3 Lokens per number). This yielded contextual distances ranging from 150 to 1200 tokens, encompassing typical CoT rationale longths (150 to 500 tokens).
TTh SCAN dataset might be subject data contamination some extent.

---

Published in Transactions on Machine Learning Research (11/2025)
Dummy Rationale (Shakespeare)
Dummy Rationale (Count Down)
Rationale Prontloading
cocs
Accuracy (MinsoC. coos)
Acouracy 11 1----
Accunsc (1b-anca
Accuradyy CD0 ARCI
Counting
franttodding
Figure 3: (a) Average performance with dummy rationale in Shakespeare's Sonnet. (b) Average performance with dummy rationale in countdown list. (c) Effect of rationale frontloading. All scores represent. mean accuracies across six LLMs.
Tt is important to acknowledge that this 'dummy rationale' approach is not without its limitations. Producing Shakespearc-like text or a procedural countdown is not identical to producing task-relevant rationale and may introduce process differences unrelated to contextual distance (e.g., stylistic or format-specific generation effects), which is distinct from the structural effect of contextual distance. Despite these potential confounders, these tasks were chosen for their highly controllable output length and low semantic overlap with the pattern-recognition problems, allowing
introduce a lengthy, unrelated text. block that structurally mimics a CoT rationale. Therefore, while not perfect control, the consistent, performance degradation observed in this experiment provides strong evidence that is highly consistent with our contextual distance hypothesis.
Note that we adopt in-response dummy rationales, as this approach preserves the relative positioning between the chat template and the rationale, consistent with standard CoT prompting. However, an alternative format that involving in-prompt dummy rationales has also been proposed (Lanham et al., 2023). Experimental results demonstrate that both methods provide evidence supporting the contextual distance curse (see Appendix B.1.2).
Rationale Frontloading Exporiment To preserve CoT semantics while climinating contextual distance, we first elicited CoT rationales from LLMs for each test instance. Specifically, for each instance, we oblained rationale under the regular CoT setting, then reused the same rationale (with the final answer removed to prevent leakage) in the front-loading condition by placing it before the demonstrations and query. Thus, the rationale content is identical across conditions, and only its position in the prompt (i.e., the contextual distance) differs. This approach ensures that the model has access to the same reasoning content, without. separating demonstrations from the answer prediction.
The experimental results are presented in Figure 3. From the dummy rationale experiment, we observe that LLM performance generally declines as contextual distance increases. The only exception occurs the countdown task on the COGS datasct, where LLMs frequently refuse to generate dummy rationales when instructed to count down from 200 or 400. From the rationale frontloading experiment, we find that performance substantially improves when rationales are prepended to the in-context demonstrations. These results provide significant evidence supporting Hypothesis 1. However, we also note that dummy rationales outperform CoT (on MiniARC and COGS), even at greater contextual distances, while foodtl rationales still underperform relative to direct answering. These observations suggest. that contextual distance alone does not fully account for the observed "curse". Additional limitations inherent to CoT itself must also contribute to its inefficacy.
Findings: Hypothesis is validated; however, it does not fully explain the CoT curse.
5.2 Pattern inference execution: two stages of failure
Chain-of Thou-hh in pattern-based in-context learning is commonly regarded a two-stage process: first, LLMs infer the underlying pattern rule from the provided demonstration pairs, and second, they apply this
7

---

Published in Transactions on Machine Learning Research (11/2025)
List Function
MiniSCAN
100
60
80
60
Accuracy
40
20
20
10-
GPT-4c-m
Lam433-705 Lamx11-1000
Direct
CoT
Execution
Execution
Average
Figure 4: Performance comparison of pattern inference and execution across two benchmarks (List Function and MiniSCAN) and six LLMs.
inferred pattern to generate predictions for test instances (Liu et al., 2024; Zheng al., 2025a). Given the observed deficiencies of CoT in our experiments, we propose two hypotheses to dissect the potential sources of this failure:
Hypothesis 2. LLMs struggle to infer underlying patterns from demonstrations when using CoT.
Hypothosis LLMs struggle apply inforred patterns test instances when using CoT.
To rigorously test. these hypotheses, designed two-phase experiment to independently evaluate L.M performance across both stages: pattern inference and pattern execution. For this analysis, we selected two Hatasets-List-Functee Function and MinICAAN allow for precise evaluation against ground-truth patterns. In the pattern inference stage, we assessed whether LLMs could correctly infer the underlying pattern (e.g,, a Python function or symbolic rule) from input-output pairs in demonstrations. stage, we evaluated their ability to apply the ground-truth pattern to test instances. We further evaluated "Inference & Execution' performance by collecting the inferred pattern (without format constraints) from CoT reasoning and directly providing it for pattern execution. This setup aims to further validate the consistency between the original CoT process and separate
the pattern execution
setrennc-then-exceuti mechanism.
The experimental results are depicted in Figure 4. Across both datasets, we observe that LLM performance in pattern inference consistently falls below that of pattern execution. This disparity suggests that the primary challenge for LLMs using CoT lics in accurately deducing the underlying rules from demonstration pairs. contrast, their ability execute a given pattern appears relatively stronger, though still imperfect.
On the other hand, also observe that the overall performance of 'Inference & Execution' remains significantly lower than that of the original CoT across both datasets. This pronounced disparity- with the observations above-suggests suggests that CoT often produces correct answers despite incorrect pattern inference in numerous cases (we provide further case studies in Appendix B.4 for an intuitive demonstration). In other words, rather than relying solely on explicit pattern execution for the test instance, the LLM's answer generation under CoT appears to be implicitly influenced by the in-context demonstrations, which improves the final performance. This interpretation supported by experimental results in Appendix B.5.
This challenges the simplistic assumption of CoT as strictly two-stage mechanism. Instead, these results suggest the presence of implicit reasoning mechanisms within CoT, whereby LLMs leverage latent pattern recognition and exccution -akin to the processes underlying direct answering- to compensate for shortcomings in explicit inference and execution.
Findings: Our results indicate that pattern inference (H2) constitutes a more significant bottleneck for LLMs than pattern exccution (H3): the primary failure point lies in inducing the correct rule from demonstrations. Nonetheless, our evidence also suggests that implicit mechanisms beyond explicit rule-following may also contribute substantially to CoT performance.
8

---

Published in Transactions on Machine Learning Research (11/2025)
List Function
MiniSCAN
30
Percentage Instances (96)
40-
p3:1
1/2
12.3
7.0
20
1-3
9.1
30.1
7.6
22.5
22.6
25.5
12.6
Uavadi-its Lanakinsta
Gwird 5-12b
Pustage
Laral.//0 Janakiaasss
Avstage
Explicit Success, Implicit Success
Explicit Fallure, Implicit Success
Explicit Success, Implicit Fallure
Figure 5: Decomposition of CoT success: contributions from explicit and implicit reasoning.
5.3 The Implict Hybrid Mechanism: uncovering divergent answers and rationales
Findings from the preceding analyses provide compelling empirical evidence for a new conceptualization of Coll in in-context learning: the Explicit-Implicit Hybrid Mechanism. This perspective posits that the final prediction in CoT emerges from a composite process involving both explicit pattern inference and execution (articulated through the CoT rationale) and implicit pattern recognition and execution (latent reasoning akin to direct answcring). The observed discrepancy between the poor performance of explicit pattern inference (Section 5.2) and the relatively high accuracy of CoT predictions suggests that implicit mechanisms may compensate for deficiencies in the explicit reasoning process. Building this insight, we formulate and investigate hypothosis that characterizes this dual process:
Hesos 4. In pattern-based ICL, CoT predictions can be modeled as arising from a dual process of explicit and implicit reasoning. This process appears to be asymmetric: implicit reasoning is the dominant driver successful predictions, compensating for the frequent ineffectiveness explicit reasoning.
It is challenging precisely disentangle the effects of implicit and explicit reasoning within a single CoT process. To investigate their relative contributions, we adopt analytical framework that relies operationalizing these concepts with proxies.
First, we treat direct answering (i.e., prediction without an intermediate rationale) as a functional proxy for a purely implicit reasoning process. While we acknowledge that direct answering may involve its own un-verbalized reasoning, it serves as a baseline for the model's performance without explicit rationalization. Second, we treat correct pattern inference from demonstrations as a proxy for successful explicit reasoning. We then analyze instances where the LLM with CoT solved a given problem successfully, attributing the success as follows:
Analytical Premises:
Implicit Contribution: We attribute the success on instance to implicit reasoning if the model also solves i using direct answering but fails to infer the correct pattern from the demonstrations (i.e., explicit reasoning fails).
Conversely, we attribute the success to explicit reasoning the model correctly infers the pattern but fails solve using direct answering (i.e., implicit reasoning fails).
Explicit Contribution:
The results of this analysis are illustrated in Figure 5, where instances of implicit-only and explicit-only contributions are shown in purple and yellow, respectively. Across both datasets, the percentage of cases where implicit reasoning drives CoT success is substantially higher- by 7.5x on List Function and 3.6
3using "pattern inference" alone provides conservative upper bound for the contribution of explicit reasoning, thereby strengthening our conclusion.

---

Published in Transactions on Machine Learning Research (11/2025)
Table 2: Performance comparison between direct answering of LLMs and long-CoT LRMs. *Token cost represents the weighted sum of context and inference tokens with a 0.25:1 ratio.
Average
Models
MiniARC COGS
RAVEN
Accuracy (%)
Token Cost
Qwen2.5-32B
24.16
21.80
12.07
19.34
207.44
Qwen2.5-72B
23.49
20.40
23.67
22.52
198.54
LLM (direct)
Gemini-1 1.5-pro
24.83
36.00
24.31
28.38
198.95
Llama-3. 1-405B
26.71
24.40
25.34
25.48
201.61
Deepseek- V3
27.52
30.80
21.05
26.46
189.71
QwQ-32B
18.70
13.00
8.82
13.51
1736.91
LRM (long-CoT))
ol-mini
30.20
10.60
15.25
18.68
3072.02
Deepseek-Rl
28,86
24.00
27.56
26.81
2432.36
on MiniSCAN- the converse scenario. This disparity underscores the dominance of implicit reasoning, which frequently compensates for flawed explicit pattern inference to achieve a correct final answer.
Findings: Implicit reasoning significantly outweighs explicit reasoning in contributing to CoT success, validating the asymmetric hybrid mechanism proposed in Hypothesis
Summary: Our analysis suggests a compelling explanation for CoT's underperformance in this domain: it operates as a dual-procoss mechanism, combining both explicit and implicit reasoning (Hypothesis 4), but this mechanism is fundamentally compromised. First, its explicit reasoning pathway appears unreliable, primarily because LLMs struggle to correctly infer underlying patterns from demonstrations (Hypothesis 2). Second, the very structure of CoT-the insertion of a cectonall-inccraan performance of its implicit reasoning pathway (Hypothesis 1), a pathway that otherwise appears robust. With its explicit component proving ineffective and its implicit, component hampered, CoT consistently underperforms direct answering, method that relies solely uncompromised implicit process.
contextual distance, which degrades the
6 The case of specialized reasoning models
A crucial test of our hypothesis involves Large Reasoning Models (LRMs), which are explicitly designed to excel at complex, multi-step reasoning. In pattern-based ICL, key distinction between these LRMs and traditional LLMs lies in the former's ability to iteratively propose and refine hypothesized patterns, thereby enhancing explicit pattern inference capabilities. However, this extended reasoning process significantly increases contextual distance, creating direct trade-off with the implicit reasoning pathway.
To evaluate this trade-off, we conducted experiments using three LRMs across three benchmarks, each selected from distinct task modality. As shown in Table 2, the results are striking: top-performing LLMs using direct answering achieve comparable superior performance to the specialized LRMs. This outcome is particularly notable given that the LRMs consume, average, 12x more total tokens and 40x more inference tokens.
These findings suggest that even when the explicit reasoning component is significantly enhanced, its benefits are insufficient to overcome the structural disadvantages imposed by the CoT framework in these tasks. The performance of the implicit pathway, hampered by increased contextual distance, remains a critical bottleneck. This underscores the severity of the "CoT curse" in this domain and highlights the need for strategies that integrate verbalized and latent reasoning more efficiently.
7
Related Work
Inductive Reasoning Our pattern-basod in-context learning setup aligns with inductive reasoning: models must infer an implicit pattern from demonstration pairs and apply it to a test instance. Inductive reasoning capabilities in large language models have been extensively studied, with foeus the design of reasoning
10

---

Published in Transactions on Machine Learning Research (11/2025)
frameworks and training prompting strategies that elicit such behavior (Wang et al., 2024; Qiu et al., 2024; Liu et al., 2024; Zheng et al., 2025a; Li et al., 2025). These capabilities play critical role across diverse application domains, including financial modeling (Goel et al., 2025; Stempień & Slepaczuk, 2025) and scientific discovery (Shojace et al., 2025; Zheng et al., 2025b:c).
Effectiveness of Chain-of-Thought Prompting Chain-ofTThought prompting (Wei et al., 2022) is widely recognized for improving reasoning performance. At the same time, evidence suggests that CoT offers limited gains for tasks that less reasoning-intensive such factuality calibration (Zong et al., 2025b;a; Liu et al., 2025) and semantic classification (Sprague et al., 2024). Recent work has systematically examined how CoT length affects performance (Jin et al., 2024; Wu et al., 2025; Nayab et al., 2025). In contrast, our work provides the first evidence, to our knowledge, that CoT exhibits substantial limitations on a reasoning-intensive task setting where success depends on inferring and generalizing latent patterns from demonstrations.
8 Conclusion
In this work, we identify and rigorously analyze a fundamental paradox in Chain-of- Thio-ghh- prompting: despite its success in various reasoning tasks, CoT consistently underperforms direct answering in pattern-based in-context learning-2 fundamental task that harnesses the inductive and abstract reasoning capabilitics of LLMs. Our investigation reveals that this failure stems from flawed explicit-implicit hybrid mechanism at the core of CoT's mechanism. We find that the explicit reasoning pathway, responsible for articulating rationales, is unreliable, as large language models struggle to correctly infer underlying patterns from demonstrations. Simultaneously, the very structure of CoT the insertion of these rationales increases contextual distance, degrading the performance of the otherwise robust implicit reasoning pathway that powers direct answering. With its explicit component proving ineffective and its implicit component hampered, CoT is fundamentally compromised in these tasks.
Our analysis of specialized Large Reasoning Models further validates this conclusion, demonstrating that even architectures designed for superior explicit reasoning fail to overcome these inherent limitations, performing par with or worse than direct answering despite incurring substantial computational overhead. These findings challenge the presumed universal efficacy of CoT, highlighting the critical need for more nuanced understanding that balances explicit and implicit processes. We advocate for the development of adaptive reasoning strategies that can harness the strengths of both modes, paving the way for more robust and efficient large language models.
Acknowledgement
We thank all the anonymous reviewers and editor for their valuable comments. The authors of this paper were supported by the ITSP Platform Research Project (ITS/189/2311) (/F9)2 from ITC of Hong Kong, SAR, China, and the AoE (AoE/E-601/245) the RIF (R6021-20) and the GRF (16205322) from RGC of Hong Kong, SAR, China. We also thank the support from NVIDIA AI Technology Center (NVAITC).
11

---

Published in Transactions on Machine Learning Research (11/2025)
References
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint ar Xiv:2000.. 16609, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyezyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16)17662- March 2024 ISSN 2159-5399. doi: 10.1609 www.aaaa......99971.... URL http://dx.dod org/10.11/ 111111...1111/11 16609//aai.. v38i16. 29720.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://
httg//b//20000 14165.
François Chollet. On the measure of intelligence, 2019. URL https: http:://axxvvvvoee /ttp:////rr/bb//1110111177
Google DeepMind. Google introduces Gemini 2.0: A new AI model for the agentic era, December 2024. URL wowg://ogooooon--e-eee
https://blog. http://llloo.gggee///ooooool//ee weccnoooog/gooolllle....ooooo
1etpoddtteddeemee--20000 2024/.
DeepSeek-AT, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Pciyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Yo, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yilan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Decpscek--33 technical report, 2024. URL https: //arxiv...ivvee/ http:///wwwwwwwwaabbbb httgg/bb//2112. 19437.
DeepSeck-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong
12

---

Published in Transactions on Machine Learning Research (11/2025)
Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nic, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Viyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in iims via reinforcement learning, 2025. URL attps://arxx:
arggabb//501..1. 12948.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantamu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu,, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Proquecte-Choo. Danila. Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa. Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Pluiisak-Plucifskaa, Harloen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jell Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengehong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Kranse, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phocbe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu,
13

---

Published in Transactions on Machine Learning Research (11/2025)
Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at practical size, 2024. URL http:://aaxivv.ogaab///////
00118.
Anubha Goel, Puneet Pasricha, Martin Magris, and Juho Kanniainen. Foundation time-series ai model for realized volatility forecasting, 2025. URL http:://axx...oogg/aa////55 11163.
Google. Introducing Gemini 1.5, google's next-generation AT model, February 2024. URL https://blog. httpcc/www....a.ooeeeeeeaaaaaaaa
Xiaoyang Hu, Shane Storks, Richard L. Lewis, and Joyce Chai. In-context analogical reasoning with pre-trained language models, 2023. URL attp:://arrii..or//ba//22255 wtt.:oorr/bb//...1111/ 17626.
Dave Hulbert. Using treo-of-thought prompting to boost chatgpt's reasoning, https://github
trre-thhouht---optingg. prompting, May 2023. URL
10323452.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL
06825.
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models, 2024. URL satps://arxivy org/abs/ 04925.
Najoung Kim and Tal Linzen. Cogs: A compositional ceneralization challenge based on semantic interpretation, 2020. URL https://arxa
abg/abs/ aa/aabss/0000 20100.00000 05465.
Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. Playgrounds for abstraction and reasoning. In NewrIPS 2022 Workshop on Newro Causal and Symbolic Al (nCSI), 2022. URL https:
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequenceeto-ssquenee recurrent networks, 2018. URL https://arxiv.orge wtt.aa/www...r//b////1......0000
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilé Lukošiuté, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Fthan Perez. Measuring faithfulness in Mlain-of-thought reasoning, 2023. URL https: //axxiv///xxi..o// org/abs/////eeoeooooog.oooeee a/////bb////..////// 23720
Chunyang Li, Weiqi Wang, Tianshi Zheng, and Yangqiu Song. Patterns http://///raa..rrraa////00..... 16169. inductive reasoning in lims under noisy observations, 2025. URL https: arxiv http:/aaxvvvvvgoo./ otg.
principles: The fragility of
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems, 2024. URL https://arxi: orgg/bs//202/..... 12875.
Emmy Liu, Graham Ncubig, and Jacob Andrcas. An incomplete loop: Instruction inference, instruction
following, and in-context learning language models, 2024. URL httpp://arrvvv.og/aab////////
Jiayu Liu, Qing Zong, Weiqi Wang, and Yangqiu Song. Revisiting epistemic markers in confidence estimation:
Can markers accurately reflect large language models' uncertainly?, 2025. URL https://arxiv..rg// 2505. 24778.
Meta. Introducing Llama 3.1: Our most capable models to date, July 2024. URL https: //ap::://a.. http:/aa.mmaaa...em meta.com/ 2log/neta-Ilana-3111
Mistral AT. Mistral small 3. https://iiistaaalaaaaeessssssttt 2025.
2025.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on 1lm reasoning and cost, 2025. URL
wttprii..rr//bb/200..99999 19825.
Maxwell I. Nye, Armando Solar-Tezama, Joshua B. Tenenbaum, and Brenden M. Lake. Learning compositional rules via neural program synthesis, 2020. URL
05562.
OpenAL. Introducing chatgpt. httpss//opeaal.ccoo///mdeeeeettt 2022.
OpenAL. Hello GPT-4o, 2024. URL.
OpenAL. Introducing openai ol preview, 2024.
URL https:///ppnaaa.co//ndddee
Accessed: 2025-02-14.
OpenAL.
Openai o3 mini: Pushing the frontier of cost-effective reasoning, January 2025. URL https: httpenaa.commnndee/e/eee toaddeexoppaaa-o-nniii 03-min//
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL. https://arziv: org/abs/2203. 02155.
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement, 2024. URL https://arrs org/ abb/110. 08559.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: //arxiv..//ii.ee/a www...rgggbb..... htbss/242..155 15115.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https: //tp////ww//mmmm github. httpoo//////////////////
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL https://arxins httpas///axxvv.ooab//a////s org/abs/2305 2tt.//22............
Joshna Rule. The child as hacker: Building more human-like models of learning. PhD thesis, MIT, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL attpe///arrdv.oogg/bb//1000 org/abs/170r 1063477777...7..
Parshin Shojace, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. Llm-sr: Scientific equation discovery via programming with large language models, 2025. URL https://arxiv: org/abs org/abs/2444 2404. 18400.
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? oh helps mainly on math and symbolic reasoning, 2024. URL https://aaxx:: http:///xxxvv..oaaaba/
org/abs//////ooooooooo.ooo ha////b///499/ 12183.
Kaya Stechly, Karthik Valmeckam, and Subbarao Kambhampati. Chain of thoughtlessness? analysis of cot in planning, 2025. URL
04776.
Dominik Stempieñ and Robert Ślepaczuk. Hybrid models for financial forecasting: Combining econometric, machine learning, and deep learning models, 2025. URL htttp:///arte org/abs/ 19617.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D. Goodman. Hypothesis search: Inductive reasoning with language models, 2024. URL https://arxiv. httpp://arrxtv.or//abb/////33/ wttp.//.rrg.bbb....aaa//// 05060/00000990006000.0.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thouu prompting elicits reasoning in large language models, 2022. URL org/abs/2200 11903.
Yuyang Wi, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang- When more is less: Understanding chain-of-thought length in llms, 2025. URL
http://awii..rr////////./..../// 07266.
Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B. Khalil. Lims and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations, 2024. URL
https://arxiv, org/abs/2305 18354.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023a. URL https://arziv. org/abs/2305.106 10601.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: 03629.
Synergizing reasoning and acting in language models, 2023b. URL https://arxiv.ot
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot, prompting for textual reasoning, 2022. 03400.
URL
Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey. ACM Comput. Surv., 56(12), October 2024. ISSN 0360-0300. doi: 10.1145/3664194. URL https://doi.org 1145/3664194.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Chun Zhu. Raven: A dataset for relational and analogical visual reasoning, 2019. URL httt::://arri..or//ab///tt/ orgg 02741.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv wtrggabbs/200 01068.
Xiang Zhang, Juntai Cao, Jiaqi Wei, Chenyu You, and Dujian Ding. Why does your cot prompt (not) work? theoretical analysis of prompt space complexity, its interaction with answer space during cot reasoning with llms: recurrent perspective, 2025. URL https: /ttp:///axvvvv// wwerii..rr//aee org htg/abs/25505 10084.
Tianshi Zheng, Jiaxin Bai, Yichong Wang, Tianqing Fang, Yue Guo, Yanwai Yim, and Yangqiu Song. Clr-fact: Evaluating the complex logical reasoning capability of large language models over factual knowledge, 2024. 20564.
URL https: www.awwwww..aa...eeeeo
Tianshi Zheng, Jiayang Cheng, Chunyang Li, Haochen Shi, Zihao Wang, Jiaxin Bai, Yangqiu Song, Ginny Y. Wong, and Simon See. Logidynamics: Unraveling the dynamics of logical inference large language model reasoning, 2025a. URL https://arxiv. http:/aaiav.orrraaaaasasaaes
11176.
Tianshi Zheng, Zhoye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang, and Yangqiu Song. From automation to autonomy: A survey on large language models scientific discovery, 2025b. URL
http://aaxiiv.oo/aab/////// 13259.
Tianshi Zheng, Kelvin Kiu- Wai Tam, Newt Huc-Nam K. Nguyen, Baixuan Xu, Zhaowei Wang, Jiayang Cheng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. Newtonbench: Benchmarking generalizable scientific law discovery in Ilm agents, 2025c. URL
attps://arxiv.or www.vv.orggabba//11111 221100072..
Qing Zong, Jiayu Liu, Tianshi Zheng, Chunyang Li, Baixuan Xu, Haochen Shi, Weiqi Wang, Zhaowei Wang, Chunkit Chan, and Yangqiu Song. Critical: Can critique help Ilm uncertainty or confidence calibration?, 2025a. URL pastps://arxer www.aawwwww...////////a...... 24505.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Qing Zong, Zhaowei Wang, Tianshi Zheng, Xiyu Ren, and Yangqiu Song. Comparisonqa: Evaluating factuality robustness of llms through knowledge frequency control and uncertainty, 2025b. URL https: aarrav..rg//bb//2222200333
17

---

Published in Transactions on Machine Learning Research (11/2025)
A Model details
In our experiment, we tosted 20 modern LLM/LRMs. All experiments with temperature set to zero.
Deepseek- V3 (671B) (DeepSeek-Al al., 2024) is a state-of-the-art open-source LLM released by Deepseek.
Deepsock-R1 (671B) (DeepSeck-AI et al., 2025) a leading open-source LRM trained with reinforcement learning using a rule-based reward system.
Gemma-2-9B Gemma-2-227 (Gemma Team et al., 2024) is an open-source, lightweight yet bigh--peffrrmace LLM series.
Llama-3.1-8B / Llama-3.1-70B / Llama-3.1-405B (Mela, 2024) is open-source dense model series incorporating Direct Preference Optimization (DPO) (Rafailov et al., 2024) for alignment.
Mistral-7B Instruct v0.3 (Jiang et al., 2023) an early high-performance and lightweight, open-source LLM.
Mistral Small 3 (24B) (Mistral AI, 2025) is the latest high-performant open-source LLM from Mistral, designed for efficiency.
Qwen-2.5-7B / Qwen-2.5-32B Qwen-2.5-72B (Qwen et al., 2025) is open-source MoE LLM series pre-trained on 18 trillion tokens and fine-tuned with million examples.
QwQ-32B (Qwen Team, 2025) is a reasoning-focusee LLM trained via reinforcement learning, achieving competitive performance through scalable RL and agent-integrated reasoning for tool and environmental adaptation.
Gemini-1.5-flash / Gemini-1.5-pro (Google, 2024) is a proprietary MoE LLM series optimized for processing ultra-long sequences.
Gemini-2.0-flast (DeepMind, 2024) is the latest proprietary LLM in the Gemini series, featuring enhanced multimodal understanding and reasoning capabilities.
GPT-3.5-turbo (OpenAI, 2022) is proprictary conversational LLM fine-tuned via RLHF (Ouyang et al., 2022) and PPO (Schulman et al., 2017).
GPT-4o-mini GPT-4o (OpenAI, 2024) proprietary multimodal LLM from OpenAl with enhanced reasoning capabilities.
o1-mini (OpenAI, 2024) is a proprietary LRM from OpenAl utilizing reinforcement learning for inference-time scaling.
18

---

Published in Transactions on Machine Learning Research (11/2025)
B
Experiment details
We here provide detailed information of our four tailored experiment to investigate the underlying cause of CoT's ineffectiveness in ICL.
B.1 Dummy rationale experiment
B.1.1 Prompt instruction
We aim to have LLMs output dummmy rationales under controlled modalities isolate the semantic content. of CoT while maintaining contextual distance. The main challenge lies in controlling LLMs to produce outputs of a specific length in symbols text while minimizing content variance and preventing unbounded outputs. To address this, we instructed LLMs to generate countdown list from specified value to one to recite selected excerpts from Shakespeare's sonnets. In the Shakespeare dummy rationale generation, LLMs occasionally produced minor errors in precise wording, but outputs were consistently controlled to exactly contrast, during countdown list generation, some 'smarter' LLMs
14 lines (approximately 150 tokens). refused to produce the full list when the starting number exceeded 200. This behavior occurred only with the textual modality dataset (COGS); for the other two symbolic/numerrcal numericall datasets, the generation performed well. Overall, these minor divergences in both experiments did not impact the experimental findings, which indicate that increase in contextual distance reliably degrades ICL performance.
The prompt instructions for our dummy rationale experiments are provided below:
Prompt Templates
Shakespeare
cregular question instructions and data?
generating your ansver, you
first sonnet(s) Shakeapeare's
output strictly follow jaon
"recitation"; "your recitation", "ansver": "your ansver"
Count Down
<regular question instructions and data>
Before generating your you first count dosm from ((n, 1)).
output strictly follow joon
"countdown': your countdokn liet, 'ansver"; "your ansver"
19

---

Published in Transactions on Machine Learning Research (11/2025)
B.1.2 In-prompt vs. In-response dummy rationale
our dummy rationale experiment (Section 5.1), we primarily adopt in-response dummy rationales, as this approach preserves the relative positioning between the chat template and the rationale, consistent with standard CoT prompting, However, an alternative format involving in-prompt dummy rationales has also been proposed (Lanham et al., 2023). To evaluate the robustness of our findings, conducted additional experiments comparing these two formats the MiniARC and COGS datasels. As shown in Tables 3 and 4, both in-response and in-prompt approaches yield similar performance trends: LLM accuracy generally declines as the contextual distance (i.e., rationale length) increases, providing consistent evidence supporting the contextual distance curse (Hypothesis 1). The minor variations between formats do not alter the overall conclusion, though in-response rationales occasionally show slightly higher baseline performance, likely due to their alignment with default prompting structures.
Table 3: Performance comparison (accuracy in %) of in-response vs. in-prompt dummy rationales across varying rationale lengths on the MiniARC dataset, including the baseline of direct answering (contextual distance=0). Results averaged across six LLMss
Approach
Direct (Dist.=0)
Count Down (100)
Count Down (400)
Shakespeare (1)
Shakespeare (4)
Response 16.89
14.77 (2212))
13.65 (-3.0))
14.77 (-2.2)
12.98 (-a.a1)
In Prompt 16.89
13.98 (-2.11)
12.42 (-1.7))
13.98 (-2.1)
13.42 (-3.97)
Table 4: Performance comparison (accuracy in %) of in-response vs. in-prompt dummy rationales across varying rationale lengths the COGS dataset, including the baseline of direct answering (contextual distance=0). Results averaged across six LLMs.
Approach
Direct (Dist.=0)
Count Down (100)
Count Down (400)
Shakespeare (1)
Shakespeare (4)
In Response 16.55
16.24 (0:31)
14.90 (-160))
15.77 (-0.78)
13.76 (-2.19)
In Prompt
16.55
14.77 (-1788)
12.42 (-4.13)
13.98 (-2.57)
13.42 (-a 13)
20

---

Published in Transactions on Machine Learning Research (11/2025)
B.2 Rationale frontloading experiment
Another dimension of controlling the contextual distance effect involves relaining the semantic content of the CoT rationale while eliminating the contextual distance between demonstrations and the final answer (as in direct answering). To achiove this, we collect CoT rationales from the same models and prepend them to the in-context demonstrations of the question. Subsequently, we feed the combined input (question + inserted rationale) the same L.LSs for direct answering. However, we observed that CoT rationales sometimes already contain concluded answers. To include only the reasoning steps, utilized GPTT-4o-mini process the rationales effectively with two-shot demonstrations, removing the final concluded answer while preserving the entire reasoning process.
The prompt instructions for rationale processing are provided below:
Prompt Templates
Rationale Processing
Demol
Ueer: <instruction> <CoT rationale with answer
Assistant: sprocessed CoT ratienale 1>
User:
cinstruction> <Col rationale with ansker 2>
Assistant: sprocassed
Ueer: <instruction> <Col rationale to process?
Assistant:
Rationale Frontloading
cregular question instruction>
<processed CoT rationales>
cia-context dencustratione>
cregular answer instruction and test input (direct anexering)>
21

---

Published in Transactions on Machine Learning Research (11/2025)
B.3 Pattern inference and execution experiment
The primary objective of this experiment is to disentangle the two stages of reasoning in CoT: pattern inference from in-context demonstrations and pattern execution on test inputs. The prompt instructions for both experiments are provided below:
Prompt Templates
Pattern Inference
cregular task description> cin-contert demenstrations>
Nox, please perform reasoning to underlying pattern (python function) napping inputs outguts.
Your output ahould strictly follow the jaon diet format belou:
'reasoning': "your reasoning steps", "pattern": "your pattarn (function)"
Pattern Execution
Cregular taak description>
imput>
'ground-truth pattern>
Now, please perforn reasoning to apply the above ground-truth pattern to transform the input into output.
Your output ahould strictly follow jaon dict fornat balou:
'reasoning": "your ressoning stepe", *outgut"; "your output"
22

---

Published in Transactions on Machine Learning Research (11/2025)
LLaMA-3.1-70B
LLaMA-3.1-405B
Qwen-2.5-7B
Total
Incumax connect
Ireamec Cones:
1200
225
10
235
Human Eval
185
Human
Correct
1269
36
1000
140
114
189
Human Evaluation
(37.01%)
(1.62%)
800
LUM Eval
Eval
2--22
GPT-4o-mini
GPT-4o
600
Human Fool coness
Incorrect
220
Human Eval
26
895
400
(1.17%)
(40.21 (40.21%)
Homan
incumee
200
139
173
Conrect
Cornect
Incornt
Comwtt
Incomett
Correct
Incorrect
LUM Eval
LIM Eval
LLM Eval
LLM Evaluation
Figure 6: Alignment of human evaluation and LLM evaluation inferred patterns.
The evaluation of inferred patterns in List Function and MiniSCAN is conducted using different approaches. For List Function, directly execute Python functions generated by LLMs all available input data and compare the program outputs with the corresponding ground-truth outputs. Over 95% of the LLM-generated programs successfully compile. For MiniSCAN, the generated rules are expressed as textual descriptions, which makes automated programmatic evaluation challenging. Therefore, we employ Qwen-2.5-72B to assess the correctnoss of the inferred rules. To evaluate the robustness this LLM-based asscssment, we also conducted human evaluation of rationales, demonstrating strong alignment between the results of LLM evaluation and human evaluation (97.22% total agreement, shown in Figure 6). Consequently, our evaluation of inferred patterns is deemed reliable.
Below shows example of the LLM pattern evaluation prompt for Qwen-2.5-72b:
Prompt Templates
Pattern Eraluation
You are tasked with judging sequence-to-sqquaeee problen.
series of input and output sequences, and sixa to deduce the rules or vord nappings that connect then.
is
In this scenario, each word in the input sequence can either:
1. directly to word in the output sequence (vord napping).
2. Represent rule constructing the output sequence.
Possible Repeat the forner part three tines: Exanple: If the input "tap thri" "thri" represents rule, the output should "top top tap."
Constructing Output Sequence
Swap the forner with the latter:
Example: If the input "tmpi tap2" and "sv" represents this output should "tap2 tmpl."
one
Example: If the input is "tapl pd tap2" and "pd" represents this rula, the output ahould be "tapl tmp2 tmpl."
Your Task
will provided with the rules word mappings the parson deduced.
Your objective is to evaluate whether the person correctly deduced these rules nappings. deduced rule only indicates
correspends napping vithout spacifying shat the mile 1a, it ahould be deened incorract.
rule
23

---

Published in Transactions on Machine Learning Research (11/2025)
B.4 Decomposition of explicit and implicit reasoning
The inconsisteney between overall CoT performance and the combined performance of explicit pattern inference and execution suggests that an implicit reasoning mechanism may also exist in the latent space of LLMs, despite the use of CoT. In Figure 5, we investigate, across all questions where CoT succoeds, how many questions the LLM fails to infer the correct pattern, while purely implicit reasoning (direct answering) succeeds, as well as the opposite case. These numbers provide intuitive yet practical estimates of the proportions of questions which explicit and implicit reasoning contribute in CoT's success proportions that are infeasible to evaluate at scale. For better understanding of this hybrid mechanism, we present case study? for both datasets where CoT succeeds despite incorrect reasoning and inferred patterns. These serve as strong evidence of the contribution of latent reasoning in CoT to pattern-based ICL.
4For List Function, required LLM generate the python function between CoT reasoning steps and final answer output. In regular evaluation, python functions not explicitly required.
24

---

Published in Transactions on Machine Learning Research (11/2025)
Case Study (List Function)
List Function Prompt
Belov several examples input and output lists.
There exint a function that naps the inpat list to the output list.
Inguti: [61, 18, 0, 41, O, 89] Outputi: [41, 0, 89)
Input2: [43, 47, 41, 83, 9, 22] Outpat2: [22]
Input3: [20, 89, 53, 731 Dutput3: [63, 73]
Now, based the imput list, please perforn ressoning and provide the output list.
Input:
output. provide python that inpat output
Note that the python function should input 1ist and return list.
'reasoning": "your reasoning process", 'function"; "only the python function hare", "output": your output
Ground Truth
Python Function:
def func (x)
alenenta the first return x[x, (0)+1()
Ansser: [63, 40, 93]
LLM Output (L1ama-3.1-44) Instruct Turbo)
Reasoning:
(incorrect)
The function appaars to be filtering out the numbera in the input 1ist that are greater than the number that follows iterating through greater than the next number, it includes the next number in the output list. If the next number is 0, it skips the non-zero number.
right,
encountera
continues
Punction: (execution output: (89, 21, 53, 40))
def filter, filtor_listtinge list
output_ []
1 0
(anpina list)
if iaput_list(i] input_liat(i list[1 1]:
liput_list(i 1] 0:
1))
alse:
While j lem and input_liet[[]] -- o:
if len (intput_liet))) list):
output_liet. apt.::ieet..pppaaaeee. (pepppp
1 += output_liat (correct)
Output:
(53, 40, 98]
25

---

Published in Transactions on Machine Learning Research (11/2025)
Case Study (MiniSCAN)
MiniSCAN Prompt
Torn in task Your tast is to convert as teper. seguene into an ourpes reguenee based or spacitic reles. Back ueed in the tegert exquonta wtther corneggonde no direes transforaation fass uond to the oraget aeguena,, vale that, doftnng her the ourges sngueece should be structured.
Bolow are now excepleo
Inpeti: actween Ocspoti: blaastiy
inpet2: systsusk Detput2:
inpet3: syme 0ctpet3: breastly
Trgete:
Intgest: glatark
zystant? gisently Outpott:
oyntesk willy syaeиs
Hlur
so01ly
blerntly
lnpett: syttenk thricks
OutputS: avenstly fiom
lnpet30: schous thricka apnisusk Datpe:i0: flamr Vieeatly
Inpetil: systess g Eantly cartske synsees Intponll i: kyanely flass flaw flanz
Inpet synaars thricks achenar. moilly systenee Deegeti2: blasetly fisw bieently syanntl
Inpeti3: systeue thrtace syssace gisessly 2utpoti3; kysntiy tysently syssotly flass
Inputté: sjnear ailly soleren ttriels syaienss 0esjettó: flacs lysessly bleestly hyseotly
loz, bated on the input sequntes, please geselde sotpot requssos.
trpet: threts swilly
thrinte thrade glanttl culpal should way virictly follow jasa
you uueer nut vithis ouily besces:
"reasoning"; octput
Ground Truth
Bequiced Balea:
"zoully": "Put the
forerr'"
"strretaa:
"Seap
fermer lnttar"
LLM Output (Qwen-2.5-77) 5-72B Instruct Turbo)
Reanoningi
cotguti "gloiark glotark glotark glasark gletarit)
the dollowing trassomation:: "sciarua" 'blesstiy', 'syniszak" +> "flser", exuppes acorided, "gastark',
inguerlly.
'systerat" 'glanntly' prwetors
placeholder leplytng change topor seilly
glaestly::
affects
"tizede" i> 'gloiark'
"tizieks' >> "tlaze' (az follows 'thzoda')
"tizedy" > 'gloiuck'
5. 'glaently' chinae the grartoce werd 'gletarte' to regest three steee.
Dotput: (curvec))
'glotare
gletark glatare"
26

---

Published in Transactions on Machine Learning Research (11/2025)
B.5 Pattern execution with in-context demonstrations
During pattern execution in the slandard CoT setting, LLMs can not only deductively apply the inferred pattern to the test instance but also be implicitly influenced by the in-context demonstrations. However, these demonstrations are absent in the exccution step of the 'Inferonce Exccution' setting. To better demonstrate this impact, we conducted an additional experiment that incorporates in-context demonstrations into the pattern execution step. The experimental results from Table 5 and Table 6 show that performance significantly improves after incorporating in-context demonstrations into the deductive execution step, even though they not directly required in the task setting. These findings serve key evidence of CoT's reliance on implicit reasoning from demonstrations, in addition to explicit rationales.
Table 5: Experimental Results of 'Inference & Exccution' with in-context demonstrations provided in the pattern execution step, in dataset MiniSCAN.
Inference & Execution (execution with demos)
Model
CoT
Inference & Execution
Gii--4-mini
4.10
1.35
2.81 (+1.46)
GPT-4o
20.60
11.24
16.97 (45.75)
Llama-3.1- 1-70b
19.80
8.89
14.49 (45.60)
Llama-3.14400 1-405b
32.00
16.74
29.23 (4-2249)
Qwen-2.5-7h
1.42
1.21
1.92 (40.71)
Qwen-2.5-72b
22.00
13.91
16.32 (+2.11)
Average
16.65
8.89
13.62 (44.7))
Table 6: Experimental Results of 'Inference & Execution' with in-context demonstrations provided in the pattern execution step, in dataset List Function.
Inference & Execution (execution with demos)
Model
CoT
Inference & Execution
GPT-4o-mini
36.88
31.00
37.81 (46.81)
GPT-40
48.72
43.81
47.50 (43.69)
1--00
37.92
31.35
37.92 (46.57)
Alama-3.1-405)
47.76
36.40
40.76 (+1.0))
Qwen-2.5-7h
29.36
17.27
19.68 (42.41)
Dwve-22.5-72b
45.04
37.99
52.01 (114.02)
Average
40.95
32.97
39.28 (46.31)
27

---

Published in Transactions on Machine Learning Research (11/2025)
C
Dataset details
Among the nine datasets in our experiment, three were not originally designed for in-context learning in natural language processing. Here, we provide further details on the data processing:
The original COGS dataset (Kim & Linzen, 2020) evaluates the compositional generalization of machine learning models through task that introduces compositional distribution shifts in input-output mappings. In this study, we use the test dataset, sampling 10 entries as in-context. demonstrations.
COGS:
List Function: The original work designed these functions to investigate the human-like learning abilities of cognitive systems (Rule, 2020). Subsequent studies have explored LLMs' capabilities in rule induction (pattern inference) (Qiu et al., 2024; Li et al., 2025) and in-context learning (output prediction) (Zheng et al., 2025a). In this work, we adopt the processed dataset from Li et al. (2025).
RAVEN: The original RAVEN dataset (Zhang et al., 2019) assesses the analogical reasoning abilities of visual models using images of symbols. We adopt the abstracted Im-RAVEN dataset (Hu et al., 2023), which tokenizes image altributes into symbolic matrices.
28

---

Published in Transactions on Machine Learning Research (11/2025)
D Prompt templates
In this section, we include our prompt templates as follows: prompt for datasct-specific instructions, prompt for reasoning frameworks, and prompt used in our tailored experiment.
D.1 Prompt for dataset- instructions
Prompt Templates
ARC-AGI
MiniARC / 1D-ARC COGS
Belov are several examples imput and output grids/lists.
Thare exists underlying pattern/facction that naps the imput grid/list the output grid/liat.
cin-centext
predict output grid/list on naa input grid/list:
Stest imput>
SCAN
Belov several exemples that convert natural language connands into action
cin-contert demenstrations>
is predict output sequence
input sequence:
imput>
MiniSCAMf
Here is a task:Your task is convert an imput sequence into output sequence based on specific rules. input sequenca corresponds to transtormation or rule that definea how the output sequence should structured.
output saquence,
cin-context denenstrations>
predict output sequence
input sequence:
Ctest imput>
SALT
Balov saveral examples Each vord in the input sequence either corresponds to a translated word in the output, syntactic rule (e.g., repeating reordering units) forning output sequence.
english
output sequence based spacific rules.
<in-context denonatrations>
Your task is to predict the output sequeace based on the new english sentence:
<teat imput>
List Function
Belov are several examplea Input and output lists. There existe underlying python function that the imput list the output list.
cin-contert demonstrations>
ie predict output 1ist
imput list:
<reat Imput>
RAVEN
Belov are several rows of abstracted cymbols. The synbols follox certain rula or pattern.
<in-context demonstratione>
Your task is predict the missing synbol based the incomplete
input>
29

---

Published in Transactions on Machine Learning Research (11/2025)
D.2 Prompt for reasoning frameworks
For fair comparison of reasoning frameworks against vanilla zero-shot CoT and direct answering, adopt one-off prompting approach rather than complex agent framework. For Tree-of-Thougeee we use the prompt proposed by Hulbert (2023). For ReAct, we employ the prompt from Qwen's implementation (Bai et al., 2023).
Prompt Templates
Direct Answering
cregular quastion instructions and data>
Please output your final ansver in the following joon dict format vithout explanation:
"ansver"; "your ansver"
Chain-of -Though
cregular queation instructions
perforn reasoning outpat your ansver following jaon
'reasoning": reasoning process",
'ansver"; "your ansver"
ReAct
should
question using folloving pipeline:
Question: the input question you must anaver
Thought: Think about shat to do
Action: Your action process
Ubservation: result the
(this Thought/Attiion/useeeeee
can be repeated zero or more tines)
Thought:
Pinal Ansver: the final ansver to the original imput question
cregular question instructions and data>
respond the folloving json
"pracess": "your full problen-solving process", "anaver"; "your final ansver*
Tree-of-Thoughe
Inagine different experta ansvering question.
All experts will write down 1 step of their thinking.
group.
Then all experts will on the next stop, etc. If any expart realisea they're urong at any point then they leave.
cregular questien instructions and data>
You ghould respond in the folloving jeen dict format:
"diacussion"; "full diacussion reasoning proceas of experta",
"ensver"; "final agreed ansver"
30

---

Published in Transactions on Machine Learning Research (11/2025)
E
Full results
The detailed LLM performances on ICL benchmarks are presented in tables below:
Table ARC-AGI
Table 8: MiniARC
Table 9: 1D-ARC
Table 10: SCAN
Table 11: MiniSCAN
Table 12: COGS
Table 13: SALT
Table 14: List Function
Table 15: RAVEN
Table 7: Detailed LLM Performances on ARC-AGI.
Direct
CoT
ReAct
ToT
Model
Ace (%)
Acc (%)
tokens
(%)
# takens
Ace (%)
# tokens
Open-source
Deepscek-V3
15.93
12.81 (33.12)
800.33
11.50 (-4.43)
719.48
11.98 (-3.98)
1055.27
Llama3.1-2 1-8B
3.95
2.75 (-1-20)
1324.51
2.28 (-1.61)
1735.61
2.16 (-1.79)
2125.06
Llama3. 1-70B
10.66
8.02 (-2.44)
645.38
10.25 (-0.41)
681.89
7.66 (-3.00)
1477.78
TJama3. 1-405B
16.45
10.42 (-6.03)
699.35
11.86 (-4.58)
1434.73
8.54 (-7.91)
1147.96
Qwen2.5-7B
4.31
1.92 (-2.39)
1681.24
0.96 (-3.35)
1841.80
1.32 (-2.99) 99)
1988.00 00
Qwen2.5-728
11.98
11.14 (-0.84)
1021.80
1.80 (-10.18)
1094.05
7.90 (-4.08)
1430.62
Mistral-7BB 7B
0.36
0.48 (10.11)
672.39
0.96 (10.00)
758.50
0.48 (10.12)
902.04
Mistral-Small 3
10.30
5.99 (-4.33)
1768.28
0.72 (-9.88)
409.83
5.15 (-5.18)
1619.02
Proprictary
Remini-115-flaas
11.26
7.90 (-3.36)
727.77
12.33 (+1.07)
872.89
13. (+1.85)
930.87
Gemini--L.5ppm
17.25
13.41
(-3.84)
787.59
4.08 (-18.17)
1080.24
15.15 (22.10)
840.94
Gemini-22.0-fast
14.25
10.06
(-4.19)
867.74
11.67 (-2.58)
1005.52
9.34 (-4.91)
3645.00
GPT-3.5-turbo
4.09
4.55 (+0.16)
459.40
3.29 (-0.80)
213.74
2.44 (-1.65)
255.09
GPT-4o-mini
5.51
5.15 (-0.30
632.05
4.01 (-1.50)
754.46
3.71 (-1.80)
840.72
GPT-40
13.77
10.42 (-8.86)
708.77
11.55 (-2.22)
777.92
8.98 (-4.79)
1019.13
Average
10.01
7.50 (-2.81)
91-.004
6.23 (-3.78)
955.76
6.99 (-3.02)
1376.96
31

---

Published in Transactions on Machine Learning Research (11/2025)
Table 8: Detailed LLM Porformances on MiniARC.
Direct
CoT
ReAct
ToT
Model
(%)
Ace (%)
# tokens
Acc (%)
# tokens
Acc (%)
# tokens
Open-source
Deepseek-V3
27.52
15.44
(-12.88)
710.46
15.44 (-12.88)
789.48
14.77 (-12:75)
395.23
Gemmaa-9B
11.41
3.36 (-8.05)
194.93
1.34 (-10.07)
435.62
2.68 (-8.73)
138.50
Gemma2-27B
14.09
11.41
(-2.68)
142.81
3.36 (-10.73)
332.99
6.71 (-7:99)
114.46
Llama3. 1-8B
10.74
6.71 (-4.03)
566.13
4.70 (66.04)
1537.8 84
3.36 (:7.8)
323.08
Llama3. 1-70B
21.48
14.09 (-7.99)
173.00
13.42 (-8.08)
816.99
8.72 (-12.70)
126.50
Llama3. 1-405B
26.71
16.11 (-10.60)
522.15
15.44 (-11.27)
645. 18
14.09 (-12.02)
301.58
Qwen2.5-7B
10.07
2.01 (-8.66)
587.68
0.00
(-10.07)
811.81
4.03 (-6.09)
335.85
Qwen2.5-72B
23.49
7.38 (-16.11)
592.43
0.67 (-22.82)
881.40
10.07 (-13-42)
336.21
Mistral-7B
2.68
0.67 (:2.01)
173.39
2.68 (0.00)
356.87
0.67 (-2.01)
108.88
Mistral-Small 3
16.11
6.71 (-9.40)
805.96
2.01
(-14.10)
750.36
9.40 (-6.71)
445.04
Proprictary
Geminni11.5-flas
16.11
14.09 (:2.02)
278.79
10.74
(65.37)
592.56
10.07 (-6.04)
181.90
Gemini-1.5-po
24.83
21.48 (-3.55)
626.63
15.44
(-9.99)
460.40
16.78 (-5.05)
205.33
Gemini-2.0-faar
25.50
14.09 (-11.41)
626.63
18.12
(-7.38)
1048.62
11.41 (-14.09)
246.71
G1-T-3.5-turbo
7.38
5.37 (-2.01)
153.30
4.70
(-2.68)
193.79
6.04 (-1.84)
116.19
GPT-1o-mini
13.42
10.74 (-2.68)
252.89
11.41
(22.01)
407.54
8.05 (-5.37)
165.94
GPT-4o
22.15
16.11
(-6.04)
308.75
19.61
(-2.54)
552.40
14.77 (-7.58)
194.06
Average
17.11
10.36 (-6.75)
419.75
8.69 (-6.42)
663.37
8.85 (-8.26)
233.47
Table 9: Detailed LLM Performances on ID-ARC.
Direct
CoT
ReAct
ToT
Model
Ace (%)
Acc (%)
# tokens
Ace (%)
# tokens
Ace (%)
# lokens
Open-source
Deepseck-V3
69.70
66.26
(-3.44)
723.92
66.93 (.2.7))
775.21
55.38
(-14.92)
733.13
Gemma2-9R
28.30
20.20
(-8110)
171.04
11.54 (-16.70)
239.97
11.10 (17.20)
357.99
Gemma2-27BB
41.62
31.08
(-10.51)
141.58
22.42
(-19.20)
212.37
23.75 (117877)
287.10
Llama3. 1-8B
23.20
13.21
(-9.89)
484.3
12.32
(-10.88)
873.66
10.65 (-12.55)
1349.24
Llama3. 1-70B
53.89
46.00
(-7.89)
165.14
40.51
(-18.88)
301.79
33.96 (-19.98)
697.90
Llama3.1 1-405B
60.60
58.49
(-2.11)
434.75
55.83
(.4.77)
665.89
44.28 (-16.32)
677 26
Qwen2.5-7B
25.86
13.67
(-12.19)
445.33
11.76
(-14.10)
506.17
3.88 (-21.96)
705.43
Qwen2.5-72B
51.67
48.22 (-3.45)
294.11
13.32
(-38.35)
353.90
36.40 (-18/21)
914.23
Mistral-7B
0.00
1.00 (-1.00)
193.50
1.22
(+1.98)
334.70
1.11 (+1.11)
341.51
Mistral-Small 3
47.50
32.74
(144776)
870.63
0.00
(-47.50)
480.67
29.74
(117.76)
694.24
Proprictory
Gemini-1.5-flssh
53.27
39.84
(-1345)
231.44
40.51
(-12.76)
466.01
34.07
(-19-20)
538.92
Gemini-1.5-pro
67.04
58.71
(-.33)
269.24
56.27
(-10.77)
420.13
48.95
(-18.09)
388.57
Gemini-2.0-fil
60.38
50.94
(-9.44)
644.63
48,83
(-11.68)
452.14 14
45.51
(-14.87)
748.79
GPT-335-turbee
8.66
14.43
(+6.77)
174.62
15.32
(46666)
142.77
11.43
(12.77)
183.69
GPT--o-mini
26.53
19.64 (-6.9)
227.64
17.76
379.46
17.43 (-9.10)
401.00
GPTT-0o
42.51
44.40
(+1.89)
281.26
41.62
(-0.80)
370.70
38.40
(-4.11)
496.21
Average
41.30
34.93 (66.37)
359.57
28.51 (-12.79)
435.97
27.88
(-13.99)
594.70
32

---

Published in Transactions on Machine Learning Research (11/2025)
Table 10: Detailed LLM Performances on SCAN.
Direct
CoT
ReAct
ToT
Model
Acc (%)
Ace (%)
# tokens
Acc (%)
tokens
Acc (%)
# tokens
Open-source
Deepseek-V3
91.85
81.70
(-10.15)
168.76
81.90
(-9.95)
248.69
78.00 (-13.85)
225.57
Gemmaa-9B
32.60
38.60
(+8.00)
85.34
36.30
(43.70)
169.07
28.17 (-4.45)
352.10
Gemma2-27B
60.86
53.70 (-716)
102.49
55.30
(-0.60)
242.08
46.46 (-14:40)
430.27
Llama3. 1-8B
24.92
23.79 (1.13)
142.52
14.19
(-10.73)
659.83
14.41 (-10.51)
1003.49
Llama3. 1-70B
68.38
60.26 (-8.12)
233.15
55.40
(-12.98)
525.03
53.70 (-14.08)
1080.48
Llama3. 1-405B
84.42
81.70 (-2.72)
154.70
86,00
(41.68)
336 10
79.20 (-5:22)
646.67
Qwen2.5-7B
11.32
33.53 (-7.79)
115.28
31.89
(-9-13)
174.10
31.16 (-10.16)
300.71
Qwen2.5-72B
88.05
88.55 (+0.50)
99.52
89.48
(+1.43)
201.55
87.00
180.86
Mistral-7B
21.21
17.07 (44.14)
122.67
20.90 (-0.31)
163.15
11.01 (-10.20)
270.43
Mistral-Small 3
70.90
67.75
(43.15)
156.90
52.10
335.95
43.89
(-27.01)
500.01
Proprictary
Geminni-1.5-llas
69.30
59.80 (-9.00)
113.15
56.60
(-12.70)
236.42
46.50
(-22.80)
342.88
Gemini-1.5-pa
93.10
87.20
(-5.80)
155.83
87.00
(66.10)
225. 16
71.00
(:22
228 23
Gemini-2.0-faar
86.30
86.20
(-0.10)
161.69
81.30
(-6.00)
355.34
76.70
(-0.60)
951.81
GP-T-3,5-turbo
37.60
41.00
(:40)
77.67
28.50
(-9.10)
73.53
22.80
(-14.80)
198 80
GPT-1o-mini
51.00
57.40
(+8.10)
115.21
57.20
(+6.20)
162.19
48.20 (-2.80)
263.84
GPT-4o
82.90
82.40 (-0.50)
147.24
83.60
(+0.70)
211.43
82.70
(-0.20)
310.11
Average
62.79
60.04 (-2.79)
134.51
57.35 (.5.44)
270.16
51.31 611.48)
455.39
Table 11: Detailed LLM Performances on MiniSCAN.
Direct
CoT
ReAct
ToT
Model
Ace (%)
Acc (%)
# tokens
Ace (%)
# tokens
Ace (%)
# lokens
Open-source
Deepseek-V3
16.50
16.30
(-0.20)
252.12
18.80 (42:80)
285.60
20.80 (+4.30)
258.93
Gemma2-9R
9.50
3.01
(-6.49)
149.56
1.00 (-8.50)
190.38
0.80 (-8.70)
403.07
Gemma2-27BB
16.40
13.50
(-2.80)
107.12
8.10 (-8.00)
239.84
7.90 (-8.50)
485.17
Llama3. 1-8B
1.10
2.30 (-1.20)
289.07
0.47 (-0.63)
737.72
2.60 (+1
1226.43
Llama3. 1-70B
27.00
19.80 (-7.20)
365.74
17.71
(-9.20)
345.85
16.70 (-10.30)
1067 .96
Llama3.1 1-405B
33.30
32.00 (-1.80)
383.64
35,00
(+1.70
385.51
24.60 (-8.70)
741.70
Qwen2.5-7B
2.80
1.42 (-1.88)
146.22
1.34 (-1.66)
182.02
3.77 (40.97)
881.11
Qwen2.5-72B
20.00
22.00 (42.000)
204.60
20.70 (40.70)
261.85
19.50 (-0.80)
316.75
Mistral-7B
0.20
1.00 (-0.80)
182.66
1.13 (+0.933
198.82
0.58 (+0.28)
348.98
Mistral-Small 3
29.63
28.30
(11.33)
385.82
23.50
(-6.13)
297.09
30.59
575.90
Proprictory
Gemini-1.5-flssh
35.40
32.10 (-3.20)
233.42
28.00 (-7.10)
523.10
23.00
(-12.400
573.68
Gemini-1.5-pro
46.80
45.75
(-1.05)
299.
42.60 (-1.20)
523.00
41.00
(-5.80)
258.07
Gemini-2.0-fH
47.30
31.30
(-16.00)
329.51
31.20
(-16.20)
523.10
32.10
(-15.20)
745.29
GPT-335-turbee
6.10
1.40
(-4.70)
98.67
0.30
(-6.80)
103.16
1.50 (-4.00)
261.70
GPT--o-mini
6.30
4.10 (-2.20)
160.05
2.80 (-3.20)
208.32
1.60 (-4.70)
329.40
GPTT-0o
35.80
20.60
(-18.20)
252.39
18.90
(-16.90)
276.85
19.60
(-16.20)
399.70
Average
20.88
17.18 (40.70)
239.99
15.72 (46116)
330.14
15.42 (-0.46)
554.62
33

---

Published in Transactions on Machine Learning Research (11/2025)
Table 12: Detailed LLM Porformances on COGS.
Direct
CoT
ReAct
ToT
Model
(%)
Ace (%)
# tokens
Acc (%)
# tokens
Acc (%)
# tokens
Open-source
Deepseek-V3
30.80
14.00 (-16.80)
405.03
27.80 (-3.00)
375.88
15.10 (-15.70)
729.94
Gemmaa-9B
9.10
8.50 (-0.80)
309.71
6.70 (-2.40)
223. 18
1.30 (-7.00)
328.61
Gemma2-27B
21.30
14.10 (-7.20)
132.83
11.40 (.0.00)
243.44
7.90 (-13.40)
267.56
Llama3. 1-8B
8.10
3.60 (44.50)
415.50
2.80 (60.00)
628.31
2.40 (6.700
1072.55
Llama3. 1-70B
20.80
17.40 (-3.40)
207.46
12.20 (-8.00)
348.43
4.70 (-10.10)
669.70
Llama3. 1-405B
24.40
13.80 (-10.60)
304.01
8.20
(-16.20)
395.69
3.20 (-21.20)
693.21
Qwen2.5-7B
11.50
9.90 (1.60)
241.29
7.10
(-1-00)
310.74
5.00 (-6.50)
511.24
Qwen2.5-72B
20.40
19.20 (-1.20)
196.61
12.40
(-8.00)
86.36
17.50 (-2.90)
593.44
Mistral-7B
7.60
5.20 (-2.40)
152.42
2.70 (-4.00)
139.00
3.10 (-4.50)
163.96
Mistral-Small 3
15.40
12.70
(-2.70)
369.97
12.90
(-2.50)
192.04
8.30 (-7:30)
595.81
Proprictary
Geminii11.5flla-
23.70
19.20 (-4.00)
181.79
11.50
(-12/20)
293.78
4.95 (-18.70)
459.69
Gemini-1.5-po
36.00
28.10 (-7800
226.60
25.29
(-10.71)
344.10
22.50 (+13.50)
463.41
Gemini-2.0-faar
31.60
28.70 (-2.90)
231.77
25.20
(-6.40)
216.32
14.81 (-10.79)
425.68
GP-T-3,5-turbo
11.60
8.40
(-8.20)
113.25
8,50 (-8.10)
116.25
7.89 (-8.71)
138.17
GPT-1o-mini
18.10
13.50 (-1.60)
171.29
11.90 (-6.20)
233.30
10.10
247.97
GPT-4o
25.30
21.70
(-3.80)
246.19
21.30
(-4.00)
208.00
19.16 (-6.14)
397.85
Average
19.73
14.88 (-4.85)
244.11
12.99 (.6.74)
272.18
9.24 (-10.49)
484.92
Table 13: Detailed LLM Performances on SALT.
Direct
CoT
ReAct
ToT
Model
Ace (%)
Ace (%)
# Lokens
Acc (%)
Lokens
Acc (%)
# tokens
Open-source
Deepseck-V3 V3
52.83
45.83
(-7.00)
161.16
36.08 (-16.76)
419.34
41.92 (-10.91)
549.59
Gemma2-9B
13.83
19.17
(+5.30)
139.96
17.08 (+3/25)
270.40
15.33
(+1.50) 50)
294.55
Gemma2-27BB
39.33
34.33
(-5.00)
100.25
26.67
(-12.66)
222.94
11.17 (-26.16)
323.54
Llama3. 1-88
25.25
17.92
(-7.33)
253.14
15.17
(-10.08)
505.72
15.00 (-10.25)
1082.77 77
Llama3. 1-70B
48.33
44.92
(-3.41)
204.69
41.42 (-6.91)
339.29
37.67 (-10.66)
602.01
Llama3. 1-405B
57.92
59.67
(#1.76)
430.02
54.17 (-3.78)
435.96
33.50 (-24.42)
616. 12
Qwen2.5-7B
15.58
13.33 (-2.25)
116.37
11.00 (-4.58)
165.47
10.83 (-4.76)
318.66
Qwen2.5-72B
35.25
39.42
(444177)
137.75
38.33
(43.08)
240.15
43.50
324.49
Mistral-7B
14.92
11.75 (-0-17)
112.82
11.17 (-3.78)
148.77 74
5.42 (-8.50)
357.95
Mistral-Small 3
44.67
36.25
(-8.42)
167.56
33.17
274.82
28.67 (-16.00)
584.81
Proprictary
Demini-115--flas
39.50
38.08 (-1.12)
201.34
41 41.58 (+2.08)
456.13
31.42
(-5 08)
542.69
Gemini-1.5-pro
53.50
45.92
(-7.88)
213.63
43.83 (-9.87)
476.21
42.92 (-10.58)
387.65
Gemini-2.06 2.0-fli-
53.67
48.92
(-4.75)
143.62
51.08 (-2.00)
403.63
47.50 (-6.17)
762.87
GPT-335-turbo
26.50
20.33
(-6.17)
107.55
15.50
(-11.00)
149.05
15.00 (-11.00)
154.97
CPT-4o-mini
31.83
24.17
(-7.86)
151.52
18,08
(-13.78)
245.74
11.67 (-20.16)
541046
GPT-4o
50.67
46.41
(-4.26)
174.41
42.67
(-8.00)
309.02
41.42 (-9.28)
440.49
Average
37.72
34.15 (40.77)
175.99
31.06
(-6 66)
316.41
27.25 (11.47)
492.73
34

---

Published in Transactions on Machine Learning Research (11/2025)
Table 14: Detailed LLM Performancos on List Function.
Direct
CoT
ReAct
ToT
Model
(%)
Acc (%)
# tokens
Acc (%)
# tokens
Acc (%)
# tokens
Open-source
Deepseek-V3
54.88
58.32
(+3-44)
487.50
34.56 (-20:32)
457.90
47.20 (-7.68)
176.20
Gemmaa-9B
32.80
25.20
(-7.00
146.12
22.24 (-10.08)
193.28
21.04 (-11.7)
370.36
Gemma2-27B
43.60
35.44
(8816)
98.08
34.88 (-6.72)
186.05
36.40 (-7:20)
243.89
Llama3. 1-8B
35.20
24.48
(-10.72)
435.90
18.48 (-16.73)
621.80
15.12 (-20.08)
978.29
Llama3. 1-70B
41.28
37.92 (-3.36)
171.67
38.16 (-3.12)
312.06
38.40 (-2.88)
578.45
Llama3. 1-405B
50.40
47.76 (-2.64)
425.44
47.68
(-2.72)
410.83
19.68 (-30.72)
545.20 24
Qwen2.5-7B
37.60
29.36 (-8-44)
382.33
21.36 (-16.20)
144.06
24.64 (-12.96)
348.04
Qwen2.5-72B
49.28
45.04 (-1.44)
427.86
42.96 (-6.22)
266.22
41.92
380.14
Mistral-7B
28.00
8.96 (-19.04)
161.29
4.96 (-23.04)
129.42
2.00 (-26.00)
392.23
Mistral-Small 3
41.76
38.32 (-3.44)
510.16
36.00 (-5.76)
224.67
32.32
524.35
Proprictary
Gemini-11.5fllas
46.96
42.00 (-4.66)
369.28
42.72 (+4.24)
485.17
39.44 (-7.58)
567.06
Gemini-1.5-po
53.28
52.00
(1.28)
341.34
53.60
(+0:32)
447.85
46.32 (-6.96)
456.78
Gemini-2.0-faar
53.76
51.12
(-2.64)
332.87
53.04
(-0.72)
451.64
40.56
(-18.20)
895.93
G1-T-3.5-turbo
42.16
31.12
(-11.04)
131.08
25.92
(-10.24)
99.82
24.48
(-17.68)
154.43
GPT-1o-mini
44.88
36.88
(-8.00)
188.62
35.04
(-9.44)
228.55
34.56 (-10.32)
331.80
GPT-4o
53.04
48.72
(-1.32)
211.48
45.84
(-7.20)
311.21
35.92
(-17.12)
141.89
Average
44.58
38.80 (-6.78)
305.49
35.25 (-9.33)
310.73
31.63 (-12.96)
486.49
Table 15: Detailed LLM Performances on RAVEN.
Direct
CoT
ReAct
ToT
Model
Ace (%)
Ace (%)
# tokens
(%)
tokens
Ace (%)
# tokens
Open-source
Deepseek-V V3
21.05
8.98 (-12.01)
397.14
6.27 (-14.78)
413.69
2.14
(-18.91)
780.90
Gemma2-9B
13.74
1.99 (-11.75)
232.07
0.87 (-12.87)
298.49
0.08
(-13.60)
435.18
Gemma2-27R:
16.60
1.11 (-1--19
196.51
1.91 (-14.68)
287.23
0.24
(-16.36)
349.96
Llama3.1-4 1-8B
7.07
0.87 (-6.20)
456.49
1.43 (-5.61)
757.91
0.32 (-6.75)
9-13.80
Llama3.1 3-1BBB
17.08
9.93 (-7.15)
325.79
5.24 (-11.84)
576.85
(-15.89)
1022.81
Llama3.1 1-405B
25.34
16.92 (-8.42)
355.79
11.68 (-13.60)
543.75
3.97
(-21.37)
843.70 70
Qwen2.5-7B
11.12
0.56 (-10.58)
673.17
1.59 (-9.58)
545.11
0.24 (-10.86)
964.45
Qwen2.5-72B
23.67
8.18 (-15.49)
476.07
6.35 (-17.52)
588.40
4.92 (-18.78)
608.39
Mistral-7B
1.51
0.00 (+1.51)
296.05
0.08 (-1.43)
326.30
0.16 (:1.35)
512.38
Mistral-Small 3
16.44
10.41 (-6.03)
828.07
5.80
(-10.64)
475.78
2.14 (+14.50)
841.24
Proprictary
Cemini-1.5-flan
19.06
5.88 (-13-18)
478.57
3.02 (-16.01)
644.90
1.27 (-17.79)
666.44
Gemini-1.5-pro
21.31
11.12 (-13.18)
545.27
12.87
(-11.41)
728.48
7.94 (-16.37)
564.41
Gemini-2.0.. f.0-llhhh
23.35
19.78 (-8.57)
719.08
20.02
(-8.38)
1017.31
13.66 (-0.60)
1634.67
GPT-3.5-turbo
12.79
5.80 (.6.90)
206.14
3.57 (-9.28)
131.61
3.73 (-0.06)
235.38
CPT-4o-mini
15.65
6.12 (-9.85)
322.94
3.18 (-12:47)
337.56
0.95 (-14.70)
598.97
GPT-4o
22.24
10.33
(-11.91)
427.07
8.90
(-18.54)
469.12
6.43 (-15.81)
718.92
Average
17.25
7.37 (69.85)
434.75
5.87
(111.8)
533.09
2.84 (+14.11)
737.64
35


Reviews:
[{"review_id": "7VHti9dJLj", "reviewer": "Reviewer_2nPd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper challenges the conventional wisdom that chain-of-thought (CoT) prompting “generally helps, especially for the pattern-based ICL settings (symbolic, textual tasks). The authors report that direct answering outperforms CoT/ReAct/ToT, and that the gap often widens with more demonstrations. To understand the mechanism behind the phenomenon, they propose various initial hypothesis including (i) a contextual-distance effect (rationales push the answer further away from the demonstrations, harming ICL),  (ii) LLMs struggle to infer underlying patterns (iii) LLMs struggle to apply underlying patterns. By designing and conducting the experiments, they finally synthesize their hypotheses to an explicit–implicit duality in CoT: explicit, verbalized pattern inference is often wrong, while an implicit mechanism (akin to direct answering) still sometimes recovers the right label. In last section, the paper also compare the SOTA LRM with SOTA LLM on these pattern-based tasks. Even long-CoT LRMs (o1-mini, DeepSeek-R1, QwQ-32B) fail to beat strong LLMs with direct answering despite far higher token costs, which further confirms the author's conclusion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "To understand the reason why CoT doesn't help in pattern-based tasks, the authors first give three hypotheses (contextual distance; difficulty of pattern inference; difficulty of pattern execution) and finally conclude a new dual-process view of CoT (explicit vs implicit) make the analysis structured and testable. \n\nSpecifically, they conduct clean ablations: 1). Dummy-rationale insertion (Shakespeare/countdown) to vary distance without adding task-semantics. Performance degrades as distance grows.  2). Rationale front-loading (prepend the rationale before demos) preserves semantics but removes the distance—performance improves, supporting the distance hypothesis.\n\nIn experiments, they compare the results of 16 LLMs on 9 datasets. They also compare the recent reasoning model with vanilla LLMs, which makes the insights more convincing.", "audience_yn": "Yes", "audience_explanation": "This paper is well-motivated and propose a novel view to understand when and why the reasoning may help for LLM. This is an interesting and important topic in LLM domain.", "requested_changes": "I appreciate the effort and all the experiments designed and conducted to help people understand why CoT doesn't always help for pattern-based tasks. However, it would be more helpful if the author can design experiments or at least discuss the root cause for 1) why the dual-process happens  2) why the dual process happens especially for these \"pattern-based\" tasks (this is a descriptive word instead of strictly defined task/term). These questions may lead to deeper understanding of the issue during training process and help LLM researchers to enhance their models. \n\nFor point 1, For example, one hypothesis here can be that during training (especially post-training), most pattern-based tasks (IFT-tasks) are trained in a naive manner instead of CoT SFT or RL. Can more CoT training data bridge the gap? \n\nFor point 2, How to define the pattern-based tasks - in the more difficult but also somehow pattern-based tasks (e.g. formal methods, math), the same phenomenon still exists? In this paper, we only see the tasks that align with the author's hypothesis but don't know what's the boundary of these \"pattern-based\" tasks.", "broader_impact_concerns": "I don't have any concerns on the ethical implication of this work.", "additional_comments": "This paper is very insightful but it would be great (as mentioned above) to connect the phenomenon to the root cause we may improve during training.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MpeINcdhFc", "reviewer": "Reviewer_2yea", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper „The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning” describes an empirical study on problems CoT has with few-shot learning. The core of the paper is built on the foundation that CoT solutions underperform direct answering for few-shot tasks for multiple benchmarks. The authors then hypothesize about the reasons, i.e. that that the distance between few-shot examples and response created through the CoT is responsible, that LLMs fail to identify the pattern within the few-shot examples as part of the CoT, and that even if they can identify this pattern, they fail at applying the pattern. Based on the evaluation of these hypotheses, the authors finally look at how often LLMs are better with a direct response (implicit reasoning) in comparison to how often CoT actually helps (explicit reasoning) and find that the direct relationship is the strong, more important signal in comparison to the CoT.", "claims_and_evidence_yn": "No", "claims_explanation": "I agree with the premise of the paper, i.e., that CoT is no guarantee to achieve better results in all settings. I also believe the authors provide strong evidence for this. I also like the general approach to then look at different hypotheses and test them to understand what might cause the, for many unexpected, behavior that CoT decreases performance. I think the analysis for H4 is very well done and the core contribution of this paper. \n\nMajor issues:\n\n1) I do not think that the experiment conducted to assess H1 is suitable. The experiment only evaluates if the LLM performance degenerates, if the LLM is forced to generate noise before answering. Frankly, I am also not sure how to setup a better experiment. But simply stating: “if LLMs generate noise in between and the performance suffers, this explains that the performance also suffers when we ask LLMs to generate contextual information before coming to a result” is too much of a stretch for me. Unless the authors can come up with a better way to measure something related to H1, I suggest dropping this hypothesis. \n\n2) Hypothesis 2 and Hypothesis 3 are independent from each other. Consequently, a statement like “Hypothesis 2 is validated over Hypothesis 3” does not make sense. Both should be independently accepted or rejected.\n\n3) The results presented in Section 6 for LRMs are weak due to the unfitting study design. To really quantify the difference between a normal LLM and a LRM, one would need to have pairs of models, such that you have a LLM and a LRM from the same model family with roughly the same size. Otherwise, any comparison is very noisy because other factors (e.g., different pre-training, different model size, etc.) can explain the differences. The current setting is hardly fair. For example, Qwen2.5-72B is used as normal LLM, but the smaller QwQ-32B is used as reasoning counterpart. \n\nSuggestions for additional improvements: \n\n1) The paper anthropomorphizes the LLMs quite a lot. I suggest avoiding concepts like cognitive load for the stochastic reasoning modeling that LLMs implement or call what LLMs do “verbalization”.", "audience_yn": "Yes", "audience_explanation": "CoT is a standard technique and knowledge about the limitations is important to guide expectations.", "requested_changes": "See major issues above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kwBr7KDXIh", "reviewer": "Reviewer_MEMh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A key strategy for improving the performance of LLMs on many tasks is \"Chain of Thought\" (CoT), where the model is prompted to generate tokens corresponding to some reasoning process before its final answer. CoT is generally viewed as an unalloyed good for performance, with the only downside being that it is more costly in terms of tokens. This paper shows interesting empirical evidence that in at least one case (in-context learning), it can actually harm performance, which is surprising.\n\nIn-context learning is the case where a few example inputs and outputs are put into the LLM's context, before generating an output for a new unlabeled example. In other words, without touching model weights, the LLM itself can function as a supervised learning algorithm. Observing this phenomenon in GPT models was an early driver of excitement about LLMs.\n\nWhat the authors show is that, compared to direct output, adding various CoT methods to an extremely wide range of models doing in-context learning on a range of tasks hurts performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the authors consider a wide range of models which are not that old, and they consider wide range of in-context learning tasks. They propose multiple hypotheses and rule out some. They also present convincing, though not totally certain, evidence explaining why CoT might interfere with in-context learning. The decline in performance from CoT is remarkably consistent across many settings. Figures 1 and 2 are very striking. The results are quite convincing, at least to me.", "audience_yn": "Yes", "audience_explanation": "A lot of people are interested in LLMs as few-shot learners, even if just as end users. I think the results in this paper would be of wide interest.", "requested_changes": "This is not strictly a request, just a remark: when I see the word \"duality\", it naturally makes me expect to see a rigorous mathematical concept. Personally, I would consider using a different name for the relationship between implicit and explicit reasoning.", "broader_impact_concerns": "No broader impact concerns in my view.", "additional_comments": "I've worked with LLMs in my research, including chain-of-thought prompting them to do various things and trying to get them to do in-context learning. So I'm not unfamiliar with the ideas here. However, empirical eval of LLMs is a deep and tricky area where I don't have special expertise. Therefore, if e.g. the chosen in-context datasets were somehow cherry-picked, if some essential datasets were left out, or if some special choices in prompting or hyperparameters explain these results, I wouldn't be able to catch this.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)