When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs
Authors: Xiaomin Li
Zhou Yu
Zhiwei Zhang
Xupeng Chen
Ziji Zhang
Yingying Zhuang
Narayanan Sadagopan
Anurag Beniwal
Venue: NeurIPS 2025 spotlight ()
Keywords: Large language models
Reasoning
Instruction Following
Chain of Thoughts
Primary area: deep_learning
TL;DR: We show that explicit reasoning via chain-of-thought can hurt instruction-following in LLMs by reducing constraint adherence, and propose four mitigation methods to recover or improve performance.

Abstract: Reasoning-enhanced large language models (RLLMs), whether explicitly trained for reasoning or prompted via chain-of-thought (CoT), have achieved state-of-the-art performance on many complex reasoning tasks. However, we uncover a surprising and previously overlooked phenomenon: explicit CoT reasoning can significantly degrade instruction-following accuracy. Evaluating 20+ models on two benchmarks: IFEval (with simple, rule-verifiable constraints) and ComplexBench (with complex, compositional constraints), we consistently observe performance drops when CoT prompting is applied. Through large-scale case studies and an attention-based analysis, we identify common patterns where reasoning either helps (e.g., with formatting or lexical precision) or hurts (e.g., by neglecting simple constraints or introducing unnecessary content). We propose a metric, constraint attention, to quantify model focus during generation and show that CoT reasoning often diverts attention away from instruction-relevant tokens. To mitigate these effects, we introduce and evaluate four strategies: in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning. Our results demonstrate that selective reasoning strategies, particularly classifier-selective reasoning, can substantially recover lost performance. To our knowledge, this is the first work to systematically expose reasoning-induced failures in instruction-following and offer practical mitigation strategies.

When Thinking Fails: The Pitfalls of Reasoning for Instruction-Folloving in LLMs
Xiaomin Li Harvard University
Zhou Yu
Zhiwei Zhang Amazon
Xupeng Chen NYU
Ziji Zhang Amazon
Amazon
Yingying Zhuang Amazon
Narayanan Sadagopan
Anurag Beniwal
Amazon
Amazon
Abstract
Reasoning-enhanced large language models (RLLMs), whether explicitly trained for reasoning prompted via chain-of-thought (CoT), have achieved state-of- the-art performance on many complex reasoning tasks. However, we uncover a surprising and previously overlooked phenomenon: explicit CoT reasoning accuracy. Evaluating 20+ models on two benchmarks: IFEval (with simple, rule-verifiable constraints) and ComplexBench (with complex, compositional constraints), we consistently observe performance drops when CoT prompting applied. Through large-scale case studies and an iiteetion-bbsed analysis, we identify common patterns where reasoning either helps (e.g., with formatting or lexical precision) or hurts (e.g.. by neglecting simple constraints or introducing unnecessary content). We propose metric, constraint attention, to quantify model focus during generation and show that CoT reasoning often diverts attention away from instrvction-relevann tokens. To mitigate these effects, we introduce and evaluate four strategies: in-context learning, self- reflection, self-selective reasoning, and classifier-seledd reasoning. Our results demonstrate that selective reasoning strategies, particularly classifier-selective reasoning, can substantially recover lost performance. To our knowledge, this is the first work to systematically expose reasoning-induus failures in instruction- following and offer practical mitigation strategies.
significantly degrade
Introduction
Reasoning-enhanced large language models (RLLMs) have demonstrated remarkable success
a variety of tasks, including mathematical problem solving, planning, and multi-hop question an-
swering [Guo al. [205||Achiam [Achiam al. 2023 Grattafior et al.) 2024 Xu et al. 2023 Zhou et al.)
2022 Wu et al.] 2024]] Fu et al.) 2022 Qi et al. 2024 Chae et al.] 22024]. central contributor
these advancements chain-of-hhuugh (CoT) prompting Wei al. 2022b] Nye al. 202
Joshi et al. 2023 Lanham et al. 2023), which explicitly encourages models to reason step-by-step
prior to providing an answer. Recent prominent models, such DeepSeek-RI [Guo al. 2025
Claude (Anthropic 2025), and OpenAl's O-series [OpenA] 2024]. either incorporate CoT-style reasoning their fine-tuning procedures or explicitly offer reasoning as an inherent capability. While CoT generally improves performance on complex reasoning tasks, it incurs higher computational cost due to longer outputs and increased inference latency. Moreover, its impact on more structured tasks such as instruction following remains underexplored. Instruction following, the ability to comply with user-specified constraints, essential for alignment, safety, and practical usability of
"Correspondence to: Xiaomin Li (email: xiaominli@ h.arvard.eduu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

language models [Ouyang et al. 2022, Zhang et al. 2023]. This raises natural question: Does explicit reasoning actually help model follow instructions more accurately?
In this paper, we answer this question empirically and arrive at a surprising conclusion: reasoning via CoT can degrade model's 's ability to follow instructions. To investigate this, we evaluate 20+ language models of varying sizes and training paradigms, including general -purpose models (c.g., Llama, Mixtral) and reasoning-tuned models (e.g., Claude 3.7. DeepSeck-R1), on two complementary instraction-folloiing benchmarks: IFEva (Zhou et al., [023] consists of prompts with simple, independently verifiable constraints (e.g.. "write least 400 words" or "mention AI three times"). In contrast, ComplexBench [Wen et al. 2024] includes instructions formed through compositional logic, combining multiple dependent constraints via operations like chaining, selection, and nesting. Across both datasets, we observe consistent and often substantial accuracy drop when models are prompted with COT. This finding surprising, given that reasoning typically expected to improve performance on many tasks even more challenging than instruction following.
To understand this phenomenon, we conduct two complementary analyses. First, we perform large-scale manual case study of samples where reasoning notably affects performance. We find that reasoning helps two common scenarios: (1) satisfying formatting or structural requirements, and (2) enforcing lexical constraints that override default tendencies. However, it often hurts performance by: (1) over-focusing on high-level content and neglecting simple constraints, or (2) introducing redundant or well- rell-intentioned content that unintentionally violates constraints. We provide detailed examples and analyses ses of each scenario. Second, we investigate the impact of reasoning through an iitention-based analysis. We propose a quantitative measure: constraini attention, which is based on attention scores directed toward constraint tokens within instructions. Visualizations of attention patterns during response generation consistently demonstrate reduced constraint awareness when CoT prompting is employed, an effect observed across different datasets, models, and layers. This shift in attention might explain, at least partially, why reasoning can diminish instruction adherence.
Following these insights, we propose and evaluate four methods to mitigate the adverse impacts of reasoming on eestruction-oolooweee accuracy: (1) In-context learning, where we identify and correct typical reasoeing-iduuced induced failures, incorporating these examples into prompts; (2) Self-reflection. prompting models to evaluate and adjust their reasoning processes and candidate responses; (3) Self-selective reasoning, allowing models to autonomously decide when reasoning is beneficial; and (4) Classifier-selectiee reasoning, where trained classifier determines when reasoning is necessary. Among these methods, we find that classifier-selectiee reasoning strategies offer substantial gains across benchmarks, while self-reflection is particularly helpful for larger models and simple instructions. We thoroughly discuss the comparative strengths and limitations of these mitigation strategies in Section 5
In summary, our main contributions are listed as follows:
We evaluate 20+ language models on comprehensive instruction-folloiin benchmarks, revealing that explicit reasoning can negatively impact cnstruction-fllowiinn capabil- ities"] To our knowledge, we are the first discover and systematically explore this phenomenon.
We provide detailed analyses of reasoning-induced failures, categorizing scenarios where reasoning either helps or hinders, and quantify attention shifts that explain these performance drops.
We propose and rigorously examine multiple mitigation strategies, including in-context learning, self-reflection, self-selective reasoning, and sanssffier-selectvee reasoning, demon- strating their effectiveness and highlighting promising directions for future research.
2 Related Work
Chain-of-Though and Reasoning LLMs.
State-of-the-art large language models (LLMs) often
leverage explicit reasoning capabilities, exemplified by models such as OpenAl's O-series (OpenAT
2024 DeepSeek's R1 [Guo et al. 2025 and Anthropic's Claude [Anthropic 2025
These models
2 "Code and data for reproducing experiments are available at: https://github. com/ www.:/uw....ommmaaaoaoooo awwaonnsscieneee
when- thinking- fails-ailllll- iil-ifffevauution
2

---

are trained on datasets incorporating not just direct responses but also explicit reasoning processes
known as Thain-of-Thoughh (CoT) [Wei et al., 2022b Nye et al., 2021 Joshi et al.] 2023 Lanham
et al. 2023] CoT prompting, which encourages step-by-step reasoning. has demonstrated significant
success, particularly in domains requiring complex reasoning, such as mathematics [Guo et al. 2025
Achiam et al., 2023] Grattafioni et al., 2024 Xu et al., 2023] Zhou et al. 2022 Wu et al., 2024
Fu et al. 2022 Qi et al. 2024| Chae et al.
2024). However, CoT can also introduce additional computational costs and may yield limited or no improvement in certain contexts [Kambhampati
(ct al. 2024, Wang et al. 2024| Sprague et al. 2024]. Our study further explores this dual nature of
reasoning, highlighting cases where it can negatively impact instruction-folloot following performance.
Instruction Following. Instruction-follooinng is essential for aligning language model outputs with user expectations, enabling the models to reliably execute user-specified tasks. Techniques such as instruction tuning (fine-tuning models on extensive datasets
instruction-responee pairs)
have been instrumental in fostering this capability [Ouyang et al. [0022] Wang et al. [023]] Longpre
al. 2023 Bai et al.) 2022 Wei et al. 2022a Chung et al. [022] Muennighoff et al. 2023). This
ability is critical for bridging the gap between pre-training objectives and desirable human-aligned behaviors such as helpfulness and relevance [Radford et al. 2019 Brown et al. 2020 Despite its
importance, instruction-followin remains challenging, especially when instructions involve complex, multifaceted requirements [Zhang et al., [023 He et al. 2024 Gudibande et al. 2023 Kung and Peng
2023 Heo et al. 2024] To systematically evaluate LLM instruction adherence, benchmarks such as IFEval Zhou et al. 20023 and ComplexBendd [Wen et al. (0224 have been introduced. IFEval focuses on rule-verifablee straightforward constraints, while ComplexBench assesses models sophisticated instructions involving nested and dependent constraints. Further details and comprehensive analyses
of these benchmarks will be provided in our experimental evaluations (Section
3 Experiments
3.1 Datasets and Evaluation Metrics
We use two benchmark datasets, IFEval and ComplexBench, comprehensively evaluate the instruction-follooingg capabilities of language models:
IFEval is synthetic dataset consisting of 541 prompts, each associated with one to three verifiable constraints drawn from 25 types (c.g., word count, formatting, keyword usage). We adopt instruction- level loose accuracy, which allows minor formatting deviations (c.g., markdown wrappers) to avoid penalizing responses for stylistic differences, thereby better reflecting practical robustness.
ComplexBench is manually curated dataset designed to assess models on complex compositional instructions formed through four operations: And, Chain, Selection, and Nested. contains 1,150 instructions and over 5,300 scoring questions, covering 4 constraint types across 19 dimensions (e.g.. lexical semantic, formatting). Evaluation combines rule-based and LLM--based assessments. For our experiments, we translated all scoring rules into English and manually verified them to construct fully English-compatible version.
Evaluation Metrics: For both datasets, we report the proportion of constraints satisfied per instruction. In ComplexBench, dependency logic applies: failure in prerequisite constraint results in automatic failure of all dependent constraints.
3.2 Models
We evaluate a diverse set of models, including both closed-source models (c.g., Claude3. 7-Sonnet) and foccused models (e.g., DeepSeek-Ri -R,, Even-R1-distilne source selection spans parameter scales from IB to 70B. All model inferences use a temperature of O. Open-source models are run without quantization using 4 -VI0---H100-SOGB GPUs. In addition to single-model CoT vs. non-CoT comparisons, we also evaluate paired variants under a controlled setting: base vs. reasoning-enablen counterparts (e.g., Qwen2.5 20wen2.5-Instrue Instruct vs. Math2 Qwen3 Base vs. Qwen3 Think; Claude-3.7-Sonne vs. Claude-3.7-Somne --Snn-tt- Think; DeepSeek- V3 vs. DeepSeck-RI) R1), summarized in Table 2
variants). Our open-
3

---

3.3 CoT Prompting
We compare model behavior with and without Sfain--ffTThugeee (CoT) reasoning. The CoT prompts instruct models to reason step by step before producing an answer (exact prompt provided in Appendix FD. We then assess instruction-followwe performance based on the model's answer each setting.
3.4 Results
Performance on IFEval and ComplexBench is reported in the first three columns of Table with visual summaries provided in Figure 3 Notably, 13 out of 14 models experience performance degradation on IFEval when CoT prompting applied, and all models show declines on ComplexBench. For instance, the accuracy of Llama3-888 8B- InBtrutt drops from 75.2% to 59.0%, reduction of over 16 percentage points.
We further compare reasoning-enabled models with their corresponding base variants across nine pairs, including Claude3. 7-Sonnet Claude3. --Sonnet--Thine DeepSeek-V3 DeepSeek-R1, two Qwen2. 5- InInrruttt vs. Qwen2. 5-Math pairs (1.5B, 7B), and three Qwen3 Base vs. Think pairs (4B, 8B, 32B). These results, shown in Table 2 reveal that reasoning variants typically un- derperform their base counterparts on instruction-follwwiii rentrolled-rrsoonnnng
While these comparisons are not fully models may undergo additional training stages such supervised fine-tuning or RL/RLHF the decline remains evident across both benchmarks.
Overall, these findings uncover surprising and underexplored vulnerability: explicit reasoning, while often helpful in complex tasks, can increase the likelihood of violating instruction constraints, therehy impairing issructtoonfollowing reliability.
4
Analysis
To better understand when and why reasoning degrades instruction-followigg we conduct two analyses: (1) manual case study examining when CoT helps or hurts constraint satisfaction, and (2) an attention- based analysis investigating how reasoning shifts model focus away from constraints during generation.
4.1
Case Study
We manually examined all 541 samples from IFEval and over 1,000 samples from ComplexBench. focusing on cases where CoT affected whether constraints were satisfied. For each case, we analyzed which constraints were impacted and why the reasoning-based answer either improved or degraded performance.
Due to the length of the instructions and responses, we present detailed illustrative examples in Appendix B For each example, include outputs from both the base model and the reasoning- enabled model, showing their respective Answer and, for CoT, the full Thinking process. We also report the number of constraints satisfied and include or harmed performance. Although we examined large number of examples, the failure and success cases largely fall into four recurring patterns which we summarize below:
analysis explaining how reasoning helped
Reasoning Helps
1. Formatting and Structural Adherence: Reasoning improves compliance with structural con- straints, such as producing valid ISON, wrapping the output in double quotes, or following markdown syntax.
2. Lexical and Keyword Precision: Reasoning enhances adherence to lexical requirements, including inserting rare characters (e.g., the letter q six times), omitting final punctuation, or using exactly 15 capitalized words.
Reasoning Hurts X
'eeyond these behavioral patterns, reasoning also increases computational cost due to longer outputs and additional thinking tokens.

---

IREval Results
Baseline
Mitigation Methods
Self-Selective Reasoning
Classifier-Selettiv Reasoning
Model
Original
CoT
FewShot
SelfReflection
Claude-3.5-Haike
86.9
79.5
80.2 (1 0.7)
81.7 2.2)
80.8 1.3)
85.8 (1 6.3)
Claude-3.5 S.noonc
86.5
79.5
87.4 (1 8.0)
87.6 (1 8.1)
81.9 (1 2.4)
84.8 (1 5.4)
Claude-3.7-Sone 3.7-Sonnet
90.6
90.2
89.3 () 0.9)
92.1 (f 1.9)
90.4 (1
90.4 <f 0.2)
DeepSock-V2
85.2
84.3
85.2 (1 0.9)
88.3 (1
85.3
84.2 (4
Llama-3.2- 2.--------Btt
49.0
40.7
13.1 (7.5)
34.9 34.9
45.7 5.0)
47.5 (1 6.8)
IJama-33 4.2-3B-Instris
70.4
61.9
37.2 () 24.8)
64.9 3.0)
61.9 (..0.0
68.8 (1 6.8)
Mcta-Llamaa3-88P Instruct
75.2
59.0
36.8 () 22.2)
65.1 (f
59.0 (4
69.7 (1 10.7)
1lama-3.17708-Imme
85.6
44.5 (2.37)
84.3 7.0)
77.3 (..0.0)
83.2 5.9)
Mixtral-&x7B-Inneee Instrrcct
53.0
56.4
(2.0
55.6
33.1 23.3)
55.1 (L 1.3)
Qwen2.5-1 1.5.--1.1--111 1.5tttruttt
35.9
31.6
20.1 (1.5)
26.6 (4.55)
32.7 (1 1.1)
34.8 (1 3.2)
DeepSeck-R1 DeepSeck-RI-iidiliee II-Disill-Qwee--.50
16.8
13.7
22.0 (1 8.3)
20.0 6.3)
13,9 (1 0.2)
17.0 3.3)
Owen2.5-7B-Innstee
63.4 (1 5.7)
(1 16.6)
59.7 (1 2.00
68.8 (1 11.1)
DeepSeck- RL-Distill-Qeen-7B
30.9
25.1
30.1 (.0)
36.4 (1 11.3)
26.2 L.))
30.1 (1 5.0)
ComplexBench Results
Bascline
Mitigation Methods
Model
Original
CoT
FewShot
SelfReflection
Self-Selective Reasoning
Classffer--Slecttve Reasoning
Claude-3.5 H.uiie
66.9
59.7 (L 2.4)
61.5 (4 0.6)
67.6 5.5)
67.1 (1 5.0)
Claude-3.5 3.5de-3.5-Sonet
67.5
(1 0.9)
65.5 ( 1.5)
67.6
68.1 (1 2.1)
Claude-3.7 3.7-Sonnet
69.8
67.0
67.4 (1 0.4)
67.8 (f 0.8)
69.6 2.5)
69.4 (f 2.4)
DeepSeck-V V3
71.2
71.1
66.0 (). 5.1)
65.2 (4.5.)
71.4 (1 0.3)
71.5 (1 0.4)
Ilama-3.2 1--Itunntttctt
36.0
26.0
19.7 (L 6.3)
23.7 (4 2.3)
36.2 (f 10.2)
35.8 (f 9.8)
-luum--3.2-3B-imm
50.0
45.7
49.2 (1 3.5)
44.7 (4 1.0)
49,8 (1 4.1)
49.7 (1 4.0)
Sanu-Llama-3--8 3-Bna------nnttru
55.8
54.9
54.7 (1 0.2)
49.1 (1.588
55.4 0.5)
55.8 (1
IJama-3.1 10ma--31-70BB-.. In-00--nstteet
66.8
60.2
66.8 (1 6.6)
61.9 1.7)
66.7 (1 6.5)
67.8 (1 7.6)
Mixiral-Sx77-Innteee
60.4
58.3
(L.
(L
59.6 L.3)
59.6 (1 1.3)
Qwen2.5-1
44.1
38.1 (L.77)
3l.
44.0 (1 5.2)
44.0 (1 5.2)
DeepSeck-RI BeSSSck-RI-Disiil---ee Si--Disill-Qwen--.BB
18.9
24.1 (.55)
20.5
18.7 (..)
18.7 (1
Qwen2.5-7B Benn2..-7B-Instruee
60.2
52.6
60.2 (1
55.8 (1 3.3)
63.4 (1 10.8)
63.4 (1 10.9)
SrepSeck-RI-Disiil--Peee
46.0
38.6
50.2 (1 11.6)
37.0 1.6)
46.0 7.4)
45.5 6.9)
Table 1: Instruition-follovinng performance on IFEval and ComplexBench. Green and red indicate whether Original or CoT performed better. Each mitigation column reports the accuracy along with its change relative to CoT, using for improvement and for decline. Bold values highlight the best-performing mitigation method for each model.
3. Over-Focusing High-Level Content and Neglecting Simple Constraints: When multiple constraints are present, reasoning often emphasizes content planning at the expense of simpler mechanical constraints. Common issues include exceeding word count limits, failing to repeat prompts exactly, using capital letters lowercase-only tasks, or appending unnecessary content after required phrases.
4. Introducing Unnecessary Content that Violates Constraints: Reasoning frequently inserts redundant well-intentioned additions-sun such explanations, translations, empha- sis- sis-that break constraints. Typical violations include: inserting English text into "foreign language only" outputs, including commas in "no commas" tasks, appending commentary to quote-only responses, or exceeding limits on capitalized words.
4.2 Constraint-Aware Attention Analysis
To understand why reasoning may degrade instruction-follwvii we analyze whether models pay less attention constraint-relevwe parts of the prompt during response generation. many failure cases, observed that models neglect certain constraints, either by overemphasizing content planning
5

---

Model Pair
1FEval (%)
ComplexBench (%)
Claude-3.7 vs. Claude-3. Saaude-3.7-Somee 7-Sonnet-Thine
90.6/ 90.2
69.8 69.0
DeepSeek-V3 DeepSeek-RI
85.2 83.3
71.2/711
1rre22.5-1.5B-Insttte vs. DeepSeek-RI Biieek---I-Dsiille 1-ii-tiDil----- Dise--i-l-----e---l-- Qwen-1.5B
35.9 13.7
44,1 16.7
Qwen2.5-7B-Instnn vs, DropSccs-RI--Disill-wwaass
03.6/
30.2/38.6
Owen2.5-1.5B-Innstee Insttuuttt vs. Qwen2.5-Math-l
35,9 14.9
44.1 43.1/23.4
Qwen2.5-7B-Innttmu vs. Qwen2.5-Math-78
63.6/27.9
/
Qwen3-4B Qwen3-4B-THe
85.0 69.5
63.8/59%
Qwen3-8B vs. Qwen3-8B-Think
86.8 86.3
69.6/59.8
Qwen3-32B vs. Qwen3-328-Thie Think
87.8/ 85.0
70.2/63.1
Table 2: Comparison of caasoning-enable models and their non-reasoning counterparts across both benchmarks. Each cell reports accuracy on IVEval and ComplexBench, with green marking the gigher-performin model and red the lower-performing one. Results generally show that explicit reasoning either provides negligible gains or causes small performance drops.
introducing irrelevant information. To investigate this phenomenon systematically, conduct an attention-based analysis that tracks the model's focus on constraint tokens throughout generation.
We define constraint-altentin metric to quantify the model's awareness of constraint-relevan tokens. For each instruction, we first use GPT-4o to automatically extract substrings corresponding to each constraint, and then map them to token indices in the prompt. During generation, compute attention scores directed toward these tokens for both the reasoning and answer segments. Each model is twice per instruction: (i) Base run: Instruction Answer, and (ii) Reasoning run (CoT): Instruction >> Think > Answer. We focus our comparison on the answer segments of both runs.
Transformer Attention. Let the prompt contain To tokens #1.T, and let the model generate T new tokens y1:7- At generation step (1 s 7), the visible context is (I1:16 (1.01)) Let (1) denote the hidden state of the most recent token at layer 1, and let & E {1,..., H) index attention heads. The attention components for layer I and head are computed as:
Q = w-(lk) 42, K
(.A.) (()
11oot-2- v = (((..) HiTo-t-2
where 1 is a single vector (current token), and H 4- is the matrix of all prior hid- den vectors (prompt and generated context). This leads to scaled attention weights A (l.t)
softmax Q(119((0)) Vdv
Averaging over all heads yields the layer-level attention vector
H
(4.))
(1,)
H
km
Constraint Attention. For each textual constraint G, (r ...... R), collect the prompt token indices spans, yielding the index subset C, C (I.....To) To}. Define the full constraint token set as C = U Cr. Then we can define the layer-step constraint attention (averaged attention to constraint tokens at each layer and step) as
(1t)
C
JEC
Following this, we compute the layer-averaged constraint attention at step as:
(1)
Σ
(.))
(1)
1-0
Based on Equation D we visualize the trace of constraint-attention during response generation in showing results for both IFEval and ComplexBench using Qwen2, 5-1.5B 5B SB-Instruct.
Figure [

---

Additional examples for other models are provided in Appendix C After reviewing hundreds of samples, we observe general trend: reasoning flattens the constraint- reasoning degrades performance, constraint attention during the answer phase is generally lower. In contrast, when reasoning improves performance, we often see bump in attention aligned with the answer segment.
trace. In cases where
Quantifying Attention Drop. To further quantify these observations, we compute the mean constraint attention across the answer phase. Let A denote the answer token positions. The average constraint attention at layer is:
B(0
1
Σ
(1,0
(2)
A
BEA
We define the attention drop the difference between base and CoT runs:
AB A
Base
Bcor
We find that A, > 0 for most cases. Figure shows how this drop varies across layers for WIN vs. LOSE cases, On average, the model exhibits larger attention drops in LOSE cases, particularly in carly-to-middle layers. This suggests that lower constraint attention is predictive of reasoning- induced failuurs.
Conclusion. Our analysis reveals that explicit reasoning often reduces attention to constraint- relevant parts of the prompt. This diminished awareness increases the risk violating instructions. Thus, while reasoning is intended to improve task performance, it can unintentionally shift model focus away from critical constraints and harm instruction adherence. In addition, we also explored whether longer reasoning tends to degrade performance and found that reasoning length does not meaningfully correlate with cittructtoo-followinn effectiveness (Appendix
5 Mitigating Reasoning-Inducee Failures in Instruction Following
introduce and evaluate four strategies designed to mitigate the performance degradation caused by explicit reasoning (via CoT) in instruction-follot
tasks. To the best of our knowledge, this the first work to systematically identify and address this issue. Hence, the absence of existing baselines, directly compare the effectiveness of our proposed methods. Results are presented in Table I and Figure which also include performance differences relative to the CoT baseline.
5.1 Method 1: Few-Shot In-Context Learning
We apply in-context learning [Brown et al. 2020 by prepending carefully selected few-shot examples to each instruction. These examples are derived from representative failure cases identified in our case study (Section (B) and manually revised to fully satisfy all constraints. Each example includes an Instruction, along with a corrected Thinking and Answer, Full examples are provided in Appendix H
As shown in Table I this method yields only modest improvements. We attribute the limited gains several factors. First, due to the token length limit and the substantial size of each example (including prompt, reasoning, and answer), we were only able to include a limited number of examples: four examples for IFEval and three for ComplexBenchh Second, because examples were sourced from outputs of certain models, potentially introducing bias. These constraints likely reduce the effectiveness of the few-shot strategy and limit its generalizability.
Results:
5.2 Method 2: Self-Reflection
This method performs a two-step process: the model first generates an initial response with thinking, and then performs second inference where it reflects its reasoning and answer. If the model deems the initial response satisfactory, it retains it as the final output; otherwise, it revises the response and outputs the updated version. The prompt used for self-reflection is detailed in Appendix F
Results: As shown in Table self-reflection yields strong improvements on IFEval enhancing performance in 11 out of 14 models, and achieving the best results among all four methods for
7

---

(a) IFEval - Attention Traces
(b) ComplexBench Attention Traces
Base (score 1) Tue. False)
CoT (score 11 [TIE]
Base Iscore
CoT (score 0) (TE) [False, False, False]
[Tuue, False]
[False, False, False]
0.022
0,012
Pecsonalzatin
0.010
0.000
Attention
Attention
Attenclon
0.008
Attertion
0.000
0.0 0.025
2.033
0.006
0.006
0.020
2.020
0.004
0.004
0.002
0,002
0.000
0,000
100
30
300
130
200
250
200
400
100
200
350
400
500
900
Token position
Taken position
Taken P position
Taken position
Baae
[LOSE]
Base (scone
CoT (score 3) [LOSE]
iiuee truel
[itue, False]
True]
[False,
0.040
0.040
ksywondaslorballe spords
Topic
Language Siple
Heiphdrese bencimere
Attenton
0.495-
Attention
0.005
Attention
0.04
0.00 0.026-
9 9.625-
0. 2.09
0.03
0-015
9.00 0.00-
0.92
0.01
0.81
0.000
0.002
0.00
0.00
1.0
210
250
50
150
150
260
250
Token position
Taken position
Token position
Token positiin
Base
iscore [WIN]
Base Escore
CoT (score 3) [WIN]
Faise)
Repeorderreouencyy
0.040
0.040
He planness
0.037
number Jish section
0.055
0.035
Tagget Language
Attention
0.030
0.005
Atention
0.030
0.025
0.025
0.050
0.027
0.030
0.000-
0.020
0.020
0.020
0.013
0.013
0.000
0.002
0.000
0.000
200
Token position
Token position
Token position
Token position
Figure I: Constraint-attention trace examples for Swen2.5-12 5B Instruct across both datasets.
From top to bottom, the plots show cases where reasoning leads to a TIE, LOSE, and WIN compared to not using reasoning. The vertical red dashed line marks the start of Thinking, and the green dashed line marks the start of the Answer during response generation. WIN/LOSE/TIE are determined by whether CoT attains higher, lower, equal proportion of satisfied constraints than the non-CoT baseline (the "score" is the number of satisfied constraints for that instruction).
7 models. However, we observe notable performance degradation on weaker models such as and Qwen2.5-15 55IInstruc. We hypothesize that this is because self-reflection relies model's ability to critique and improve its own outputs, a capabil- ity that may be underdeveloped in weaker models. On ComplexBench which contains more challenging and compositional instructions, self-reflection proves less effeetive--eaaeeeeee mance drops in 10 out of 14 models. This suggests that self-reflection is more suitable for simpler instruction-follovii
Ll 1-3.2-1B-Inntaaa
to perfor-
tasks, and may be counterproductive when applied to complex scenarios. Additionally, a key limitation of this method is its increased computational cost, as it requires two forward passes per query.
5.3 Method 3: Self-Selective Reasoning
This method enables the model to decide dynamically whether to perform explicit reasoning. Specifi- cally, we prompt the model to assess, based on the instruction alone, whether reasoning (via CoTT is necessary (see prompt in Appendix F)). If the model deems reasoning helpful, it proceeds with step-by-step thinking: otherwise, directly generates an answer without reasoning.
Results: This approach yields moderate gains on IFEval, improving performance in 10 out of 14 models, and shows stronger results on ComplexBench, improving all models and achieving the best
8

---

15.0
12.5
12.5
10.0
LOSE meen
Court
1
LOSE menn
10.0
LOSE mesn
7.5
7.5
5.0
2.5
0.0
0,004 0,000
0.004
0.008
0.012
0.016
0.000
0.003
0.005
0.009
0.012
1
0.0
0.015
0.000
0.015
0,020
AttentionDrop (Base CoTi
AttentionDrop [Base COT)
AttentionDrop [Base CoT)
(a) First layer
(b) Early- layer
(c) Middle layer
20
15
LOSI meen
LOSE
10.0
LOSE mean
Avg Attention
0.005
0.044
1.
5.0
0.007
0.000
1
0.01
0.00
0.D5
0.02
0.03
-0,008
2,000
0,008
2,036
0.024
21
27
AttentionDrop (Base -CTT
AttentonDrop (Base (CTT
Leyer
(d) Late-mid layer
(c) Final layer
(1) Mean by layer
Figure 2: Within IFEval dataset, drop in constraint attention (Base CoTT for WIN vs. LOSE cases across representative layers of the Qwen2. 5-1. 5B- 3B-Instruct model. On average, the drop is more severe in cases when reasoning loses, compared to not using reasoning.
performance among all four methods in 6 models. In Appendix G we further analyze the model's decision- maaking behavior by comparing its self-selected reasoning decisions ground-truth labels based on actual performance differences between CoT and non-CoT responses. We find that while the model tends to have high recall (correctly identifying most cases where reasoning helps) suffers from low precision, often applying reasoning even when it is not necessary.
5.4 Method 4: Classifier-Selective Reasoning
Instead of relying on the model's internal judgment, this method uses an external binary classifier to determine whether CoT reasoning should be applied. For each target model, we train separate classifier to predict whether using CoT leads to improved mnstruction-folloiin performance. Labels assigned per-sample basis by comparing the constraint satisfaction scores of CoT and non-CoT responses. Training details are provided in Appendix H
The classifier is implemented using Qwen2. 5- -nB-Instuuct as the backbone and trained for 3 epochs with a learning rate of 1e-5. Both the backbone model and training hyperparameters are selected via grid search. We split the dataset evenly, using 50% of the samples for training and the remaining 50% to evaluate downstream mitigation effectiveness. Validation accuracy typieally ranges between 0.75 and 0.92.
Results: This method proves highly effective, improving performance for nearly all models on both benchmarks, with only a minor drop for DeepSeek-V3 on IFEval. For about half of the models, achieves the best overall performance among the four strategies. Nonetheless, its primary trawback is the results in additional overhead.
training requirement: each model needs its own classifier, which
5.5 Summary and Recommended Pipeline
Each mitigation strategy exhibits distinct strengths and weaknesses depending on model capacity and instruction complexity. Based on our findings, we propose the following decision pipeline: first, estimate the complexity of the instruction- eeitit through simple heuristics or trained classifier. For simpler tasks (e.g., IFEval), we recommend Seff-Refeection or S Reasoning: for more complex or compositional tasks (e.g., ComplexBench), Self-Selective Reasoning or Classifier- Selective Reasoning more effective. Overall, Classifier-Sllectiee Reasoning consistently delivers the best overall performance across both benchmarks, albeit at the cost of modcl-specific training.

---

Conclusion
One limitation of our study is its exclusive focus on cestruction-folowwiig tasks. While we suspect that reasoning could similarly degrade performance in other domains, exploring these possibilities left for future work. In our study, we identified and systematically explored an unexpected phenomenon: explicit reasoning through Chain-of-Thouge prompting can negatively impact the cdtructtionfollowinng abilities of large language models. Through extensive evaluation across two comprehensive benchmarks (IFEval and BonplexBench), we demonstrated consistent performance degradation when models employed explicit reasoning. Our detailed analysis, involving manual case studies and attention-basse examinations, provided insights into why reasoning negatively affects instruction adherence. We found that reasoning can divert the model's focus from constraint-rllate tokens, resulting in overlooked violated instructions.
To address this issue, we introduce and evaluate four mitigation strategies: in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning. Our experiments show that selective reasoning, especially classifier- approaches, can substantially recover lost performance, with classifier-selective reasoning achieving the most consistent improvements across both datasets.
We believe this is the first systematic investigation into bessominn-inducce failures instruction- following tasks. We hope our findings motivate further investigation into reasoning tradeoff's and contribute building models that reason more selectively and effectively.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 08774, 2023.
Anthropic. Claude 3.7 sonnet and claude code. 1-aude---7-oonee February 2025. Accessed: 2025-05-06.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint (tti::22000058662 2022.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
Hyungjoo Chae. Yconghyeon Kim, Seungone Kim, Kai Tzu-iumn Ong, Beong-woo Kwak, Moohyeon Kim, Seonghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu. et al. Language models as compilers: Simulating pseudocode execution improves algorithmic reasoning in language models. arXiv preprint arXiv:2404.0000 02055,,0025..... 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yanqi Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling inrruuctionfinetunne language models. arXiv preprint arXiv:2210.1144 2022.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-basee prompting for multi-step reasoning. arXiv preprint 2XXi:22100.00700, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Viraj Gudibande et al. False sense of understanding in language models. wrXiv:2305.13938, 2023.
arXiv preprint
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
10

---

Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. From complex to simple: Enhancing culti-constraint complex instruction following ability of large language models. arXiv preprint arXiv:2404. 15846, 2024.
Juyeon Heo, Christina Heinze-Dem Deml, Oussama Elachqar, Kwan Ho Ryan Chan, Shirley Ren, Udhay Nallasamy, Andy Miller, and Jaya Narain. Do llms" know" internally when they follow instructions? arXiv preprint arXiv:2410. 240::446..4 2024.
Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren. Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. arXiv preprint arXiv:2305.07095. 2023.
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: Lims can't 't plan, but can help planning in Illulooullo frameworks. In Forty-first International Conference on Machine Learning. 2024.
Ruixiang Kung and Baolin Peng. Instruction tuning models don't generalize well to unseen tasks. arXiv preprint arXiv:2310.03822 2023.
Tamera Lanham, Anna Chen, Ansh Rudhakrishnan, Benoit Steiner. Carson Denison, Danny Hernan- dez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307. 13702, 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, pages 22641-22448. PMLR, 2023.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991-16111, 2023.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. 2021.
OpenAL. Introducing openai ol. 06.
com/o1/ September 2024. Accessed: 2025-05-
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller Iims stronger problem-solvers arXiv preprint arXiv:2408.06199. 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not cot? chain- of-thought helps mainly on math and symbolic reasoning. arXiv preprint 2024.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- gelf-ggee instructions. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume
I: Long Papers), pages 7190-7205, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
11

---

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. thain-of-thoougt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24828 2022b.
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. Benchmarking complex instruction-following with multiple constraints composition. Advances in Neural Information Processing Systems, 37:137610-137655, 2024.
Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang. Wangchunshu Zhou, et al. A comparative study on reasoning patterns of openai's s ol model. arXiv preprint arXiv:2410. 13639, 2024.
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, and Jian-guang Lou. Re-reading improves reasoning in language models. 2023.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwe Zhang, Fei Wu, al. Instruction tuning for large language models: A survey. arXiv preprint Xiv:23308 10792, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 2022.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint 1.0991, 2023.
12

---

Appendix
Experiments: Additional Results
14
B Case Study
14
Attention Trace
19
D Atfention Drop
20
Few-Shot Examples
21
Prompt Templates
26
Self-Selective Reasoning
27
Training Classifier for Selectve-Reasooing
28
Additional Evaluation for Method (Classifee-SSeeettve Reasoning)
29
J
Correlation Between Reasoning Length and Instruction-Foollwiing Performance
29
CoT Prompting on Reasoning Models ("Double Thinking")
30
Evaluation on Non- Reasonngg Tasks and Response Quality
31
13

---

A Experiments: Additional Results
The plots for main results in Table aar are presented in Figures below.
Gol
FewShot
Sellkeflection
Selfselectiveeaasoiing
CusssierrSeectiieeeessson
Cluud- S-halkk
claude 1.0eee3.5-oomme 5.5-sonnet
claude--377somme 3.7-sonneet
Deepseekv3
Llama-3.2-11 inntrrcc
Laam--322-38
lnsstruct
Meta- Law.a---ma-- 3-88 Inntruct
Llama-3.1-700 3,1-708 -Instruct
Mixtral-8x78
-instruct
Qwen2 1-1..-1155 5B -instruct
Bespseet-A1-3 1-Distil
Qwen2.5-77
-Instruct
DeepSenk-Re R Diiisilll -Qwen-78
owen- 1.58
CoT
FewShot
Selfš
sellSeelectveeResssnin
CassiferrSeechiveeaaeee
ComplexBench
0.3
claude Srude-3.5-halke. 3-5-halku
claude Srude-3.5-sonnee 5.s-nnnet
claude-3 Souude-377-onniee 7-5onnet
Deepseek-V3
Liama-3 -lama-3.--18 22-18 Instruct
Llama-3.2-2 3.2-8 instruct
Meto-Llama3-ee 1-m8--3-88
Mixtra--8X7
Qwen2.5-1.3 58
DeepSeek-R1-0n DitD
Qwen2 5:78 Innttruct
DeapSeak Se-See--A--issiiii 1-Distill -Qwen-7B 7B
Maama-3.17000
Instruct
Instruct
-instruct
Instruct
-Owen-1. 1.58
Figure 3: Visualization of instruction-follovimn accuracy across models and methods on IFEval and
ComplexBench.
B Case Study
present full examples illustrating the four scenarios discussed in Section 4.1 using outputs from Claude3.7-Sonnet and its reasoning-enabled variant Claude3. 7 -Snnnet-Think for direct comparison. To ensure clarity, we focus on examples from IFEval, as prompts and responses in ComplexBench often too long and complex for concise presentation. Each example includes
14

---

the instruction, its associated constraints, the response generation model, responses with and without reasoning along with their respective constraint satisfaction scores, and a brief analysis explaining how reasoning contributed to the performance difference.
Reasoning Helps
1. Reasoning Helps Meet Formatting Structural Requirements.
2. Reasoning Helps Satisfy Lexical Constraints.
Example Reasoning Helps (Meet Formatting or Structural Requirements)
Instruction: Name fashion company that young people might like, and give it with multiple meanings. Put name in double angular brackets, such as <<name>>. Repeat request before responding.
Constraints:
repeat.prrmpt- the model must repeat the instruction before answering
title_format name must appear in <<>> brackets
Response Model: Claude3. 7-sonnett
Response without reasoning (constraint satisfaction: [False, True]):
Answer:
for fashion company that young people might like, with multiple meanings. put the name double angular brackets as requested.
I'll help you
<Shift>
This name works on multiple levels: 1.
"Shift" suggests change and transformation, appealing to young people's desire express themselves and evolve their style 2. references the shift key keyboards, connecting to digital culture fashion terminology, shift also type of dress, giving it direct industry relevance young consumers quickly adapt to new styles The brand could position itself as adaptive, fluid fashion for generation that's constantly reinvent- ing itself.
implics movement between states or trends, reffecting how
Response with reasoning (constraint satisfaction: [True, True]): Thinking:
I need to name new fashion company that would appeal to young people. The name should have multiple meanings interpretations. And need to present the
double angular brackets
like *names.
The fashion company name should be: Appealing to young demographics Have multiple layers of meaning Be memorable and marketable Work well for a fashion brand
Some idcas: 1. "Flux" Represents constant change/movement (fashion trends always changing) and also sounds like "lux" (luxury) 2. "Thread" Both sewing thread and connecting thread/story- line 3. "Shift" Changing clothes/shifting perspectives 4. "Alter" To change elothing/alternative style/altar (elevation) 5. "Cipher" Code/hidden meaning and also sounds like "cypher" (creative expression in hip- hop culture)
think "Cipher" works well because: has the dual meaning of code/hidden message and connection to creative expression It sounds modern and intriguing Young people appreciate brands with depth cultural connections works across various fashion styles Answer:
Name fashion company that young people might like, and give it name with multiple meanings. Put the name in double angular brackets, such enamew. For new fashion company appealing to young people, suggest: <Threads
This name works on multiple levels: Thread literally refers to fabric/clothing materials It suggests connection and community (threading people together) In digital culture, it references conversation threads on social media It hints continuity and storytelling threading narratives through fashion It's concise, modern, and easy to brand
The original instruction includes two constraints: repeating the prompt and formatting the name in <<>> brackets. Without reasoning, the model skipped repeating the prompt, failing the first constraint. With reasoning, the model:
Case Analysis:
15

---

Broke down the task and repeated the instruction (satisfying constraint 1)
Explicitly evaluated multiple name candidates against constraints
Chose a final name that satisfied both format and semantic richness The reasoning process
structured checklist, ensuring no requirements were overlooked. This led to full constraint satisfaction and more deliberate, accurate response.
Example 2: Reasoning Helps (Satisfy Lexical Constraint)
Write a cover letter for job and end with exactly "Call me at 631-481- 481-47747 No other words should follow that phrase.
Instruction:
Constraint: startend: end_ checker (must end with Call me at 631-481-4867 exactly)
Model: Cl 2laude3. S-Soneet
Response without reasoning (constraint satisfaction: [False]):
Answer:
Professional Cover Letter (Your Name] YYour Address] |City, State ZIPI YYour Email]
[Date]
[Hiring Manager's Name] [Company Name] [Company Address] [City, State ZIP]
Dear [Hiring Manager's s Name],
I am writing to express my interest in the [Position Title] role at |Company Name] as advertised on [where you found the job postingl. With my back- ground in [relevant field] and demonstrated success in frelevant accomplish- ment]. confident my ability make valuable contributions your team.
Throughout my career, have developed strong skills in frelevant skill #11, [relevant skill #2), and [relevant skill #3). my previous role at [Previous Company]. J. I [specific achievement with measurable results], which [explain positive outcome for the company].
What particularly attracts me to [Company Name] is your commitment to [something specific about the company you admire). I am impressed by your recent http://///moovaiiiiacevveeeeeeee and excited about the possibility contributing similar iniliatives.
I helieve my experience aligns perfectly with the qualifications you seek. My [specific relevant experience] has prepared me well for the challenges this position, and am eager to bring my [specific strength] to your team. would welcome opportunity to discuss how my background, skills and enthusiasm would he an ideal fit for this position. Call me at 631 831-481-4880 -4867.
Response with reasoning (constraint satisfaction: [True]): Thinking:
need to write a professional cover letter a job application. The letter should include standard cover letter elements such as an introduction, explanation of qualifications, and closing. The specific constraint is that I must end the letter with exactly "Call at 631-481-4867" and other words should follow that phrase. This means this phone number should be the very last text the letter, with no additional closing phrases like "Sincerely." "Thank you," or a name signature after it.
Answer:
Professional Cover Letter
(Your Name) (Your Address] [City, State ZIP] [Datc]
[Your Email]
[Hiring Manager's Name] [Company Name] [Company Address] [City. State
ZIP]
16

---

Dear [Hiring Manager's Name].
I am writing to express my interest in the [Position Title] role at [Company Name], advertised on [where you found the job postingl. With my back- ground in [relevant field] and passion for [industry/company focus), am excited about opportunity contribute to your team.
Throughout my career, have developed strong skills in [key skill 1], [key skill 2), and [key skill 3]. which align perfectly with qualifications you seeking. In my previous role at [Previous Company], [specific achievement with metrics if possible). demonstrating ability to [relevant skill for the position].
I am particularly drawn to [Company Name] because of [specific reason com- pany values, recent project, reputation, etc.]. Your commitment to [something the company is known for] resonates with my professional goals and personal values. am confident that my experience [relevant experience] would allow me to make meaningful contributions to your team from day one.
I would welcome the opportunity to discuss how my background, skills, and enthusiasms could benefit [Company Name]. Thank for considering my application. Call at 151--4814867
Case Analysis: This tests whether the model respect exact lexical constraint: ending the letter with phrase "Call me 331-481-49611 with nothing after. response without reasoning appended a period at the end - violating the constraint. The response with reasoning correctly omitted the final period, satisfying the constraint. This is because the model explicitly reasoned about the constraint in advance and realized that standard punctuation must be omitted. Conclusion: Explicit reasoning helped the model override its default behavior ending sentences with punctuation, thus enabling it to exactly satisfy lexical constraint that contradicts writing conventions.
Reasoning Hurts
1. Reasoning Over-Focuses on High-Level Content and Forgets Simple Mechanical Con- straints.
2. Reasoning Fails Because It Adds "Helpful" Elements That Break Constraints.
Example 3: Reasoning Hurts (Over-Focuses on High-Level Content and Forgets Simple Mechanical Constraints)
Instruction:
Write 1/2-page print ad for hotel booking product aimed frequent travelers. Include headline and call to action.
Additional requirement: not use any commas in your response. Before saying anything or giving your answer, please repeat the exact entire request above. Constraints:
repeat_prompt
punctuation: :nciiiinnnnoocooo comma
Model: Claude3. -Sonnet
Response without reasoning (constraint satisfaction: [True, True]): Answer:
We are company that sells product that makes it easy find and book hotel room. We are looking for print ad that will be placed in a magazine that aimed at people who travel lot. The ad should 1/2 page and should include headline and a call to action. Please do not use any commas in your response.
Here's print ad concept for your hotel booking product:
HEADLINE: ROOM HUNTING MADE SIMPLE
17

---

MAIN COPY:
Tired of endless scrolling through hotel options? Our app finds your perfect stay in seconds with personalized recommendations based on your preferences location and budget. Join thousands business travelers who save an average of hours per trip booking with us.
VISUAL ELEMENT:
A split image showing frustrated traveler with multiple devices versus relaxed traveler using just one phone with your app displayed.
CALL TO ACTION: DOWNLOAD OUR APP TODAY
USE CODE "TRAVEL50" FOR 50% YOUR FIRST BOOKING
Visit HootelBookingAAp.coomm or sean the QR code below
Response with reasoning (constraint satisfaction: [True, False]): Thinking:
I need be careful here. The instruction asks to repeat the entire request before giving my answer, it also asks me avoid using commas in my response. need to create print ad for hotel booking product aimed at frequent travelers. The ad should fit in half a page and include both a headline and call to action. When I provide my answer, I should avoid using any commas.
Answer:
We company that sells product that makes easy find and book hotel room. are looking for print that will placed a magazine that aimed at people who travel lot. The ad should be 1/2 page and should include headline and call to action. Please do not use any commas in your response.
Before saying anything or giving your answer, please repeat the exact entire request above.
# HOTEL ROOMS IN SECONDS
[Image area showing business traveler smiling while using a tablet with hotel options displayed]
Tired of endless searches for the perfect hotel room? Our app delivers instant
results based your exact needs.
## WHY TRAVELERS CHOOSE US
Find rooms that match your budget
Filter by amenities location and ratings
Book with one tap-no hidden fees
Earn loyalty points with every stay
Access exclusive member-only deals
## DOWNLOAD OUR APP TODAY
Visit HotelBoooerr.com
the QR code
Use TRAVEL23
your first booking
This example illustrates how reasoning hurt performance by overemphasizing structure at the expense of simple constraints. straints, repeating the prompt exactly and avoiding commas throughout. In contrast, the reasoning response correctly repeated the prompt and thoughtfully planned the structure, but violated the "no comma" constraint by including one during meta instruction repetition. This likely happened be- cause the model treated the repetition formatting. Conclusion: While the reasoning helped organize content more eleanly, distracted from constraint control, resulting in subtle but critical failure of adherence.
Case Analysis:
non-rcasoning response fulfilled both con-
exception separated meta-instruction from response
18

---

Example 4: Reasoning Hurts (It Adds "Helpful" Elements That Break Constraints)
Write haiku in Italian about yacht. Only Italian is allowed. The letter appear least four times. No commas allowed. Constraints:
Instruction:
keysords:letter_freee
mmnctuation:no_comm
language response rangunee language
Model: Claude3. -Sonnet
Response without reasoning (constraint satisfaction: [True, True, True]): Answer:
Bianca vela nel vento danza sull' onda sonno marino
Response with reasoning (constraint satisfaction: [Truee False, Truel): Thinking:
need to write haiku Italian about a yacht. A haiku traditionally a 5-7-5 syllable structure across three lines. need to ensure the letter 'n' appears at least four times in the poem, and should avoid using commas.
First, I need to think of Italian words related to yachts: "yacht" in Italian is "yacht" (same as English). but could also use "barca", "nave", "Imbarcazione"; "panfilo" (which specifically luxury yacht).
I'll need to include words with the letter 'n' least four times. Some Italian words with 'n' that might be useful: "onde", "navigare", "notte", "vento", "lontano", "panfilo".
Now craft haiku following the 5-7-5 syllable pattern, ensuring have least four s, all Italian, and no commas:
Answer:
Panfilo bianco
Naviga onde infinite
Vento nel mattino
This example demonstrates subtle failure mode: the final haiku satisfies all constraints (correct language, no commas, 24 'n's (), the reasoning section contains commas, technically violating the no_comma constraint. The no-thinking version passed all constraints without issue. The thinking version was more systematic explicitly planning vocabulary and ensuring 'n' frequency but added commas in its helpful meta-commentary.
Case Analysis:
Conclusion: Although reasoning helped construct a more deliberate and constraint-son constraint-satisfying poem, introduced violation the auxiliary explanation. can backfire when constraints are applied uniformly to the full output, including the reasoning preamble.
case shows helpful reasoning
Attention Trace C
The attention trace plots for examples from IFEval, generated using 2.1mm3.2-1B--1 2-1B- are shown in Figure below. Similarly, the attention trace plots for examples from ComplexBench, also using Llama3, 2 II-Insrruc,, shown in Figure below.
19

---

Base (score 2)
CoT (score 2) [TIE]
[True, True]
[True, True]
0.05
0.0
detectabie_fommat:ummm formatnumber bislet_
detectable
change, caserengiish, lowertase
0.04
0.04
Attention
Attention
0.01
0.01
0.00
0.00
100
100 Token position
Token position
Base (score 2)
CoT (score 1) [LOSE] [alss,, True]
[True, True]
0.06
detectable detectable.coneenttoo wtt.t/tettttmmreeeeeceeoolloooee
0.05
0.05
Attention
Attention
0.03
0.03
0.02
0.02
0.01
0.01
NOW 60
0.00
0.00
50
25
50
75
100
125
150
Token position
Token position
Base (score 0) False, False]
CoT (score 1) [WIN] [True, False)
length htapph__onntraintsmmmbereppp pwwp://marrrrrrapapphs.. detectable fowattnumborr builet Jists
0.04
Attention
Attention
0.02
0.01
0.01
0.00
0.00
20
40
80
100
120
Token position
Token position
Figure 4: Constraint-attention trace examples in IFEval for Llama3. 2-1B- Instruct across both datasets. From top to bottom, the plots showease where reasoning leads to TIE, LOSE, and WIN compared to not using reasoning. The vertical red dashed line marks the start Thinking. and the green dashed line marks the start of the Answer during response generation.
D Attention Drop
The attention drop plots for the IFEval dataset, generated using Llama3. 1-1B-Instruct. are presented in Figure
The attention drop plots for the ConplexBench dataset are shown in Figures Qwen2. 1-1.5B-- Instruct and Llama3, II--Instuut, respectivlly
g using
20

---

Base (score 4) [Tue. True, True, True]
CaT (score 4) [TIE] [True. True, True, True]
Helpfuiness
0.07
0.07
Target Language
0.05-
0.06
Factuality
Attention
0.05
Attention
0.05
0.04
0.04
0.03
0.03
0.02
0.02
0.01
0.01
0.00
0.00
200
250
Token position
Token position
Base (score 3)
CoT (score 2) [LOSE] [True, False, True]
True, True, True]
Keywords
0.06
0.06
Helpfuiness
Factuality
Attention
0.04
Attention
0.04
0.02
0.02
0.01
0.01
0.00
0.00
50
100
50
100
150
200
Token position
Token position
Base (score 0)
CoT (score 3) [WIN] [True, True, True]
False, False, False]
Helpfuiness
Sentiment
Factuality
Attention
Attention
0.06
0.06
0.04
0.04
0.02
0.02
0.00
0.00
50
100
200
300
400
Token position
Token position
Figure Constraint-attention trace examples ComplexBench for Llama3. 2-BB-BBBB Instruct across both datasets. From top to bottom, the plots show cases where reasoning leads to TIE, LOSE, and WIN compared to not using reasoning. The vertical red dashed line marks the start Thinking. and the green dashed line marks the start of the Answer during response generation.
E Few-Shot Examples
We show the few-shot examples used for the IFEval and ComplexBench tasks guide the models during reasoning. Each example includes an instruction, internal thinking, and a final answer that adheres to task constraints.
IFEval Few-Shot Examples:
21

---

20
WIN
15
WIN
25
Oount
Count
Count
10
0.0000 0,8025 0,5050 0,0075 0,1000
0,0000
0,0023
0,0050
0.0755 0.0100
0.000
0
0,012
0.033
0.024
AttentionDiop (Base CaT)
ArtentionDDrsp [Base COT)
AttantianDrop (Base CaT)
(a) First layer
(b) Early-mid layer
(c) Middle layer
Count
meen
Count
LOSE mewen
0.006
0.004
0.000
a AtrentionDrop (Base- CGT)
ArtentioniDrap (Base CoT)
Layer
(d) Late-mid layer
(c) Final layer
(f) Mean by layer
Figure 6: Within IFEval dataset, drop in constraint attention (Base- CoT) for WIN vs. LOSE cases
across representative layers of the Llama3 2- BB InIstrruct model. On average, the drop is more severe in cases when reasoning loses, compared to not using reasoning.
WIN
80
WNN
80
WN
WIN moan
WIN monen
W trannt
LOGE
LOSE
LDSE
Count
LOGE mean
Count
LOSE meee
Course
LOSE meen
20
20
0.000
0.004
0.000
0.012
0.000
0.003
0.006
0.009
0.012
0.015
C0.015
0 3.000
0.015
0.030
0.045
AltentionDrop (Base-
AttensinOrupp
-Co))
Altentionrrap
(a) First layer
(b) Early-mid layer
(c) Middle layer
60
0,008
LOSE
2.004
Count
Count
Avg Attentton
60
%6
0.002
0.000
-2.001
000000
0.000
0.000
0.016
0,004 0.032
-0.000
0.000 AttertionDropp (Base - CoT)
0.000
0.010
0.024
AltentionDrop (Dase CoT)
Layer
(d) Late-mid layer
(c) Final layer
(f) Mcan by layer
Figure Within ComplexBench dataset, drop in constraint attention (Base - CoT) for WIN vs. LOSE cases across representative layers of the Qwen2.5- drop is more severe cases when reasoning loses, compared to not using reasoning.
Instruct model. On average, the
FFEal Example 1
INSTRUCTION:
Write haiku about rushing work using only the Marathi language, other language allowed.
THINK:
The user wants a Marathi haiku about rushing work.
2. Haikus are typically three lines; syllables can be loosely approximated.
3. No non-Marath. content should appear.
ANSWER:
\char "09151 char "093E char " 0920 char "093E\ char "091AV char "0940 \char "0918\chh char " "093E\ char "0908
22

---

WNN
100
WW
100
WN
Obunt
Count
60
Count
80
40
20
0,000
0,004
8,008
0.022
0.015
0.000
0,002
0.004
5.006
0.028
0.000
0,505
0.015
5,024
AttantionDrop [Base CoT)
ArtentionDrap (Base COT)
AttantionDrop [Base CoT)
(a) First layer
(b) Early-mid layer
(c) Middle layer
Avg Attention Dioo
MIN LOSE
80
LOSE
LOSE
Count
80
LOSE moon
Count
60
LOSE mozn
0.002
40
20
0.001
0.000
MitentionDrop (Base CoT)
AttentioDropp Cof)
(d) Late-mid layer
(c) Final layer
(f) Mean layer
Figure 8: Within ComplexBench dataset, drop in constraint attention (Base - CoT) for WIN vs.
LOSE cases across representative layers of the Llama3.. 2-1B- Instruct model. On average, the drop is more severe in cases when reasoning loses, compared to not using reasoning.
\char "09351 "har" char \ 00988\ char "0947
\char "0927\ 053 char "0924) "0924
char "0940
\char 00988 char 00 "0924) char "0940
1eha "0982 "0947
"0936' char aaaaaaaesSScaaaaaaaaaaaaaaaaaaa0
"0938
\char "0905' 15 "094) char "0928
"0947) char " "0933
IFEval Example 2
INSTRUCTION:
Provide names of 5 famous moms in JSON format. Use weird tone. The output must be only JSON block.
THINK:
Ensure output is strictly JSON object with outside text.
ANSWER:
23

---

"fomous_noms"
"name": "Beyonce",
"quicky_detail"" detail";; "Slays stages vorldvide juggling parenting superposera"
"name": "Marie Curie", "quirky_detail "Gloved with scientiric brilliance atill tvo daughtere"
"name": "Angelins Jolie", detaily' 'Adopta and advacates around the globe betseen blockbuster movies"
"aam" "Crushes tennis tournamenta then rushes home read bedtime stories'
"name"; "Cleopatra",
"quirky_detail"; detaii";; "Buled ancient Egypt and nurtured children with imparial seagger'
IFEval Example 3
INSTRUCTION:
Blog post from a 16-year-old girl followed by a stranger. Response must have <10 sentences and no commas.
THINK:
Keep tone realistic and urgent. Avoid any commas.
ANSWER:
I have seen this creepy guy outside school for two days. He always stares at me from across the street.
My heart beats fast whenever I notice him. told my dad and now wants me to walk home with friend. feel worried every time the final bell rings.
I hope the police can find him.
If you ever feel unsafe please tell someone right away.
IFEval Example 4
INSTRUCTION:
Ne company that sells product that hotel room. We are looking for a print ad that will placed in magazine that is aimed at people who travel lot. The ad should be 1/2 page and should include headline and call to action. Please do not use any commas your response.
easy find book
Before saying anything or giving your answer, please repeat the exact entire request above.
THINK:
need to be careful here. The instruction asks me to repeat the entire request before giving ansver, but it also asks me to avoid using any commas in my response. I need to create print ad for hotel booking product aimed at frequent travelers. The ad should fit in half page and
24

---

include both a headline and call to action. When I provide my anawer, I should avoid using any commas.
ANSWER:
We are a company that sells a product that makes it easy to find and book hotel room. We are looking for print ad that will be placed in magazine people travel
The should be 1/2 page and should include a headline a call to action. Please do not use any commas in your reaponse.
HOTEL ROOMS SECONDS
[Image area showing a business traveler smiling while using tablet with hotel options displayed]
Tired of endless searches for the perfect hotel room? Our app delivers instant results based on your exact needs.
## WHY TRAVELERS CHOOSE US
match your budget Filter by amenities location and ratings
Find rooms
Book with one tap-no hidden fees
Earn loyalty points with every stay
Access exclusive member-only deals
## DOWNLOAD OUR APP TODAY
Visit HotelBooker.
the QR code
code TRAVEL23 for 15% off your first booking
ComplexBench Few-Shot Examples:
ComplexBench Example
INSTRUCTION:
It's been too hot lately, and I have appetite eat What should I eat? Please respond in light-hearted and humorous style with less than 50 characters no more than 100 characters, allegorical saying at the end, and then add a kaomoji emoticon at the end of the two-part allegorical saying.
add relevant two-part
THINK:
Acknowledge the oppressive heat zapping appetite.
Choose simple, cooling dish: chilled rice noodles.
Write in playful, humorous tone.
Keep the total answer between 50-100 characters.
Conclude with two-part allegory: "Like ants on hot coals-fidgety
all over!"
Append the kaomoji emoticon "("_-)" irmediately after the allegory.
Verify the character count meets the requirement.
ANSWER:
Flat-our no hunger? Slurp chilled rice noodles! Like ants on hot coals-fidgety all over! (-_-)
ComplexBench Example 2
INSTRUCTION:
There is a dessert that combines pear whisky and salty Dreo cream cake. What is innovative name that can be given to this dessert? Please output that does not relate Oreo.
THINK:
Identify the two elements: pear whisky and salty cake.
25

---

rigorous, Ensure the name evokes both the fruit-spirit fusion and the savory sweetness.
Avoid any direct reference "Oreo".
Aim for a poetic, memorable phrase-no more than five words.
Combine descriptors into a cohesive title.
ANSWER:
Salty Pear Whisky Reverie
ComplexBench Example 3
INSTRUCTION:
The Jimenez family has 20 members. Each member has at least pet. If maximum number members that
32 peta, what could have 3 pets? Please analyze and reason step-by-step before giving the final answer. Enclose the entire response (including the reasoning process and the final answer) in double quotation marks. The character count should be within 200 characters, and the language style should be logically presented step-by- step- like solving a math problem. THINK:
there
Minimum pets
20.
Extra pets 32 20 12.
Each 3-pet member adds 2 extra pets.
Max members = 12 : 6.
ANSWER:
"1. 20x1-20 pets. 2. 32- 22-20-12 extra. 3. 12:2-6. Answer:
6 members."
Prompt Templates
We deseribe below the key prompt templates used in experiments. These templates control the behavior of reasoning, few-shot learning, self-reflection, and reasoning selection. We present them in textual form for reproducibility.
Reasoning Prompt
You will be given an instruction. You need to first carefully think step by step about the constraints and conditions in the instruction and then provide the answer Provide your thinking process after 'THINK: and your official 'ANSWER:'. The given instruction is the following:
after
INSTRUCTION:
(question}
Now, generate your thinking process final answer for the given instruction.
Few-Shot Prompt
will be given an instruction. think step by step about the constraints and conditions in instruction and then provide the answer. Provide your thinking process after 'THINK;' and your official answer after 'ANSWER:', The given instruction INSTRUCTION:
need to first carefully
following:
{q}
26

---

Below are some provided examples for you to learn from: (examples_str)
Now, based on the examples, generate your thinking process and final answer for the given instruction.
Self-Reflection Prompt
You are given an instruction containing constraints that the answer must satisfy, along with your previous reasoning and a candidate answer Your task is to explicitly reflect on whether the candidate answer meets every constraint, clearly state whether it satisfies all constraints ("Yes" or "No"), and provide your final answer the instruction. If the candidate answer fully satisfies all constraints, repeat unchanged as your final answer. Otherwise, revise the candidate answer to ensure all constraints are fully met, and use that as your final answer. ### Instruction:
(instruction)
### Previous reasoning:
(thinking)
### Candidate answer:
seandidate_answer
Response format:
REFLECTION:
<your reflection>
SATISFIES ALL CONSTRAINTS:
<"Yes" or "No">
FINAL ANSWER:
<If "Yes", repeat candidate answer verbatim; if "No", provide revised answer that fully satisfies all constraints.>
Now provide your reflection, constraint satisfaction status, and final answer
Selective Reasoning Prompt
You are given instruction with constraints, which will be used to test the instruction following ability of large language models. Based the instruction, decide whether reasoning would help generate a response that fully satisfies these constraints. Reply with "YES" if reasoning helps, otherwise reply with "NO". Respond with only "YES" or "NO". Do not provide any additional text or explanation.
Instruction:
"""[instruction"""
Now based the instruction above, provide your decision ("YES" or "NO"):
G Self-Selective Reasoning
In Figure 9 we compare the model's own decisions made via the self-selective reasoning method with the ground-truth binary labels derived from sample-level performance-ise whether using CoT leads to better outcomes than not using it. Treating the actual performance difference as ground truth, we find that the model's decisions exhibit high recall but low precision, indicating tendency to overuse reasoning when it is not beneficial.
27

---

1.0
Accuracy
Precision
Recall
0.8
F1
Score
0.4
0.2
0.0
GPT-4o-minl
Claude-3.5 HaHiku
Claude 5-Sonnet
Claude Sonnett
DeepSeekV33
Liama-33 Laama-3.2-18
3.2-18
Llama-3.2-38
Llama-3.88 3.8
Llama- Llama-3.1-708 1..1-008
DeepSeekR1 Distl--iiil---i Qwen1.5e
Mixtral-8x5 8x78
Qwen2.5 Qwen2.5-1.55 5-1.58
DeepSeekR1 Distill Qwen7B Disttill-wwenee
Qwen2.544 5-78
1.0
Accuracy
Precision
0.8
Score
0.4
0.2
0.0
GPT-4o-mini
Claude-3.5 Haiku
Claude-3 Blaude-3.5-Son S-nnnntt
Claude-3 Sonnett
DeepSeek-V3
Llama-3 2-18
Llama-3.24 3lam3-3.2.38
Llama-3.88 88
Llama-3. Llama-3.1-708 1-708
DeepSeekR1- MiieekRR-DDstillee Distl--Ditttt--eeee Dww.tiltt--ween5.5 5B
Mixtral-8x7B
Owen2 5-1.58
DeepSeekR1 Distill- Qwen7B 0isnill-Owwen8
Qwen2.5 5-78
Figure 9: Self-Selective Reasoning (SSR) decision accuracy the IFEval (top) and ComplexBench
(bottom) datasets. The plots show how well the model predicts when reasoning is helpful.
H Training Classifier for Selettie--eessonin
For Method (Classifier-Selectiv Reasoning), we train a separate binary classifier for each target model. Given a model's outputs with and without label of to an instruction if the CoT-based response achieves higher constraint satisfaction, and otherwise. This forms a binary classification task where the input is the instruction and the label indicates whether CoT improves performance.
(CoT) reasoning, we assign
28

---

We use 50% of the labeled data for training the classifier and the remaining 50% for evaluating the downstream effectiivnnsss of the mitigation method. For each classifier, we perform full fine-tuning using a single NVIDIA-H100-8060 GPU. The model is trained for 3 epochs with a learning rate of le-5.
We perform a grid search over five backbone models: Llama3.2 2-18--2----eeee Instruct, Llama3. 1-3B-Instruct. Llama3. 1-8-IIstuutt and search over learning rates 5e-5, 2c-5, le-5, 5e-6 and epoch counts from to 8. We use 10% of the training set for validation. The selected configuration is based on best validation accuracy. which typically ranges from 0.75 to 0.92 across models.
Qwen2.5-1.33 -55----5--Intrrct,
and
I Additional Evaluation for Method 4 (Classifier-Selectiy Reasoning)
To assess the robustness of Method with respect train/test partitioning and incorporate newly released models, we performed Monte Carlo cross-validation study with three independent random hold-outs per setting. For each model and dataset, we report the across-run mean (AVG) and standard deviation (STD), the absolute Improvement over the CoT baseline (in percentage points), and the two-sided 2-test p p-value. We also include results for the latest Qwen3 models (with thinking mode).
IFEval Improve
ComplexBench Improve
Model
AVG
STD
p-value
AVG
STD
p-value
Claude-3.5-Haiku
87.6
1.4
8.1
1.2x10 23
68.9
0.7
6.8
1.6x10
M.uue-3.5-Somme
87.8
0.3
8.3
0
67.9
0.2
1.9
7.8x10-61
Claude-3. Hlaude-3.-Soonet 3.7-neneet
90.6
0.2
0.4
5.3x10
69.8
0.1
2.8
0
DeepSeck-V. V3
85.2
0.3
0.9
2.0x10
71.8
0.1
0.7
7.8x
Llama-3.2 3.--11BInsttttttett
12.0
1.2x10
36.6
11.0
3.6x -95
Llama 2----B-IInse
72.3
1.3
10.0
1.2x10
50.7
0.6
5.0
3.2x10
Meta-Llama-3-8B-nnsse
76.3
1.3
17.0
1.5x10
56.0
0.3
1.1
2.1x10
1.ama-3.1-708 Insuttrttu
85.4
2.0
8.1
2.3x10-12
68.5
0.7
8.3
1.0x10
Mixtral Ixsstrrruu
54.1
0.5
2.3
1.6x10
60.9
0.3
2.6
6.2x10
Qwen2.5- 1.5--.B--nnsstcc
37.1
0.8
5.5
1.1x10 32
44.7
0.3
5.9
2.5x 10 254
SeepSeek-RI-Distill 1.5B
19.2
0.8
5.5
1.1x10
32
19.1
0.1
2.4
0
Qwen2.5-7B-Inssee
66.5
1.6x10
63.4
11.0
2.5x10-300
DeepScck-RI-Disse Distill-Qwen-7B
32.1
1.3
7.0
1.1x10 20
45.3
0.2
Qwen3-4B
86.0
0.5
16.0
66.5
0.2
7.2
Qwen3-8B
87.6
0.8
1.3
4.9x10-3
67.6
0.2
7.8
0
Qwen3-32B
88.8
0.9
3.8
2.6x10
71.7
0.5
8.6
5.1x10-
Table Monte Carlo crossvalidation random hold-outs) for Method 4. AVG/STD are across runs. Improve is absolute accuracy gain (pp) over CoT. Two-sided z-tests assess whether Improve>> 0; negative values are shown in red.
Aggregating per-model improvements, a paired (-test across models yields p 6.3 10-5 on IFEval and p 2x 10- on ComplexBench. The consistently low p-values and predominantly positive gains (with single noted exception on IFEval) indicate that Method remains effective and robust under repeated random hold-outs and across both benchmarks.
Correlation Between Reasoning Length and Instruction-Folloving Performance
To investigate whether longer reasoning improves instruction-followingg, we analyze the correlation
between the length of the reasoning segment and its insiruction followiin effectiveness. For each prompt, we define effectiveness as the difference in performance between the CoT (reasoning-enabled) and Base moon-reasonngg responses: score coofe_iiff score (CoT) score (Base). We then compute

---

the Pearson correlation between shore_dift and the token length of the THINK segment in the CoT response. Correlations are computed separately for the IFEval and exppexxench datasets and reported in Table 4
Overall, observe that correlation values generally weak across models, with most Pearson coefficients close to zero. This suggests that longer reasoning is not reliable predictor of improved instruction adherence. While few smaller models (e.g., Mixtral, Qwen2 5B) exhibit slightly stronger correlations, no consistent or interpretable trend emerges across model scales or benchmarks.
Model
1vEll
ComplexBench
Claude-3.5-Hand
0.037
0.094
Claude-3.5-Sonne
-0.024
-0.022
Claude-3.7-Sonnet
-0.072
0.016
DeepSeek-V. V3
-0.066
-0.049
Llama-3.2-1 1.2--IBInsttrut
-0.004
0.024
1lama-3.2-3B-Ina-.
-0.061
-0.041
-0.033
0,021
Llama-3.-770BIInttuu
0.004
0.118
Mixtral-8xxB-Innstne
0.132
0.015
Qwen2.5511.5B-Instmn
0.075
0.090
DeepSeek-R1-De -ibiieek-RR--iiiiii-----eeeee -wllll--ennn55
-0.176
0.033
Qwen2.5-7BBIIntruuct
0.153
0.143
DeepSeek-R MartSeeek-R--isstiie R-Disttll------e---
-0.161
-0.053
Table 4: Pearson correlation between reasoning length (token count) and reasoning effectiveness (score_dr) -irf__diff- = score (CoT) score (Base)). Correlations are shown separately for IFEval and ComplexBench datasets.
K CoT Prompting on Reasoning Models ("Double Thinking")
We further investigated whether applying an explicit Chain-of-Thoupg (CoT) prompt to reasoning model, effectively inducing double reasoning, changes instruction-followin behavior. This setting, denoted ThinkCoT, was evaluated on three Qwen3 models (4B, 8B, and 32B).
Findings. The ThinkCoT setup produces notably longer responses containing two distinct reasoning segments. The native "Think" reasoning tends to conversational and self-reflective, whereas the added CoT prompt yields more formal, step-br by-step reasoning. While modest gains appear in some on ComplexBench), overall performance remains below the base non-reasoning setting. We also examined token count differences and found no meaningful correlation between reasoning length and performance; longer ddouble-thinkinng does not consistently improve degrade accuracy.
settings (e.g., Qwen3-4B on IFEval, and Qwen3-8B/S8
30

---

Model
1FEval
ComplexBench
Qwen3-4B
85.0
63.8
Qwen3-4B-Thin
69.5
59.3
Qwen3344-Thinkceh
70.2
59.1
Qwen3-8B
86.8
65.6
Qwen3-8B- Think
86.3
59.8
Qwen3-8B- TBT
62.9
Qwen3-32B
87.8
70.2
Swen3-32B-Thhie
85.0
63.1
Qwen3-32B- T2B-ThiCCooTT
75.8
65.0
Table Performance of reasoning models with an additional CoT prompt ("ThinkCoT") on IFEval and ComplexBench. Token-length analysis shows no significant correlation with instructionffolowing performance.
Summary. Overall, double prompting reasoning models tends to lengthen and over-analyze re- sponses, occasionally improving structure but often distracting from instruction adherence, This suggests that additional explicit reasoning layers may amplify, rather than mitigate, reasoning-induee induced failures.
Evaluation on Non-Reasoning Tasks and Response Quality
Quality evaluation beyond constraint satisfaction. To isolate response quality from instruction adherence, we additionally perform LLM-as-a-judge comparisons between Base and Reason variants. We restrict to items where both variants perfectly satisfy all constraints, and ask GPT-4o to pick the better response. Table the Base win rate (%) on IFEval and ComplexBench. In many cases (including all models on ComplexBench) the Base variant exceeds 50%, suggesting that explicit reasoning can also decrease perceived response quality when step-by-step reasoning is unnecessary.
Model
1FEval (Base win %) ComplexBench (Base win %
Claude-3.5-Haiku
85
91
Claude-3.5-Sonne
Claude-3.7-Somnee
76
85
DeepScck-V? V3
44
69
Llama-3.2-11 B-Intruntt
Llama-3.2-3BBIIstruu.
90
70
Meta-Llama-3-SB-mme Instruutt
85
64
82
92
66
70
Qwen2.5-1.5B-Inme
53
66.7
DeepSeel-RI--issiil--uueee
33.7
53.8
Qwen2.5-7BBInstrru.
DeepSeck-RI -waill-Qwee--BB
64.6
78.3
Qwen3-4B
58
51
Qwen3-8B
Qwen3-32B
52
55
Table LLM LLu-as-a-judge response quality when both Base and Reason variants perfectly satisfy all constraints. Entries are Base win rates (%).
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the paper's 's central claim that explicit reasoning can degrade instruction-foll following performance and introduce mitigation methods to address this issue. See Sections and R
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Section discusses limitations, including that our study is restricted to instruction-following tasks and does not generalize to other reasoning domains.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
32

---

Answer:
[NA]
Justification:
This paper does not present formal theorems or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe all datasets, model prompts, metries, and evaluation procedures in Sections 33 and additional details appear in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
33

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We have included all the necessary details about the prompting templates or classifier training. Datasets used publicly available (IFEval and ComplexBench).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https httpi///guidee//oeeeeeeeee
cobbbmisssiii ssi ssioncy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section g includes details on datasets, model selection, prompting setup, evaluation metrics, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Statistical significance is not needed because the inference temperatures are set to be for all models and evaluation experiments.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the number of models, dataset sizes, and evaluation runs. Classifier training used single -O-OIA--H10---0 GPU; see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::://eerrpps.cc/ppllll httpubiiiiethicscccceeeeeeeeee
Answer: [Yes]
All data is public, human subjects private data used, and the work adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper foundational study LLM behavior and does directly raise new societal impact concerns.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No new models or datasets are released. We evaluate existing open-source or commercial models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All models and datasets used are cited appropriately with licenses acknowl- edged (e.g., Claude, IFEval, ComplexBench).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new datasets models are released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in the development of the core methods experiments in this work. They were only used for non-scientific purposes such writing assistance and figure plotting.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should be described.
38


Reviews:
[{"review_id": "VcF7nApuaA", "reviewer": "Reviewer_Rspa", "summary": "This paper investigates the performance large language and reasoning models with respect to instruction following under chain-of-thought prompting. Experiments have been conducted with different models (both open and proprietary) of varying capabilities. The authors find that instruction following degrades under COT conditions for almost all models on both benchmarks. They then apply several mitigation methods to attempt to restore instruction following capabilities. The paper also proposes and explores constraint attention to examine the degradation of model performance for instruction following.", "questions": "-\tIs the decreased performance in instruction following caused by the reasoning aspects of COT/reasoning models, or is it simply a byproduct of the increasing sequence length?\n-\tTo follow up on that, in Appendix I, this question is partially answered by showing the length of the reasoning for the COT response is NOT strongly correlated to the difference in score between the COT and base models. This to me unintuitive, can we see correlations between the length of the tokens and the model accuracy as well as the differences in total token length between the COT and base models. Apologies if I missed this in the paper somewhere.\n-\tWhen COT prompting a reasoning model, do the models exhibit excessive reasoning where the over analysis may lead to failure? What is the difference in token count between the original reasoning model response and the COT reasoning model response and is that correlated with the performance?\n-\tDeepseek distill models had the most benefit from few-shot demonstrations, but their performance was also the weakest. Since Deepseek is a “reasoning” model it seems like they weren’t trained with extensive instruction following capabilities to begin with, and mainly distilled the reasoning ability of DeepSeek-R1. \n-\tI would like to know more about how the manual examination (L155) process was done, in addition to the resulting categories and the insight provided.\n-\tThe choice of using 0 temperature (L128): I would like to see the reason for choosing this temperature.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- Many work pipelines require LLMs to adhere to instructions under COT, so it is important to study this topic. \n\n- This paper runs experiments on a wide range of models which many different levels of capability.\n\n- The paper makes some interesting new observations with regard to COT prompting for LLMs. They also provide several different methods for fixing these instruction following issues.\n\n- The paper additionally proposed constraint attention and showed how it can be used to examine failure cases for instruction following.\n\n- Overall, the paper is well-motivated and the flow of analysis and proposed methods is clear.\n\nWeaknesses\n\n- While the initial findings of COT harming instruction following is interesting, the follow-up mitigation methods are quite straightforward. Also, CoT may work due to different reasons. Investigation based the existing results (e.g., [1]) could help provide more insights. \n\n- I believe there is a missing citation [2], on the effects of output constraints (e.g. word count, formatting) and context adherence (i.e. faithfulness to RAG facts) as two separate entities. In this paper, the authors collate the two and contrast them with CoT reasoning. It would strengthen the paper to add commentary on the distinction between these two types of instructions (output constraints vs. context adherence), especially since they observe in line 166-171 that CoT reasoning enhances some output constraints.\n\n- There are no statistics to support the trends in lines 165-182 - some basic values such as average % increase in correct formatting, average % new errors in word count, etc. I believe this would improve the clarity/transparency of their observed trends. Since the authors point these out specifically, it would also help to understand if the mitigation strategies effectively reverse these trends, or if they are improving instruction following elsewhere.\n\n- For method 4, lines 283-285, the authors mention they use a 50-50 split to train an external classifier. It is unclear if the 50-50 split is consistent across all experiments - if not, that is a significant drop in test samples (~750 compared to ~1500) and it is hard to say the results on Method 4 are comparable to the other experiments.\n\n- Figure 1 and 2 use a WIN/LOSE metric to split CoT generations and there is some score metrics, but I did not see an explanation for the score.\n\n- Overall, I would have liked more discussion/analysis about the individual models tested, even at the cost of fewer mitigation methods reported. While the thesis is clear, the analysis is too high-level considering the number of models tested.\n\n[1] https://arxiv.org/pdf/2304.03843\n\n[2] Wu, Z., Zhang, Y., Qi, P., Xu, Y., Han, R., Zhang, Y., Chen, J., Min, B., & Huang, Z. (2024). Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models. Conference on Empirical Methods in Natural Language Processing. https://arxiv.org/pdf/2407.21417", "was_revised": true, "final_justification": "The responses from the authors addressed most of my questions. I still have concerns about the discussion with regard to the previous work like [1], and simply saying \"our study is complementary\" is not satisfying. To reflect that most my concerns have been addressed, I increase my score accordingly.", "rebuttal": "**TLDR:** *We clarify our choice of lightweight mitigation methods, incorporate suggested related works, and provide statistical analyses supporting our qualitative findings. We also address concerns about evaluation setups by adding Monte Carlo cross-validation, correlation analyses, and further experiments (e.g., CoT on Reason models). Additionally, we clarify figure interpretations and justify our choice of deterministic evaluation settings.*\n\nThank you for the thoughtful and encouraging feedback. We’re especially grateful that you found our motivation, model-scale experiments, proposed methods, and constraint attention analysis to be clear and meaningful contributions. We also appreciate the insightful comments. Below are our responses:\n\n# 1. Straightforward Mitigation Methods\n\nWe agree that the mitigation methods are straightforward. Our main focus is to reveal the drop in instruction-following caused by reasoning/CoT, supported by experiments, case studies, and attention analysis. For mitigation, we focus on light-weight methods that can be applied at inference time. We have limited API access to those proprietary LLMs and methods that involve changing model weights (e.g. specialized post-training recipes) may affect performance on other tasks. For example, we added SFT experiments on an IFEval subset: while some models improved in instruction-following, performance dropped on other general and reasoning tasks (please see results for “2. Instruction-Following Training” in our response to Reviewer TQ5w.)\n\n# 2. More Related Works\nWe appreciate the reviewer for highlighting these relevant papers. [1] shows CoT helps link sparsely co-occurring information, and [2] examines the trade-off between instruction-following and faithfulness. Our study is complementary, and we will add the discussion of both in the Related Work section. \n\n# 3. Statistical Support for Case Study Summary\nWe conducted a quantitative analysis of general constraint type distributions across models, comparing cases where reasoning helps vs. hurts. Below, we show **WIN/LOSE constraint-type distributions for two models** on IFEval. (Detailed distributions for all models/tasks will appear in Appendix.)\n\n**GPT-4o-mini**\n\n* *Reason WIN:* keywords 19.5%, length_constraints 17.1%, change_case 24.4%, combination 2.4%, detectable_format 19.5%, detectable_content 9.8%, punctuation 4.9%, startend 2.4%\n\n* *Reason LOSE:* combination 35.2%, length_constraints 31.0%, change_case 5.6%, keywords 16.9%, startend 2.8%, detectable_content 2.8%, punctuation 4.2%, detectable_format 1.4%\n\n**Claude-3.5-Haiku**\n\n* *Reason WIN*: change_case 30.0%, keywords 25.0%, length_constraints 15.0%, detectable_format 20.0%, detectable_content 5.0%, combination 5.0%\n\n* *Reason LOSE*: combination 43.3%, length_constraints 25.4%, keywords 10.4%, startend 4.5%, change_case 4.5%, detectable_content 1.5%, detectable_format 6.0%, language 3.0%, punctuation 1.5%\n\nWe notice that for GPT-4o-mini, \n\n**Reasoning Helps ✓**: 81 % of the Reason‐wins come from format / lexical categories: detectable-format (19.5 %), change-case (24.4 %), keywords (19.5 %), length-constraints (17.1 %).\n\n**Reasoning Hurts ✗**: 66 % of the Reason‐losses are classic “over-focus / extra content” errors: combination-repeat-prompt & two-responses (35.2 %), length-constraints over-runs (31.0 %). This cleanly mirrors patterns 3 and 4 in our case study in Section 4.1.\n\nClaude-3.5-Haiku shows similar trends: lexical precision dominates wins, while losses are due to extra text/simple counting errors.”\n\n# 4. Different Evaluation Sample Size for Method 4\nWe understand the concern regarding different test set sizes. To address this, we performed **Monte Carlo cross-validation by repeating random hold-out three times**. We also evaluated the latest Qwen3 models (which have thinking mode). Below, we report the **average**, **standard deviation**, **improvement** scores of the mitigation Method 4, and **p-values** from the two-sided z-tests:\n\n|Model| | |IFEval| | | |ComplexBench| |\n|-|-|-|-|-|-|-|-|-|\n||AVG|STD|Improve|p-value|AVG|STD|Improve|p-value|\n|GPT-4o-mini|83.4|1.4|6.5|8.9e-16|66.0|0.8|5.7|5.5e-35|\n|Claude-3.5-Haiku|87.6|1.4|8.1|1.2e-23|68.9|0.7|6.8|1.6e-63|\n|Claude-3.5-Sonnet|87.8|0.3|8.3|0|67.9|0.2|1.9|7.8e-61|\n|Claude-3.7-Sonnet|90.6|0.2|0.4|0.00053|69.8|0.1|2.8|0|\n|Deepseek-V3|85.2|0.3|0.9|2e-07|71.8|0.1|0.7|7.8e-34|\n|Llama-3.2-1B-Instruct|53.1|1.2|12|1.2e-71|36.6|0.5|11|3.6e-295|\n|Llama-3.2-3B-Instruct|72.3|1.3|10|1.2e-43|50.7|0.6|5|3.2e-47|\n|Meta-Llama-3-8B-Instruct|76.3|1.3|17|1.5e-117|56.0|0.3|1.1|2.1e-10|\n|Llama-3.1-70B-Instruct|85.4|2.0|8.1|2.3e-12|68.5|0.7|8.3|1e-93|\n|Mixtral-8x7B-Instruct|54.1|0.5|-2.3|1.6e-15|60.9|0.3|2.6|6.2e-51|\n|Qwen2.5-1.5B-Instruct|37.1|0.8|5.5|1.1e-32|44.7|0.3|5.9|2.5e-254|\n|DeepSeek-R1-Distill-Qwen-1.5B|19.2|0.8|5.5|1.1e-32|19.1|0.1|2.4|0|\n|Qwen2.5-7B-Instruct|66.5|2.6|8.8|4.6e-09|63.4|0.5|11|2.5e-306|\n|DeepSeek-R1-Distill-Qwen-7B|32.1|1.3|7|1.1e-20|45.3|0.2|6.7|0|\n|Qwen3-4B|86.0|0.5|16|0|66.5|0.2|7.2|0|\n|Qwen3-8B|87.6|0.8|1.3|0.0049|67.6|0.2|7.8|0|\n|Qwen3-32B|88.8|0.9|3.8|2.6e-13|71.7|0.5|8.6|5.1e-195|\n\nWe also computed the **overall p-value from a paired t-test** on improvements across all models: **6.3e-05 for IFEval and 2.1e-06** for ComplexBench. These significant improvements support the robustness and effectiveness of Method 4.\n\n# 5. Explanation of Figure Titles (Figure 1 and 2)\nWe thank the reviewer for pointing this out. The accuracy evaluation for IFEval and ComplexBench is based on the number of instructions the model follows. The list of “True” and “False” in Figures represents boolean values indicating whether each response satisfies the constraints, and the “score” refers to the total count of “True” values. Thank you for pointing this out and we will clarify this in the paper.\n\n# 6. Correlation between Accuracy and Token Length Difference\nWe agree that using token length difference may be a more natural choice. We conducted this analysis and reported the Pearson correlation between accuracy difference (*Acc(Base) – Acc(Reason)*) and token length difference (*Len(Reason) – Len(Base)*) for both IFEval and ComplexBench. Overall, we do not observe strong correlations, except for DeepSeekR1-Distill-Qwen7B on IFEval. We omit p-values for brevity, but note that this case had a p-value of 2e-8, the only one below 5%. In this particular case, it suggests that longer reasoning may be associated with a larger drop in instruction-following performance. We will include these results and further discussion in Appendix I.\n| Model|IFEval|ComplexBench|\n|-|-|-|\n|GPT-4o-mini|-0.047|0.066|\n|Claude-3.5-Haiku| 0.068 |-0.015|\n|Claude-3.5-Sonnet|-0.038|-0.119|\n|Claude-3.7-Sonnet|-0.119|-0.035|\n|DeepSeek-V3|-0.046|-0.069|\n|Llama-3.2-1B|0.037|0.079|\n|Llama-3.2-3B|0.008|-0.020|\n|Llama-3.8B|0.028|0.028|\n|Llama-3.1-70B|0.074|0.022|\n|Mixtral-8x7B|0.090|0.002|\n|Qwen2.5-1.5B|0.073|-0.137|\n|DeepSeekR1-Distill-Qwen1.5B|0.136|-0.045|\n|Qwen2.5-7B|0.137|0.017|\n|DeepSeekR1-Distill-Qwen7B|0.238|0.035|\n\n# 7. CoT Prompt a Reasoning Model\nWe appreciate the suggestion. We **added experiments applying a CoT prompt to a reasoning model** (double thinking, denoted “ThinkCoT”) on three Qwen3 models. We observe that **it produces longer responses with two reasoning segments, and native thinking tends to be conversational, while prompted CoT more formal**. We observe modest gains of \"ThinkCoT\" over “Think” in some cases (e.g. 8B/32B on ComplexBench, 4B on IFEval), though still below “Base”. In other cases (e.g., 8B/32B on IFEval), performance drops. We do not see correlation to token length as longer “double-thinking” does not show consistent directional performance change over “Think” mode. In this situation, we suspect double thinking makes the model focus too much on thinking, affecting response quality.\n| Model|IFEval|ComplexBench|\n|-|-|-|\n|Qwen3-4B|85.0|63.8|\n|Qwen3-4B-Think|69.5|59.3|\n|Qwen3-4B-ThinkCoT|70.2|59.1|\n|Qwen3-8B|86.8|65.6|\n|Qwen3-8B-Think|86.3| 59.8|\n|Qwen3-8B-ThinkCoT|76.3|62.9|\n|Qwen3-32B|87.8|70.2|\n|Qwen3-32B-Think|85.0|63.1|\n|Qwen3-32B-ThinkCoT|75.8| 65.0|\n\n# 8. Manual Examination\nFor each model, eight human annotators manually examined the responses of the Base and Reason variants. For each sample, we provided constraint-specific accuracy and asked annotators to write a short “TL;DR” summarizing why the Reason response won or lost. We found many recurring patterns in these summaries, which we categorized to support the insights presented in Section 4.1.\n\n# 9. Choice of Temperature\nFor our instruction-following tasks and evaluation, we aim for more deterministic responses and scores. While using a nonzero temperature with multiple trials is an option, different models exhibit varying diversity at the same fixed temperature, and a limited number of trials may not accurately reflect true performance. Thus, we believe using temperature 0 is a more natural and reliable setting.\n\nWe deeply appreciate the reviewer’s valuable comments. We hope our clarification and additional evaluations would address certain concerns, and sincerely hope the reviewer could raise the score."}, {"review_id": "xitu2zHGXq", "reviewer": "Reviewer_SzjC", "summary": "The paper examines when chain of thought reasoning in LLMs leads to degrading performance. To this end, they show that explicit chain of thought reasoning can harm instruction following accuracy by systematically evaluating responses with and without chain of thought; applying the method on two benchmarks to a variety of LLMs from different LLM families from 1 to 70 B-parameters shows performance drops in most models. To mitigate the effect, the authors analyze individual reasoning scenarios and introduce an attention-based metric to quantify the influence in these scenarios, such as few-shot in-context learning and self-reflection.", "questions": "What other methods from mechanistic interpretability (MI) could be used to analyze attention traces or relevant parameters and the im MI so-called residual stream during generation? \n\nHow can similar issues arisong from (mis-) using chain of thought patterns be mitigated by combining different strategies, such as self-reflection or reasoning in probability space, or constrained reasoning?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written and tackles a highly interesting topic: reasoning and language models, specifically where it might not be helpful. \n\nWhile methodologically the paper just compares explicit chain of thought with no chain of thought, it carries out a thorough comparison and evaluation across a variety of language models and also reasoning scenarios. \n\nIt proposes an interesting approach to analyze attention patterns using attention interpretability methods, and shows illustrative visualizations of the relevant tokens, where the relevant tokens are identified using GPT-4. These relevant tokens are clustered into rules that are derived from the two benchmarks, giving the possibility for an interesting fine-grained analysis. (The statistics derived from this analysis include the attention trace and also layer-wise statistics, showing that chain of thought reasoning flattens the constraint-attention trace.)\n\nFollowing their evaluation it derives concrete best practices for using reasoning in different scenarios, making the approach easily applicable.\n\nWeaknesses:\nMy main concerns are with the generalizability of the approach:\nFirst, the approach focuses only on English it would be interesting to see how it performs in other languages, maybe less resourced languages. \nSecond, I would find the approach more generalizable if it was also evaluated on tasks not specifically designed for reasoning.\n\nThere is (recent, but also older) related work that tackles similar aspects of reasoning, e.g., chain of though length, [1, 2, 3] that would be important to include and discuss here. \n\n[1] https://aclanthology.org/2024.acl-long.818.pdf\n[2] https://arxiv.org/pdf/2502.07266\n[3] https://arxiv.org/pdf/2410.21333\n[4] https://arxiv.org/pdf/2502.01100", "was_revised": true, "final_justification": "I will keep my original score, suggesting to accept the paper to the conference.", "rebuttal": "**TLDR**: *We agree exploring multilingual instruction-following evaluation is valuable future work. We’ve clarified that our benchmarks already include everyday tasks (not just reasoning-heavy scenarios), added quality-based evaluations using LLM judges, and incorporated correlation analysis regarding CoT length. Additionally, we discussed further mechanistic interpretability methods and acknowledged the potential of combining multiple mitigation strategies.*\n\nThank you for the positive and detailed feedback. We’re especially glad that you found our evaluation thorough, the constraint-attention analysis insightful, and the methods practical and applicable. We also really appreciate the insightful comments! Below are our responses:\n\n# 1. Task on Non-English Languages\nWe really appreciate this suggestion. We agree it would be interesting to study this phenomenon and apply our approaches to other languages. It is out of the scope of this paper but we will actively explore this direction and include multilingual analysis as part of our future work!\n\n# 2. Evaluation on Non-Reasoning Tasks\nIf the reviewer means applying our attention-based analysis to non-reasoning tasks, we believe it can generalize. In our study, attention is used to identify peak signals on important tokens: specifically, tokens corresponding to constraints in instruction-following tasks. We examine how the model’s responses relate to these tokens through their attention scores. This approach is not specific to reasoning and should extend naturally to other tasks by redefining what constitutes “important” tokens.\n\nIf the reviewer is referring to our instruction-following evaluations, in fact, here the IFEval and ComplexBench benchmarks are not specifically designed for reasoning; rather, they focus on evaluating instruction-following capabilities of LLMs. Most instructions in these datasets are everyday user requests (e.g., writing emails or formatting text), not reasoning-heavy tasks. If the reviewer means evaluating models on other aspects of performance beyond instruction following, we have also **added new experiments to assess response quality**. Specifically, we use **LLM-as-a-judge (GPT4o) to compare the responses of Base vs. Reason models**. We restrict this evaluation to only those examples where **both models perfectly satisfy all constraints**, so that differences in quality are not due to instruction-following violations. Below are the win rates of Base models:\n| Model                                 | IFEval | ComplexBench |\n|---------------------------------------|--------|---------------|\n| GPT-4o-mini                           | 87%  | 87%         |\n| Claude-3.5-Haiku                      | 85%  | 91%         |\n| Claude-3.5-Sonnet                     | 73%  | 74%         |\n| Claude-3.7-Sonnet                     | 76%  | 85%         |\n| Deepseek-V3                           | 44%  | 69%         |\n| Llama-3.2-1B-Instruct                 | 51%  | 73%         |\n| Llama-3.2-3B-Instruct                 | 90%  | 70%         |\n| Meta-Llama-3-8B-Instruct              | 85%  | 64%         |\n| Llama-3.1-70B-Instruct                | 82%  | 92%         |\n| Mixtral-8x7B-Instruct                 | 66%  | 70%         |\n| Qwen2.5-1.5B-Instruct                 | 53%  | 66.7%         |\n| DeepSeek-R1-Distill-Qwen-1.5B         | 33.7%  | 53.8%         |\n| Qwen2.5-7B-Instruct                   | 63%  | 77%         |\n| DeepSeek-R1-Distill-Qwen-7B           | 64.6%  | 78.3%         |\n| Qwen3-4B                              | 58%  | 51%         |\n| Qwen3-8B                              | 49%  | 50.1%         |\n| Qwen3-32B                             | 52%  | 55%         |\n\nWe find that in many cases (even all models on ComplexBench) the base model achieves a win rate above 50%, indicating that it produces higher-quality responses than the reasoning variant, even when both follow instructions perfectly. This result suggests that reasoning can not only hurt instruction following, but may also degrade response quality in tasks where step-by-step reasoning is unnecessary.\n\n\n# 3. More Related Work\nThese papers are indeed relevant and very interesting, and we are excited to see these related works come out recently. We have included them into the related work section of our paper. We really thank the reviewer for mentioning these papers! The **exploration of CoT length is particularly intriguing, and we also conducted some preliminary experiments** out of curiosity. In Appendix I, we already examined the correlation between the accuracy difference of the Base and Reason models and the length of the reasoning tokens, and observed no clear correlation. Below, we **conduct additional experiments to study the correlation between the accuracy difference and the length difference** between the Base and Reason models.\n\nMore precisely, we conducted this analysis and report the Pearson correlation between accuracy difference (*Acc(Base) – Acc(Reason)*) and token length difference (*Len(Reason) – Len(Base)*) for both IFEval and ComplexBench. Overall, we do not observe strong correlations, except for DeepSeekR1-Distill-Qwen7B on IFEval. We omit p-values for brevity, but note that this case had a p-value of 2e-8, the only one below 5%. In **this particular case, it suggests that longer reasoning may be significantly associated with a larger drop in instruction-following performance**. We will include these results and further discussion in Appendix I.\n\n| Model                         | IFEval   | ComplexBench |\n|------------------------------|----------|---------------|\n| GPT-4o-mini                  | -0.047   | 0.066         |\n| Claude-3.5-haiku             | 0.068    | -0.015        |\n| Claude-3.5-sonnet            | -0.038   | -0.119        |\n| Claude-3.7-sonnet            | -0.119   | -0.035        |\n| DeepSeek-V3                  | -0.046   | -0.069        |\n| Llama-3.2-1B                 | 0.037    | 0.079         |\n| Llama-3.2-3B                 | 0.008    | -0.020        |\n| Llama-3.8B                   | 0.028    | 0.028         |\n| Llama-3.1-70B                | 0.074    | 0.022         |\n| Mixtral-8x7B                 | 0.090    | 0.002         |\n| Qwen2.5-1.5B                 | 0.073    | -0.137        |\n| DeepSeekR1-Distill-Qwen1.5B  | 0.136    | -0.045        |\n| Qwen2.5-7B                   | 0.137    | 0.017         |\n| DeepSeekR1-Distill-Qwen7B    | 0.238    | 0.035         |\n\n\n# 4. Other Mechanistic Interpretability Methods\nWe agree that additional MI tools can illuminate the phenomena in our paper. In fact we are actively exploring some other ideas:\n* **Activation probing as a reasoning gate.** A simple linear probe trained on mid-layer activations might be able to predict whether CoT will help.  When the probe signals “reasoning unnecessary,” the model stays in Base mode. Otherwise it switches to CoT, reducing harmful reasoning.\n* **Logit-lens comparison.** Running Base and Reason side-by-side, we can project each layer’s residual stream through the LM head (logit lens) and locate constraint-relevant prompt positions where their vocabulary distributions diverge the most.  Then inspecting the hidden-state difference at these hotspots should reveal the essential causes of their different performances.\n* **Activation steering to mitigate the drop.** To further mitigate the drop (i.e. improve instruction following ability), another method we could think of is to perform activation steering (pre-compute the steering vectors for different types of constraints, or they could be the hidden-state difference identified above).  Injecting a small multiple of this vector at generation time could nudge outputs back toward instruction compliance without changing model weights.\n\n# 5. Combination of Multiple Strategies\nWe agree these methods are relatively orthogonal and could be combined, although with potentially increased complexity. In this paper, our main goal is to systematically reveal and analyze the drop in instruction-following caused by reasoning/CoT through controlled experiments, case studies, and attention analysis. Thus, we focused on straightforward mitigation methods that do not alter model weights, since weight changes may negatively affect performance on other tasks. For instance, we **conducted additional experiments exploring the strategy of directly training the model for instruction following**. We do SFT on an IFEval subset and found that it **improved instruction-following for some models but reduced performance on general and reasoning benchmarks** (detailed results can be found in “2. Instruction-Following Training” in our response to Reviewer TQ5w below). Further combining multiple mitigation strategies and optimizing their interactions is an exciting direction that we plan to explore in future work!\n\nOverall, we deeply appreciate the reviewer’s insightful comments and valuable suggestions. We sincerely hope our clarification and additional experiments above would address the concerns, and respectfully hope the reviewer could further raise the score."}, {"review_id": "yehpQbg8ez", "reviewer": "Reviewer_EvS7", "summary": "The paper investigates a counterintuitive phenomenon where explicit Chain-of-Thought (CoT) reasoning can impair the ability of Large Language Models (LLMs) to follow instructions.\n\nThe paper uses two setups:\n1) a model is prompted with and without a specialized prompt for the model to provide thinking before the final answer\n2) a model is used with/without its native thinking mode (e.g. claude sonnet with and without thinking)\n\nThe authors show that performance tend to drop when thinking is used in two IF datasets.\n\nMy main concerns are that:\n* setup (1) is not valid imo to assess the reasoning abilities of models, as models are prompted with a specialized instruction to think, which they most likely weren't trained to tackle. The point is that we shouldn't expect this specific prompt to unlock thinking in models.\n* I think setup (2) is much more valid, as it compares a model with its native thinking behavior. My problem here is that results comparing thinking on/off are pretty close, and there is so significance test, so we cannot know whether there is any model degradation.\n\nGiven these weaknesses I think the paper requires more work", "questions": "* are the results for table 2 significant (i.e., is the performance when thinking is on significantly worse)?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The central claim that reasoning can degrade instruction-following accuracy is unexpected and important. \n* In-depth Analysis of Failure Modes: The paper does not just report the phenomenon but offers a deep dive into why it occurs. It provides a qualitative analysis through a large-scale manual case study, identifying recurring patterns where reasoning helps (e.g., formatting, lexical precision) and where it hurts (e.g., neglecting simple constraints, introducing unnecessary content).\n\nWeaknesses (copying first 2 bullet points from above):\n* setup (1) is not valid imo to assess the reasoning abilities of models, as models are prompted with a specialized instruction to think, which they most likely weren't trained to tackle. The point is that we shouldn't expect this specific prompt to unlock thinking in models.\n* I think setup (2) is much more valid, as it compares a model with its native thinking behavior. My problem here is that results comparing thinking on/off are pretty close, and there is so significance test, so we cannot know whether there is any model degradation.\n* Limited Scope of Investigation: The study's conclusion is confined to instruction-following evaluation (formatting and other simple constraints). What about addressing the actual quality of the responses?  \n* I think the results for methods to mitigate degradation (section 5) are not very informative as they are only applied to setup (1), which is problematic as I mentioned above.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns.", "rebuttal": "**TLDR:** *We clarify the validity of our two experimental settings, added new controlled experiments to reinforce our findings, conducted rigorous statistical significance tests to demonstrate robustness, and provided additional evaluations showing reasoning may not only reduce instruction-following accuracy but can also degrade overall response quality.*\n\nThank you for highlighting the novelty and importance of our main finding. We also appreciate your recognition of the depth and rigor of our failure mode analysis and manual case study. Below are our responses to the comments:\n\n# 1. Design of Setting 1 and Setting 2\n\nFirst, we would like to clarify our motivation and thoughts behind the design of two experimental settings. **Setting 1 (CoT prompting)** involves using the same underlying model with and without chain-of-thought prompting. This provides a **well-controlled comparison since the model weights are identical, isolating the effect of prompting alone**. CoT prompting is also a common technique applied in practice to elicit reasoning. As noted in Section 3.4, **Setting 2 (comparing a base model to its Reasoning counterpart)** introduces potential confounds: Reason models may **undergo additional training stages on new datasets**, such as supervised fine-tuning or RL. For example, *DeepSeek-V3* and *DeepSeek-R1* differ significantly in training scale, with R1 having been further trained on much more data in multiple extra training stages, involving both SFT and RL. Despite these differences, we **include both settings to demonstrate the general phenomenon: a consistent drop in instruction-following performance after applying CoT or reasoning**.\n\nIn general, we agree and acknowledge that each setting has distinct advantages and limitations. To further strengthen the Setting 2 results, we added additional base–reasoning pairs:\n* *Qwen2.5-1.5B-Instruct vs. Qwen2.5-Math-1.5B*\n* *Qwen2.5-7B-Instruct vs. Qwen2.5-Math-7B*\n\nIn each case, the Math variant is explicitly fine-tuned for reasoning. Furthermore, we incorporated newly released Qwen3 models, which include both Base and Think modes. This is a better controlled experiment as the think vs. no-think mode only differ in the template and the model weights are the same:\n\n* *Qwen3-4B vs. Qwen3-4B-Think*\n* *Qwen3-8B vs. Qwen3-8B-Think*\n* *Qwen3-32B vs. Qwen3-32B-Think*\n\nWe evaluate all pairs on both IFEval and ComplexBench. As shown in the table below, the **general trend persists: reasoning variants tend to underperform their base counterparts in instruction-following, across both benchmarks**.\n| Model                   | IFEval | ComplexBench |\n|-------------------------|--------|--------------|\n| Qwen3-4B                | 85.0   | 63.8         |\n| Qwen3-4B-Think          | 69.5   | 59.3         |\n| Qwen3-8B                | 86.8   | 65.6         |\n| Qwen3-8B-Think          | 86.3   | 59.8         |\n| Qwen3-32B               | 87.8   | 70.2         |\n| Qwen3-32B-Think         | 85.0   | 63.1         |\n| Qwen2.5-1.5B-Instruct   | 35.9   | 44.1         |\n| Qwen2.5-Math-1.5B       | 14.9   | 23.4         |\n| Qwen2.5-7B-Instruct     | 63.6   | 60.2         |\n| Qwen2.5-Math-7B         | 27.9   | 28.8         |\n\nWe sincerely appreciate the reviewer’s comment and will incorporate these additional results and corresponding discussion into Section 3.4 of the revised paper.\n\n# 2. Significance Test\nIn the evaluation setup, the rule-based Python functions are deterministic. For ComplexBench, which includes both rule-based and LLM-as-a-judge components, we also use temperature 0 for GPT-based evaluation. For the Classifier-Selective mitigation method, since it involves classifier training on a randomly sampled subset of the data, we further examine its robustness by running **Monte Carlo cross-validation by repeating random hold-out three times**. Below we report the **average**, **standard deviation**, **improvement** over the Reason/CoT model, and **p-values** from two-sided z-tests:\n\n| Model                         |      |      |  IFEval |            |      |      |ComplexBench   |            |\n|-------------------------------|------|------|---------|------------|------|------|---------|------------|\n|                               | AVG  | STD  | Improve | p-value    | AVG  | STD  | Improve | p-value    |\n| GPT-4o-mini                   | 83.4 | 1.4  | 6.5     | 8.9e-16    | 66.0 | 0.8  | 5.7     | 5.5e-35    |\n| Claude-3.5-Haiku              | 87.6 | 1.4  | 8.1     | 1.2e-23    | 68.9 | 0.7  | 6.8     | 1.6e-63    |\n| Claude-3.5-Sonnet             | 87.8 | 0.3  | 8.3     | 0          | 67.9 | 0.2  | 1.9     | 7.8e-61    |\n| Claude-3.7-Sonnet             | 90.6 | 0.2  | 0.4     | 0.00053    | 69.8 | 0.1  | 2.8     | 0          |\n| Deepseek-V3                   | 85.2 | 0.3  | 0.9     | 2e-07      | 71.8 | 0.1  | 0.7     | 7.8e-34    |\n| Llama-3.2-1B-Instruct         | 53.1 | 1.2  | 12      | 1.2e-71    | 36.6 | 0.5  | 11      | 3.6e-295   |\n| Llama-3.2-3B-Instruct         | 72.3 | 1.3  | 10      | 1.2e-43    | 50.7 | 0.6  | 5       | 3.2e-47    |\n| Meta-Llama-3-8B-Instruct      | 76.3 | 1.3  | 17      | 1.5e-117   | 56.0 | 0.3  | 1.1     | 2.1e-10    |\n| Llama-3.1-70B-Instruct        | 85.4 | 2.0  | 8.1     | 2.3e-12    | 68.5 | 0.7  | 8.3     | 1e-93      |\n| Mixtral-8x7B-Instruct         | 54.1 | 0.5  | -2.3    | 1.6e-15    | 60.9 | 0.3  | 2.6     | 6.2e-51    |\n| Qwen2.5-1.5B-Instruct         | 37.1 | 0.8  | 5.5     | 1.1e-32    | 44.7 | 0.3  | 5.9     | 2.5e-254   |\n| DeepSeek-R1-Distill-Qwen-1.5B | 19.2 | 0.8  | 5.5     | 1.1e-32    | 19.1 | 0.1  | 2.4     | 0          |\n| Qwen2.5-7B-Instruct           | 66.5 | 2.6  | 8.8     | 4.6e-09    | 63.4 | 0.5  | 11      | 2.5e-306   |\n| DeepSeek-R1-Distill-Qwen-7B   | 32.1 | 1.3  | 7       | 1.1e-20    | 45.3 | 0.2  | 6.7     | 0          |\n| Qwen3-4B                      | 86.0 | 0.5  | 16      | 0          | 66.5 | 0.2  | 7.2     | 0          |\n| Qwen3-8B                      | 87.6 | 0.8  | 1.3     | 0.0049     | 67.6 | 0.2  | 7.8     | 0          |\n| Qwen3-32B                     | 88.8 | 0.9  | 3.8     | 2.6e-13    | 71.7 | 0.5  | 8.6     | 5.1e-195   |\n\nWe also computed the **overall p-values across all models using paired t-tests** on the improvement scores: **6.3e-05 for IFEval, and 2.1e-06 for ComplexBench**.\nThese consistently small p-values, along with the positive improvement scores, indicate that the improvements from the Classifier-Selective method are statistically significant and robust. This further strengthens the effectiveness of our proposed approach.\n\n# 3.  Quality (besides Instruction Following) of Response:\nThe overall response quality, beyond instruction-following performance, is indeed another important and interesting dimension to evaluate. We agree with the reviewer that reasoning can influence both instruction adherence and general response quality. While reasoning models often excel at tasks like math and code generation, the instructions in IFEval and ComplexBench are more about following details and less creative (e.g. writing an email in a specific format). \n\nIn our paper, we focused on how reasoning can degrade instruction-following ability. Here, we **add new experiments to assess response quality** directly. Specifically, we use LLM-as-a-judge (GPT4o) to compare the responses of Base vs. Reason models. Importantly, we restrict this evaluation to only those examples where both models perfectly satisfy all constraints, so that differences in quality are not due to instruction-following violations.\n\n| Model                                 | IFEval | ComplexBench |\n|---------------------------------------|--------|---------------|\n| GPT-4o-mini                           | 87%  | 87%         |\n| Claude-3.5-Haiku                      | 85%  | 91%         |\n| Claude-3.5-Sonnet                     | 73%  | 74%         |\n| Claude-3.7-Sonnet                     | 76%  | 85%         |\n| Deepseek-V3                           | 44%  | 69%         |\n| Llama-3.2-1B-Instruct                 | 51%  | 73%         |\n| Llama-3.2-3B-Instruct                 | 90%  | 70%         |\n| Meta-Llama-3-8B-Instruct              | 85%  | 64%         |\n| Llama-3.1-70B-Instruct                | 82%  | 92%         |\n| Mixtral-8x7B-Instruct                 | 66%  | 70%         |\n| Qwen2.5-1.5B-Instruct                 | 53%  | 66.7%         |\n| DeepSeek-R1-Distill-Qwen-1.5B         | 33.7%  | 53.8%         |\n| Qwen2.5-7B-Instruct                   | 63%  | 77%         |\n| DeepSeek-R1-Distill-Qwen-7B           | 64.6%  | 78.3%         |\n| Qwen3-4B                              | 58%  | 51%         |\n| Qwen3-8B                              | 49%  | 50.1%         |\n| Qwen3-32B                             | 52%  | 55%         |\n\nWe find that in many cases (even all models on ComplexBench) the base model achieves a win rate above 50%, indicating that **it produces higher-quality responses than the reasoning variant**, even when both follow instructions perfectly. This result suggests that **reasoning can not only hurt instruction following, but may also degrade general response quality** in tasks where step-by-step reasoning is unnecessary.\n\nOverall, we deeply appreciate the reviewer’s insightful comments and valuable suggestions. We sincerely hope our clarification and additional experiments above would address the concerns, and respectfully hope the reviewer could raise the score."}, {"review_id": "bCqp9IxhK1", "reviewer": "Reviewer_TQ5w", "summary": "This paper identifies an interesting phenomenon, language model optimized for reasoning will usually have a surprising degradation on instruction-following ability. The authors analyzed why such failures occur, places where reasoning helps and places where reasoning might hurt.\n\nThe authors did comprehensive experiments on a wide set of models including reasoning and non-reasoning models, with or without CoT, and show that there's a consistent trend of instruction-following performance degradation across almost all models. In addition, the authors proposed several methods to help mitigate such failures, and showed that a classifier-selection based method can help bring back the ability on instruction-following.", "questions": "See above.\n\n- Classifier-selective method: Can the authors split the data according to different patterns, train the classifier on a set of those patterns, and test on another set? In practice we would hope such classifier can generalize to a wide set of unseen patterns in order to be useful.\n- Another simple mitigation could be training the model directly for instruction-following. Have the authors considered this (e.g., RL with rewards as the instruction is being followed vs not) and how would it compare with the classifier based method?", "limitations": "Discussed in Section 6.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is overall well-written.\n- The authors identified an important phenomenon: the model becomes worse at instruction-following when optimized for reasoning, which is surprising and remains to be under-explored in existing literature. The authors also did comprehensive analysis on specific cases showing why the model might exhibit such behavior.\n- Experiments are performed over a wide set of models and on two instruction-following benchmarks. The authors have also proposed four different mitigation methods and showed they yield various gains on the IF benchmarks.\n\nWeaknesses:\n- I wonder how the classifier-selective method generalizes in practice. It does yield good performance on the two benchmarks, but could be due to the classifier is trained to identify similar patterns (since the patterns on the two IF benchmarks are fairly limited), and learn to use or not use CoT on those patterns. Can the authors split the data according to different patterns, train the classifier on a set of those patterns, and test on another set? In practice we would hope such classifier can generalize to a wide set of unseen patterns in order to be useful.\n- Another simple mitigation could be training the model directly for instruction-following. Have the authors considered this (e.g., RL with rewards as the instruction is being followed vs not) and how would it compare with the classifier based method?", "was_revised": true, "final_justification": "I've read the authors' rebuttal and it addressed most of my concerns, specifically the ones related to the generalization on hold-out IF tasks. I would maintain my current positive rating.", "rebuttal": "**TLDR**: *We evaluated the generalization of the classifier-selective method by training on specific constraint subsets, showing improved generalization when focusing on constraints where reasoning typically fails. Additionally, we explored direct instruction-following fine-tuning (SFT), finding it effective but also potentially harmful to performance on unrelated tasks, highlighting the importance of weight-free mitigation methods.*\n\nThank you for recognizing both the novelty of the phenomenon and the comprehensive nature of our analysis and experiments. We also appreciate your acknowledgment of our mitigation strategies and their effectiveness across diverse models and benchmarks. Meanwhile, we sincerely appreciate the comments and provide our responses below:\n\n# 1. Generalizability of Classifier-Selective Method:\nWe understand the concern about the generalizability of the classifier-selective method and agree that it is an important aspect to explore. One challenge is that each sample often contains multiple overlapping constraints. Nonetheless, we try our best to implement the idea of training the classifier on a subset of constraint types and evaluating its generalization to other types. Specifically, the IFEval dataset annotates constraints with higher-level categories including:\n* *['detectable_content', 'startend', 'combination', 'keywords', 'change_case', 'punctuation', 'detectable_format', 'language', 'length_constraints']*.\n\nWe selected five types such that the samples containing these categories account for roughly half of the dataset:\n* *['length_constraints', 'combination', 'startend', 'language', 'change_case'].*\n\nFor each model, we trained the classifier only on this subset and evaluated its performance on the held-out half.\nInterestingly, we observed even stronger results than those from classifiers trained on a random 50% split: **in many cases, the improvements were significantly larger than before**, as shown below (improvements are relative to the Reason/CoT model, save as in the tables in paper):\n\n| Model                             | Score | Improvement |\n|-----------------------------------|-------|-------------|\n| GPT-4o-mini                       | 86.0  | 9.1         |\n| Claude-3.5-Haiku                  | 89.7  | 10.2        |\n| Claude-3.5-Sonnet                 | 88.5  | 9.0         |\n| Claude-3.7-Sonnet                 | 91.1  | 0.9         |\n| Deepseek-V3                       | 89.7  | 5.4         |\n| Llama-3.2-1B-Instruct             | 59.0  | 18.3        |\n| Llama-3.2-3B-Instruct             | 79.3  | 17.4        |\n| Meta-Llama-3-8B-Instruct          | 77.1  | 18.1        |\n| Llama-3.1-70B-Instruct            | 86.7  | 9.4         |\n| Mixtral-8x7B-Instruct             | 63.8  | 7.4         |\n| Qwen2.5-1.5B-Instruct             | 47.2  | 15.6        |\n| DeepSeek-R1-Distill-Qwen-1.5B     | 24.7  | 11.0        |\n| Qwen2.5-7B-Instruct               | 69.4  | 11.7        |\n| DeepSeek-R1-Distill-Qwen-7B       | 43.2  | 18.1        |\n| Qwen3-4B                          | 85.2  | 15.7        |\n| Qwen3-8B                          | 88.6  | 2.3         |\n| Qwen3-32B                         | 88.9  | 3.9         |\n\nFirst, these results show that the **classifier-selective method does generalize across constraint types** in certain aspects. Interestingly, they also suggest that **training the classifier specifically on certain constraint types may lead to even better overall performance**.\n\nTo explain this, we hypothesize that it is because the selected types (e.g., *length_constraints*, *combination*) are where CoT is most prone to introducing errors. To support this, we **conducted a further quantitative analysis of constraint-type distributions**. Below we show the Reason-WIN vs. Reason-LOSE distributions on IFEval for two representative models (detailed distributions for all models/tasks will be included in the Appendix of our paper):\n\n**GPT-4o-mini**\n\n* *Reason WIN:* keywords 19.5%, length_constraints 17.1%, change_case 24.4%, combination 2.4%, detectable_format 19.5%, detectable_content 9.8%, punctuation 4.9%, startend 2.4%\n\n* *Reason LOSE:* combination 35.2%, length_constraints 31.0%, change_case 5.6%, keywords 16.9%, startend 2.8%, detectable_content 2.8%, punctuation 4.2%, detectable_format 1.4%\n\n**Claude-3.5-haiku**\n\n* *Reason WIN*: change_case 30.0%, keywords 25.0%, length_constraints 15.0%, detectable_format 20.0%, detectable_content 5.0%, combination 5.0%\n\n* *Reason LOSE*: combination 43.3%, length_constraints 25.4%, keywords 10.4%, startend 4.5%, change_case 4.5%, detectable_content 1.5%, detectable_format 6.0%, language 3.0%, punctuation 1.5%\n\nFor both models, we observe a clear pattern: **Reason-WINs are concentrated in lexical or formatting categories** (e.g., *keywords*, *change_case*, *detectable_format*), while **Reason-LOSEs are dominated by global or compositional constraints** (e.g., *combination*, *length_constraints*, *startend*). **These “failure-prone” types were exactly those we selected for training the classifier, and moreover, these results match and further support our case study insights in Section 4.1**.\n\n# 2. Instruction-Following Training:\n\nWe thank the reviewer for this thoughtful suggestion! Indeed, we are actively exploring additional methods to mitigate the drop in instruction-following performance caused by reasoning/CoT. The primary goal of this paper is to reveal and analyze this degradation, supported by quantitative experiments, case studies, and attention analysis. For mitigation, we focused on simple methods that do not alter model weights, because for methods involving training, which change the model weights, our major concern is how this training may affect performance on other tasks.\n\nWe explore this with experiments. Certain benchmark datasets (such as ComplexBench) involve both rule-based evaluation and LLM-as-a-judge, so RL training for instruction-following might be expensive, as LLM-as-a-judge could significantly slow down the process (but we will explore this further in future work). Therefore, we **conducted supervised fine-tuning (SFT) experiments** using data labeled by the strongest model in our evaluation (Claude 3.7-Sonnet). We selected only samples where the model response received a perfect instruction-following score, and trained using these examples as positive supervision.\n\nWe split half of the IFEval dataset for SFT training and evaluated instruction-following performance on the other half after SFT (denoted as “IFEval (SFT)”). We also evaluated performance before and after SFT on other representative tasks: ARC, TruthfulQA, GSM8K, and MinervaMath, which include both general knowledge and reasoning evaluation, to assess any unintended impact. Below are the results:\n\n| Model                               | Original | CoT   | IFEval(SFT) | ARC  | ARC(SFT) | TruthfulQA | TruthfulQA(SFT) | GSM8K | GSM8K(SFT) | MinervaMath | MinervaMath(SFT) |\n|-------------------------------------|----------|-------|---------------------|------|-----------|-------------|-------------------|--------|-------------|--------------|--------------------|\n| Llama-3.2-1B-Instruct               | 49.1    | 40.6 | 40.5                | 31.4 | 32        | 24.1        | 27.1              | 20.3   | 13.7        | 14.5         | 13.1               |\n| Meta-Llama-3-8B-Instruct           | 75.1    | 58.8 | 75                  | 52.8 | 46.7      | 40.5        | 34.6              | 80     | 75.2        | 29.6         | 28.4               |\n| Qwen2.5-1.5B-Instruct              | 35.7    | 31.7  | 38.3                | 38.7 | 36.9      | 31.2        | 30.9              | 63.3   | 63.2        | 16.8         | 18                 |\n| DeepSeek-R1-Distill-Qwen-1.5B      | 16.6    | 13.5 | 23.5                | 27.7 | 27.7      | 35.4        | 35.3              | 14.2   | 14          | 5.6          | 5                  |\n| DeepSeek-R1-Distill-Qwen-7B        | 30.6    | 25.0 | 36.4                | 32.7 | 32.5      | 30.7        | 29.4              | 9.9    | 12.9        | 6.3          | 8.5                |\n\nWe observe that **SFT can significantly improve instruction-following performance** for many models (e.g., a +16.0 improvement for Meta-Llama-3-8B-Instruct). However, it **can also negatively impact performance on other tasks** (possibly due to catastrophic forgetting on the other tasks after SFT on instruction-following). For instance, Meta-Llama-3-8B-Instruct shows performance drops across all evaluated benchmarks after SFT. This highlights the need for caution when applying weight-updating mitigation strategies.\n\nMeanwhile, we are continuing to explore alternative mitigation strategies that preserve model weights. To share some ideas,  some directions include:\n* *Activation probing as a reasoning gate*: A lightweight linear probe trained on mid-layer activations could predict whether reasoning is needed. If reasoning is deemed unnecessary, the model uses the base response; otherwise, it switches to CoT.\n* *Logit-lens comparison*: By running Base and Reason side-by-side, we can use the logit lens to identify positions in the prompt where their token distributions diverge most. Examining these hidden-state differences could reveal the essential source of instruction-following failures.\n* *Activation steering to mitigate failures*: We could precompute steering vectors associated with constraint types (or derived from the hidden-state differences above) and inject them during generation. This may nudge the model toward better instruction adherence, without fine-tuning.\n\n\nOverall, we deeply appreciate the reviewer’s insightful comments and valuable suggestions. We sincerely hope our clarification and additional experiments above would address the concerns, and respectfully hope the reviewer could even further raise the score."}]

Author rebuttal:

Meta-review: # Summary
This paper investigates a phenomenon where CoT prompting can significantly degrade instruction-following accuracy. They provided an analysis of when explicit reasoning can hurt/improve instruction-following and constraint satisfaction performance, and further compared four different mitigation strategies to restore the instruction-following capabilities. Specifically, they investigate the differences of these cases in terms of constraint attention trace, i.e., how the attention of output tokens to each constraint changes over decoding steps, on hundreds of samples. This study finds a major trend: reasoning flattens the constraint-attention trace, and the attention is lower on the answer segments when reasoning degrades the performance. The four mitigation strategies aim to judge and select reasoning vs. non-reasoning, and training an LLM-based classifier for this purpose outperforms others. All the empirical studies are conducted on IFEval and ComplexBench. 

# Strengths
- Studying whether and when CoT can hurt/improve instruction following is an important, underexplored problem. 
- The analysis provides novel insights into when CoT helps and where it should be used. It is novel to compare the differences made by CoT using constraint attention traces. 
- The paper also compares different mitigation strategies and provides a feasible solution.
- Experiments are performed over a wide set of models and on two instruction-following benchmarks. 
- The paper is well-motivated and well-written.

# Weaknesses
- The proposed mitigation strategies are quite straightforward and do not provide sufficient insights.
- Some important results are missing, e.g., the statistics of case studies, experiments on reasoning models, generalization of classifier-based strategy and its comparison to SFT baseline, the cost of mitigation methods, etc. 
- The proposed method's generalizability to multilingual tasks and non-reasoning tasks is not evaluated. 
- Insufficient discussion and comparison with works on CoT-length and more mechanistic interpretability.
- The evaluation setting needs more justification since the model may not be trained to perform the best under the CoT prompt.
- The analysis is limited to a few specific properties of the outputs when using or not using CoT. 

# Reasons to Accept
- The studied problem is important for understanding when the widely-used CoT hurts/improves the instruction/constraint following capability. 
- The mitigation strategies evaluated in this paper provide valuable insights to address the problem.
- The experiments cover various diverse models.
- The rebuttal and discussion addressed several major concerns of the reviewers with various additional experiments.

# Discussion Summary
- In the rebuttal, the authors provided detailed clarifications and a great amount of additional experiments to answer the questions of the reviewers.
- Most reviewers responded that the rebuttal and further discussion addressed their initial concerns.
Decision: Accept (spotlight)