Chain-of-Thought Hijacking

## Abstract

Abstract Large reasoning models (LRMs) achieve higher task performance by allocating more inference-time compute, and prior works suggest this scaled reasoning may also strengthen safety by improving refusal.
Yet we find the opposite: the same reasoning can be used to bypass safeguards. We introduce Chain-of-Thought Hijacking , a jailbreak attack on reasoning models. The attack pads harmful requests with long sequences of harmless puzzle reasoning. Across HarmBench, CoT Hijacking reaches a 99%, 94%, 100%, and 94% attack success rate (ASR) on Gemini 2.5 Pro, GPT o4 mini, Grok 3 mini, and Claude 4 Sonnet, respectively—far exceeding prior jailbreak methods for LRMs. To understand the effectiveness of our attack, we turn to a mechanistic analysis, which shows that mid layers encode the strength of safety checking , while late layers encode the verification outcome . Long benign CoT dilutes both signals by shifting attention away from harmful tokens. Targeted ablations of attention heads identified by this analysis causally decrease refusal, confirming their role in a safety subnetwork.
These results show that the most interpretable form of reasoning—explicit CoT—can itself become a jailbreak vector when combined with final-answer cues.
We release prompts, outputs, and judge decisions to facilitate replication.

## 1 Introduction

Large reasoning models (LRMs) extend traditional language models by allocating inference-time compute to generate step-by-step reasoning before providing an answer. This ability improves performance across a wide range of tasks, including mathematics, programming, and scientific problem solving (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022). Moreover, recent works (Guan et al., 2024; Jaech et al., 2024) also have shown that reasoning can make refusals more consistent and robust.

In contrast with previous findings, in this study we show that rather than strengthening refusals, long reasoning sequences weaken them, creating a new attack surface. We introduce Chain-of-Thought Hijacking (CoT-Hijacking ), a simple jailbreak where harmless reasoning is prepended before a harmful instruction. This attack consistently reduces refusal rates and achieves state-of-the-art success. For example, across HarmBench (Chao et al., 2023), CoT Hijacking reaches 99% attack success on Gemini 2.5 Pro, compared to 44% for Mousetrap (Chao et al., 2024a), 60% for H-CoT (Kuo et al., 2025), and 69% for AutoRAN (Chen et al., 2023).

For clarity, we adopt the following taxonomy: inference scaling refers to allocating more compute at inference time; reasoning is a subset of inference scaling where models generate multi-step latent or verbal processes; and CoT is a further subset where those steps are explicitly verbalized. Prior work has cautioned against treating CoT as a faithful interpretability method (Barez et al., 2025), yet its transparency also makes it an attractive attack surface. Inspired by this, our focus is on CoT hijacking—a jailbreak that exploits this most interpretable form of reasoning.

Our work is related to H-CoT by Kuo et al. (2025), where they hijacks visible safety chain-of-thought using the Malicious-Educator benchmark. In contrast, our attack does not rely on exposed safety reasoning or educational framing: simply padding with benign reasoning and a final-answer cue dilutes the refusal signal, yielding broader generality and state-of-the-art success rates.

We then analyze why the attack works. Our results show that refusal relies on a fragile, low-dimensional safety signal that becomes diluted as reasoning grows longer. Attention shifts toward the final-answer region, while refusal features in later layers weaken, allowing harmful instructions to slip through.
These findings suggest that jailbreaks of LRMs are not isolated prompt tricks but systematic failures of safety under reasoning. Addressing them will require alignment strategies that scale with reasoning depth rather than relying on brittle refusal signals.

Figure: Figure 1: The upper part illustrates a safe example: the target model refuses a harmful request. The lower part shows a successful jailbreak example: the target model complies with the harmful request under our attack. Grey highlights indicate the puzzle content, whereas red highlights mark the malicious request or content.
Refer to caption: x3.png

#### Contributions.

- •
We introduce CoT Hijacking, a new jailbreak attack in which benign reasoning systematically weakens refusal.
- •
We show that CoT Hijacking achieves state-of-the-art success on HarmBench, reaching 99%, 94%, 100%, and 94% on Gemini 2.5 Pro, GPT o4 mini, Grok 3 mini, and Claude 4 Sonnet, respectively.
- •
We analyze why refusal fails under CoT Hijacking, using attention patterns, probes, and causal interventions to show dilution of refusal features.
- •
We highlight broader risks: the same mechanism may undermine other safety behaviors, including truthfulness, privacy, and bias mitigation.

## 2 Related Work

#### Large Reasoning Model

Traditional language models (Touvron et al., 2023a; b; Dubey et al., 2024) rely on intuitive and fast system-1 thinking (Frankish, 2010; Li et al., 2025c). Despite their remarkable performance in the general domain, these models usually underperform in difficult math Cobbe et al. (2021), coding (Jimenez et al., 2023), or formal theorem proving (Zhao et al., 2024) problems that require complex reasoning to attain the final answer. To fill the gap, with OpenAI o-series model (Jaech et al., 2024) and DeepSeek-R1 (Guo et al., 2025) as two representatives, a surge of large reasoning models has been proposed, focusing on enhancing system-2 thinking ability, or slow, step-by-step thinking with chain-of-thoughts Wei et al. (2022); Wang et al. (2022); Shaikh et al. (2022). Trained with supervised fine-tuning or reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2024), large reasoning models exhibit a substantial growth in the chain-of-thought length (Zeng et al., 2025; Luo et al., 2025; Fatemi et al., 2025), which is usually associated with the emergence of sophisticated reasoning abilities like reflection and self-correction (Liu et al., 2025). However, as a side-effect of reasoning-oriented training, recent studies find that stronger reasoning models are more likely to produce harmful content, especially in the chain-of-thought, suggesting potential safety risk of LRMs (Zhou et al., 2025a).

#### Jailbreaking Large Language Models

Jailbreaking attacks target breaking or bypassing the inner safety mechanism of language model and induce it to produce harmful and unsafe content. Existing approachs to Jailbreaking can be divided into two categories based on their accessibility to victim model. To be more specific, white-box methods directly alter the parameters (Sun et al., 2024), activations (Li et al., 2025a), logits (Guo et al., 2024), or modify the prompt with gradient information (Zou et al., 2023). On the other hand, black-box methods simply rewrite the prompt to into a seemingly benign queries by designing a convoluted scenario (Li et al., 2023), creating a role-play context (Ma et al., 2024), enciphering the query into a cipher text or program coding (Yuan et al., 2024; Jiang et al., 2024), translating it into multiple language (Shen et al., 2024), or simply optimizing sampling strategies (Hughes et al., 2024). Our work complements red-teaming and adversarial robustness efforts such as PoisonBench (Fu et al., 2024), which highlight vulnerabilities from poisoned training data, whereas we target inference-time reasoning.
Compared with general language models, LRMs introduce new vulnerability to jailbreaking attacks even after sophisticated safety alignment (Guan et al., 2024). An important factor could be the decline of instruction-following ability after a long chain-of-thought (Fu et al., 2025) due to attention dilution (Li et al., 2025b).
Moreover, the chain-of-thought can reveal the refusal criteria of LRMs, which can be exploited by malicious attackers to bypass the safety check (Kuo et al., 2025).

#### Mechanistic Interpretability of Safety

Mechanistic interpretability builds a causal connection between the internal representation and the behavior of language models. Different from the previous finding that attributes the success of language model jail-breaking to competing objectives and mismatched generalization, interpretability studies on language model safety reveal safety mechanisms through component analysis (Ball et al., 2024) or circuit analysis (Chen et al., 2024).
Specifically, previous work finds that the activation for harmful prompt and harmless prompt can be clustered with a clear boundary (Lin et al., 2024; Gao et al., 2024) while the activation for jailbreaking prompts are close to or even beyond the boundary, therefore misleading the harmfulness detection of language models. Furthermore, the difference between the mean of two clusters, or refusal direction (Arditi et al., 2024a) can be used for manipulating the refusal mechanism when added or subtracted from activation at inference time (Li et al., 2025a; Ghosh et al., 2025). Most relevant to our study, He et al. (2024); Zhou et al. (2025b) finds that safety-related parameters are sparse and located within specific attention heads. Additionally, our findings connect to broader patterns where safety behaviors are subverted under pressure, such as sycophancy and reward-tampering (Denison et al., 2024).”

## 3 Motivating Evidence: CoT Length on S1

We begin with a controlled experiment on the S1 model (simplescaling/s1-32B) (Muennighoff et al., 2025) to test whether CoT length systematically affects refusal reliability.
S1 provides a reproducible, mid-scale reasoning model where interventions on CoT length are straightforward, making it a useful sandbox for probing this effect before scaling to larger models. For each condition, we report the attack success rate (ASR) using chatgpt-4o-mini as the judge model. We evaluate three CoT length conditions under the same harmful instruction:
Minimal: reasoning truncated to a short placeholder (“I have finished thinking”). Natural: model generates its default reasoning length and Extended: prompt encourages long, step-by-step reasoning (e.g., “Take a deep breath and work on this problem step-by-step…”).

**Table 1: Attack success rate (ASR, %) on S1 under different CoT length conditions.**
| Setting | Minimal | Natural | Extended |
| --- | --- | --- | --- |
| ASR (%) | 27 | 51 | 80 |

The results reveal a clear pattern: longer reasoning traces substantially increase the likelihood of harmful outputs, with ASR rising from 27% (Minimal) to 80% (Extended). This provides an initial behavioral clue that refusals in reasoning models degrade as CoT length grows. In Section [4](https://arxiv.org/html/2510.26418v1#S4), we build on this observation by introducing *Chain-of-Thought Hijacking*, a systematic jailbreak that exploits this vulnerability.

Figure: Figure 2: Jailbreak Method Pipeline Figure. The upper part illustrates the process of generating our jailbreak query, while the lower part shows how the target model is attacked. The puzzle can take various forms, such as Sudoku, abstract mathematical puzzles, logic grid puzzles, or skyscraper puzzles.
Refer to caption: x4.png

## 4 Chain-of-Thought Hijacking: Attack Design and Empirical Results

### 4.1 Jailbreak methodology

We define a prompt-based jailbreak, CoT Hijacking.
The attack prepends a long, benign reasoning preface to a harmful instruction, followed by a final-answer cue.
This structure systematically reduces refusals: the benign CoT dilutes the refusal signal while the cue shifts attention to the answer region.

Figure: Figure 3: Structure of a CoT Hijacking prompt.

To scale attack construction, we implement an automated pipeline (“Seduction”) using an auxiliary LLM to generate candidate prefaces and integrate payloads.
Each candidate is scored by a judge call to the target model, providing (i) whether the output was a refusal and (ii) the CoT length.
This black-box feedback loop iteratively refines prompts, producing effective jailbreaks without access to model internals.
Prompt examples can be seen in Figure [1](https://arxiv.org/html/2510.26418v1#S1.F1).

### 4.2 Main experiments on HarmBench

We adopt several LRM-specific jailbreak methods as baselines, including Mousetrap (Yao et al., 2025), H-CoT (Kuo et al., 2025), and AutoRAN (Liang et al., 2025). Given the substantial computational cost per jailbreak sample, we use the first 100 samples from HarmBench (Mazeika et al., 2024) as our benchmark. The target models include Gemini 2.5 Pro, ChatGPT o4 Mini, Grok 3 Mini, and Claude 4 Sonnet, all evaluated under the unified judging protocols of Chao et al. (2024b). We report Attack Success Rate (ASR) as the primary metric to assess jailbreak effectiveness.

**Table 2: Main Experiments on HarmBench with Attack Success Rate (%).**
| Model<br>Method | Mousetrap | H-CoT | AutoRAN | Ours |
| --- | --- | --- | --- | --- |
| Gemini 2.5 Pro | 44 | 60 | 69 | \cellcolorgray!1599 |
| ChatGPT o4 Mini | 25 | 65 | 47 | \cellcolorgray!1594 |
| Grok 3 Mini | 60 | 66 | 61 | \cellcolorgray!15100 |
| Claude 4 Sonnet | 22 | 11 | 5 | \cellcolorgray!1594 |

Across all models, CoT Hijacking consistently outperforms baselines, including on frontier proprietary systems.
This demonstrates that extended reasoning sequences can act as a new and highly exploitable attack surface.

### 4.3 Reasoning-effort study on GPT-5-mini

We further test CoT Hijacking on GPT-5-mini with reasoning-effort settings (*minimal, low, high*) on 50 HarmBench samples.

**Table 3: Additional Experiments on GPT-5-mini under different reasoning-effort settings.**
| Reasoning Effort | Minimal | Low | High |
| --- | --- | --- | --- |
| ASR (%) | 72 | \cellcolorgray!1576 | 68 |

Interestingly, attack success is highest under *low* effort, suggesting that reasoning-effort and CoT length are related but distinct controls.(^1^11The GPT5 official document mentions ’The reasoning.effort parameter controls how many reasoning tokens the model generates before producing a response.’. We are not quite sure whether this means a direct CoT length control.)
Longer reasoning does not guarantee greater robustness—in some cases it reduces it.
Full prompt templates and logs are in Appendix [D](https://arxiv.org/html/2510.26418v1#A4).

## 5 Refusal Direction in Large Reasoning Models

We study whether refusal behavior in large reasoning models (LRMs) can also be traced to a single activation-space direction. Following Arditi et al. (2024b), we compute a *refusal direction* by contrasting mean activations on harmful versus harmless instructions.
This direction represents the dominant feature that separates refusal from compliance.
To better capture the refusal feature, we switch to a more robust and sophisticated Reasoning Model, Qwen3-14B (Yang et al., 2025), which has 40 layers. The strongest refusal direction is observed at layer 25, position $-4$, selected based on ablation scores, steering effectiveness, and KL-divergence constraints.
All evaluations use the JailbreakBench dataset, with substring matching and DeepSeek-v3.1 as judges (see Appendix [C](https://arxiv.org/html/2510.26418v1#A3)).

### 5.1 Experiment Setup

We probe refusal behavior in Qwen3-14B by directly intervening on activations:

- •
Ablation: remove the refusal direction during harmful instructions.
This tests whether refusals depend on a single direction—if so, ablating it should eliminate guardrails and increase attack success.
- •
Addition: inject the refusal direction during harmless instructions.
This tests whether the same feature can be used to steer the model in the opposite way—if so, adding it should cause the model to over-refuse and reject benign prompts.

Together, these interventions evaluate whether refusal in LRMs is governed by a low-dimensional signal, as shown by Arditi et al. (2024b) for standard LMs.

### 5.2 Bidirectional Control of Refusal

The interventions behave exactly as predicted. On harmful instructions (Table [4](https://arxiv.org/html/2510.26418v1#S5.T4)), ablating the refusal direction disables guardrails almost entirely: ASR rises from 11% to 91% with the DeepSeek judge. On harmless instructions (Table [5](https://arxiv.org/html/2510.26418v1#S5.T5)), adding the refusal direction has the opposite effect: the model becomes hyper-cautious, with ASR collapsing from 94% to 1%. These results confirm that refusal in Qwen3-14B can be bidirectionally controlled through a single direction in activation space. This aligns with recent work modeling deceptive behaviors mechanistically (Chaudhary & Barez, 2025), suggesting subnetworks governing safety can be isolated and monitored.

**Table 4: Experiments for ablating the refusal direction of activations on harmful instructions. Attack Success Rates (ASR) for Qwen3-14B on JailbreakBench (Chao et al., 2024b) (harmful instructions). Deepseek-v3.1 serves as Judge model.**
| Intervention | DeepSeek Judge |
| --- | --- |
| Baseline | 11% |
| Direction Ablation | 91% |

**Table 5: Experiments for adding the refusal direction of activations on harmless instructions. Attack Success Rates (ASR) for Qwen3-14B on ALPACA (Taori et al., 2023) (harmless instructions). Substring Matching serves as Judge method.**
| Intervention | Substring Matching Judge |
| --- | --- |
| Baseline | 94% |
| Direction Addition | 1% |

### 5.3 Evidence for a Low-Dimensional Refusal Feature

These experiments show that refusal behavior in Qwen3-14B—an LRM with structured reasoning—is mediated by the same low-dimensional feature observed in standard LMs (Arditi et al., 2024b).
This extends the refusal-direction phenomenon beyond conventional models to reasoning-augmented architectures.

### 5.4 Mechanism: Refusal Dilution

Arditi et al. (2024b) demonstrates direct manipulation of refusals by editing activations.
Our jailbreak complements this view: instead of editing the signal, we weaken its *formation*.
We call this effect refusal dilution.

During inference, the next-token activation reflects attention over prior tokens. Tokens of harmful intent amplify the refusal direction, while benign tokens diminish it.
By forcing the model to generate long chains of benign reasoning, harmful tokens make up only a small fraction of the attended context (as shown in Figure [23](https://arxiv.org/html/2510.26418v1#A10.F23) and Appendix [J](https://arxiv.org/html/2510.26418v1#A10)).
As a result, the refusal signal is diluted below threshold, allowing harmful completions to slip through.

## 6 Mechanistic Analysis: Refusal Component and Attention under CoT Growth

### 6.1 Defining the Refusal Component

Building on prior work (Arditi et al., 2024b), we quantify refusal behavior by projecting residual activations onto the refusal direction vector.
For each prompt, we extract the residual activation of the final input token and compute its component along this vector:

$$ $R=\langle h_{\text{last}},,v_{\text{refusal}}\rangle,$ (1) $$

where $h_{\text{last}}$ is the residual activation and $v_{\text{refusal}}$ is the normalized refusal direction.
We refer to $R$ as the refusal component: a scalar signal representing the model’s safety check at that position.

### 6.2 Effect of Chain-of-Thought Length

Figure: Figure 4: Refusal components across layers for different CoT lengths (Qwen3-14B). Longer reasoning on puzzle diminish the refusal signal (component) in late layers, reducing the likelihood of refusal.
Refer to caption: Extra_Figs/overlaid_layer_comparison_3.png

Figure: Figure 5: Ablating 60 attention heads (layers 15–35) flattens refusal components. Harmful instructions become indistinguishable from harmless ones, proving that selected heads are responsible for safety.
Refer to caption: Extra_Figs/new_head_ablation_comparison_1.png

We analyze Qwen3-14B across six CoT lengths (1k–47k tokens) and three instruction types (harmless, harmful, stealthy harmful).
Figure [6.2](https://arxiv.org/html/2510.26418v1#S6.SS2) shows that for harmful and stealthy-harmful inputs, longer CoT sequences consistently reduce refusal components in the later layers.
This supports our behavioral finding (Section [4](https://arxiv.org/html/2510.26418v1#S4)) that extended reasoning makes LRMs easier to jailbreak.
(A same pattern is also observed for GPT-OSS-20B in Figure [16](https://arxiv.org/html/2510.26418v1#A7.F16).)

### 6.3 Attention Patterns

To probe the mechanism, we analyze how attention is distributed between the harmful instruction tokens and the benign puzzle tokens.
We define the *attention ratio* as ’the sum of attention weights on harmful tokens in the prompt’ divided by ’those on puzzle tokens in the prompt’, thereby eliminating the influence of token length on this metric.
As shown in Figure [7](https://arxiv.org/html/2510.26418v1#S6.F7), this ratio declines as CoT length increases, indicating that harmful instructions receive progressively less weight, thus diluting the safety mechanisms that detect and refuse unsafe requests.
Layer-wise analysis (Figure [7](https://arxiv.org/html/2510.26418v1#S6.F7)) pinpoints layers 15–35 as the locus where this decline is most pronounced.
(Due to GPU memory constraints, the CoT length is limited to 1k-4k tokens in these attention-related experiments, but still showing an evident pattern).

Figure: Figure 6: Attention ratio vs. CoT length (Qwen3-14B). Longer CoT sequences reduce relative attention to harmful instructions, weakening the safety check.
Refer to caption: Extra_Figs/attention_ratio_trend_1.png

### 6.4 Attention-Head Intervention Experiments

We test whether the attention patterns identified above are merely correlational or actually causal for refusal. Building on the observation that layers 15–35 concentrate safety-checking attention, we ablate selected attention heads to see if refusal weakens.
In total, we select 60 specific heads that exhibit the ’longer CoT, lower ratio’ pattern, such as those in Figure [19](https://arxiv.org/html/2510.26418v1#A8.F19).
As shown in Figures [8](https://arxiv.org/html/2510.26418v1#S6.F8) and [9](https://arxiv.org/html/2510.26418v1#S6.F9), removing these heads flattens the distinction between harmful and harmless prompts, sharply reducing refusals. Targeted head removal is far more effective than random ablation, and among selected heads, front-layer heads (layer 15–23) have greater impact than later ones (layer 23-35). These causal interventions confirm that CoT Hijacking undermines a specific subnetwork of safety-critical heads, rather than degrading the model in a diffuse or incidental behavior.

Figure: (a) Ablating 6 selected heads.
Refer to caption: Extra_Figs/6_baseline_vs_predefined_ablation_1k_1.png

Figure: (a) Ablating 6 front-layer heads (layer 15–23).
Refer to caption: Extra_Figs/6front_baseline_vs_predefined_ablation_1k_1.png

### 6.5 Summary

Mechanistic evidence from both refusal components and attention maps shows that extended CoT length dilutes safety signals in late layers.
Together, these findings suggest that jailbreaks succeed not only because refusal is a one-dimensional feature (Section [5](https://arxiv.org/html/2510.26418v1#S5)), but also because its expression weakens as benign reasoning tokens dominate the context.
We return to this synthesis in Section [7](https://arxiv.org/html/2510.26418v1#S7).

## 7 Results and Discussion

Our results establish that chain-of-thought reasoning, while improving accuracy, creates a new safety vulnerability.
Across HarmBench, CoT Hijacking achieves state-of-the-art attack success rates (up to 99% on Gemini 2.5 Pro), outperforming prior jailbreak methods such as Mousetrap, H-CoT, and AutoRAN (Table [2](https://arxiv.org/html/2510.26418v1#S4.T2)).
These results hold across multiple proprietary LRMs (Gemini, ChatGPT, Grok, Claude) and under consistent evaluation criteria, highlighting the generality of the attack.

#### Mechanistic evidence.

Sections [5](https://arxiv.org/html/2510.26418v1#S5) and [6](https://arxiv.org/html/2510.26418v1#S6) show that refusal is mediated by a low-dimensional signal (the refusal direction) even in reasoning-augmented architectures.
However, this signal is fragile: long chains of benign reasoning dilute refusal activation, while attention shifts away from harmful tokens.
Implications for scaling:
Our findings directly challenge the assumption that “more reasoning means more robustness” (Guan et al., 2024).
Instead, scaling inference-time reasoning can exacerbate safety failures, especially in models explicitly optimized for long CoT.
This calls into question alignment strategies that rely on shallow refusal heuristics without mechanisms that scale with reasoning depth.

#### Toward mitigation.

The systematic nature of CoT Hijacking suggests that patching individual prompts is insufficient. Existing defenses like Wang et al. (2024) are often narrow-domain and do not account for reasoning-specific vulnerabilities. Effective defenses may require deeper integration of safety into the reasoning process itself, such as monitoring refusal activation across layers, penalizing dilution, or enforcing attention to harmful spans regardless of reasoning length.
We hope our findings motivate alignment strategies that are robust not only to short adversarial prompts but also to long chains of reasoning.

## 8 Conclusion

We introduced CoT Hijacking, a simple jailbreak attack against reasoning models. By padding harmful requests with long benign reasoning and a final-answer cue, we showed that refusal signals are diluted, resulting in high attack success rates across both open and proprietary LRMs. Unlike prior attacks that rely on visible safety reasoning or disguises, our method exploits a more fundamental weakness: safety checks depend on residual activations that become less discriminative as CoT length increases.

Through mechanistic analysis we found refusal components encode both the strength of safety checking in middle layers and the outcome of verification in later layers. Long CoT hijacking suppresses these signals, shifting attention away from harmful tokens and flattening refusal directions.
Interventions on targeted attention heads confirmed their causal role, showing that hijacking undermines a specific safety subnetwork. These results have two implications. First, reasoning models—despite higher task accuracy—are more vulnerable to jailbreaks when CoT traces are exploited.
Second, mechanistic insights into refusal dynamics suggest defenses: monitoring safety-checking layers, strengthening attention to harmful payloads, or making refusals robust to long reasoning.

## Acknowledgments

We thank Open AI and Anthropic for generous API credits.

## Ethics Statement

In this study, we propose CoT Hijacking as a new jailbreaking method for reasoning models. The evaluation of our method does not involve recruiting crowdsource workers or human annotators. Our study adheres strictly to the ICLR Code of Ethics with an emphasis that our method should only be used for research, but not for any malicious purpose.

## Reproducibility Statement

To ensure the reproducibility of our results, we introduce the experimental setup in Section [4](https://arxiv.org/html/2510.26418v1#S4) and Section [5](https://arxiv.org/html/2510.26418v1#S5). The code will be released to facilitate relevant research.

## References

- Arditi et al. (2024a)
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda.
Refusal in language models is mediated by a single direction.
*Advances in Neural Information Processing Systems*, 37:136037–136083, 2024a.
- Arditi et al. (2024b)
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda.
Refusal in language models is mediated by a single direction.
*Advances in Neural Information Processing Systems*, 37:136037–136083, 2024b.
- Ball et al. (2024)
Sarah Ball, Frauke Kreuter, and Nina Panickssery.
Understanding jailbreak success: A study of latent space dynamics in large language models.
*arXiv preprint arXiv:2406.09289*, 2024.
- Barez et al. (2025)
Fazl Barez, Tung-Yu Wu, Iván Arcuschin, Michael Lan, Vincent Wang, Noah Siegel, Nicolas Collignon, Clement Neo, Isabelle Lee, Alasdair Paren, Adel Bibi, Robert Trager, Damiano Fornasiere, John Yan, Yanai Elazar, and Yoshua Bengio.
Chain-of-thought is not explainability, 2025.
URL [https://www.alphaxiv.org/overview/2025.02v3](https://www.alphaxiv.org/overview/2025.02v3).
Preprint, AlphaXiv.
- Chao et al. (2023)
Chengzhi Chao et al.
Harmbench: A standardized benchmark for evaluating harms of language models.
In *NeurIPS Datasets and Benchmarks Track*, 2023.
- Chao et al. (2024a)
Chengzhi Chao et al.
Mousetrap: An automated framework for jailbreaking and defending large language models.
In *International Conference on Learning Representations (ICLR)*, 2024a.
- Chao et al. (2024b)
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al.
Jailbreakbench: An open robustness benchmark for jailbreaking large language models.
*Advances in Neural Information Processing Systems*, 37:55005–55029, 2024b.
- Chaudhary & Barez (2025)
Maheep Chaudhary and Fazl Barez.
Safetynet: Detecting harmful outputs in llms by modeling and monitoring deceptive behaviors.
*arXiv preprint arXiv:2505.14300*, 2025.
- Chen et al. (2024)
Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li.
Finding safety neurons in large language models.
*arXiv preprint arXiv:2406.14144*, 2024.
- Chen et al. (2023)
Xiangyu Chen et al.
Autoran: Automated red teaming via reinforcement learning.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Denison et al. (2024)
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al.
Sycophancy to subterfuge: Investigating reward-tampering in large language models.
*arXiv preprint arXiv:2406.10162*, 2024.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Fatemi et al. (2025)
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula.
Concise reasoning via reinforcement learning.
*arXiv preprint arXiv:2504.05185*, 2025.
- Frankish (2010)
Keith Frankish.
Dual-process and dual-system theories of reasoning.
*Philosophy Compass*, 5(10):914–926, 2010.
- Fu et al. (2024)
Tingchen Fu, Mrinank Sharma, Philip Torr, Shay B Cohen, David Krueger, and Fazl Barez.
Poisonbench: Assessing large language model vulnerability to data poisoning.
*arXiv preprint arXiv:2410.08811*, 2024.
- Fu et al. (2025)
Tingchen Fu, Jiawei Gu, Yafu Li, Xiaoye Qu, and Yu Cheng.
Scaling reasoning, losing control: Evaluating instruction following in large reasoning models.
*arXiv preprint arXiv:2505.14810*, 2025.
- Gao et al. (2024)
Lang Gao, Jiahui Geng, Xiangliang Zhang, Preslav Nakov, and Xiuying Chen.
Shaping the safety boundaries: Understanding and defending against jailbreaks in large language models.
*arXiv preprint arXiv:2412.17034*, 2024.
- Ghosh et al. (2025)
Shaona Ghosh, Amrita Bhattacharjee, Yftah Ziser, and Christopher Parisien.
Safesteer: Interpretable safety steering with refusal-evasion in llms.
*arXiv preprint arXiv:2506.04250*, 2025.
- Guan et al. (2024)
Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al.
Deliberative alignment: Reasoning enables safer language models.
*arXiv preprint arXiv:2412.16339*, 2024.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Guo et al. (2024)
Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu.
Cold-attack: jailbreaking llms with stealthiness and controllability.
In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR.org, 2024.
- He et al. (2024)
Zeqing He, Zhibo Wang, Zhixuan Chu, Huiyu Xu, Wenhui Zhang, Qinglong Wang, and Rui Zheng.
Jailbreaklens: Interpreting jailbreak mechanism in the lens of representation and circuit.
*arXiv preprint arXiv:2411.11114*, 2024.
- Hughes et al. (2024)
John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma.
Best-of-n jailbreaking.
*arXiv preprint arXiv:2412.03556*, 2024.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Jiang et al. (2024)
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran.
Artprompt: Ascii art-based jailbreak attacks against aligned llms.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 15157–15173, 2024.
- Jimenez et al. (2023)
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.
Swe-bench: Can language models resolve real-world github issues?
*arXiv preprint arXiv:2310.06770*, 2023.
- Kojima et al. (2022)
Takeshi Kojima, Shixiang Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
Large language models are zero-shot reasoners.
*arXiv preprint arXiv:2205.11916*, 2022.
- Kuo et al. (2025)
Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen.
H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking.
*arXiv preprint arXiv:2502.12893*, 2025.
- Lambert et al. (2024)
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al.
Tulu 3: Pushing frontiers in open language model post-training.
*arXiv preprint arXiv:2411.15124*, 2024.
- Li et al. (2025a)
Tianlong Li, Zhenghua Wang, Wenhao Liu, Muling Wu, Shihan Dou, Changze Lv, Xiaohua Wang, Xiaoqing Zheng, and Xuan-Jing Huang.
Revisiting jailbreaking for large language models: A representation engineering perspective.
In *Proceedings of the 31st International Conference on Computational Linguistics*, pp. 3158–3178, 2025a.
- Li et al. (2025b)
Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal.
When thinking fails: The pitfalls of reasoning for instruction-following in llms.
*arXiv preprint arXiv:2505.11423*, 2025b.
- Li et al. (2023)
Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han.
Deepinception: Hypnotize large language model to be jailbreaker, 2023.
- Li et al. (2025c)
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al.
From system 1 to system 2: A survey of reasoning large language models.
*arXiv preprint arXiv:2502.17419*, 2025c.
- Liang et al. (2025)
Jiacheng Liang, Tanqiu Jiang, Yuhui Wang, Rongyi Zhu, Fenglong Ma, and Ting Wang.
Autoran: Weak-to-strong jailbreaking of large reasoning models.
*arXiv preprint arXiv:2505.10846*, 2025.
- Lin et al. (2024)
Yuping Lin, Pengfei He, Han Xu, Yue Xing, Makoto Yamada, Hui Liu, and Jiliang Tang.
Towards understanding jailbreak attacks in llms: A representation space analysis.
*arXiv preprint arXiv:2406.10794*, 2024.
- Liu et al. (2025)
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Understanding r1-zero-like training: A critical perspective.
*arXiv preprint arXiv:2503.20783*, 2025.
- Luo et al. (2025)
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al.
Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl.
*Notion Blog*, 2025.
- Ma et al. (2024)
Siyuan Ma, Weidi Luo, Yu Wang, and Xiaogeng Liu.
Visual-roleplay: Universal jailbreak attack on multimodal large language models via role-playing image character.
*arXiv preprint arXiv:2405.20773*, 2024.
- Mazeika et al. (2024)
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al.
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
*arXiv preprint arXiv:2402.04249*, 2024.
- Muennighoff et al. (2025)
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
s1: Simple test-time scaling.
*arXiv preprint arXiv:2501.19393*, 2025.
- Shaikh et al. (2022)
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang.
On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.
*arXiv preprint arXiv:2212.08061*, 2022.
- Shen et al. (2024)
Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi.
The language barrier: Dissecting safety challenges of llms in multilingual contexts.
*arXiv preprint arXiv:2401.13136*, 2024.
- Sun et al. (2024)
Chung-En Sun, Xiaodong Liu, Weiwei Yang, Tsui-Wei Weng, Hao Cheng, Aidan San, Michel Galley, and Jianfeng Gao.
Iterative self-tuning llms for enhanced jailbreaking capabilities.
*arXiv preprint arXiv:2410.18469*, 2024.
- Taori et al. (2023)
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.
Stanford alpaca: An instruction-following llama model, 2023.
- Touvron et al. (2023a)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
Llama: Open and efficient foundation language models.
*ArXiv*, abs/2302.13971, 2023a.
- Touvron et al. (2023b)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023b.
- Wang et al. (2022)
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun.
Towards understanding chain-of-thought prompting: An empirical study of what matters.
*arXiv preprint arXiv:2212.10001*, 2022.
- Wang et al. (2024)
Tony T Wang, John Hughes, Henry Sleight, Rylan Schaeffer, Rajashree Agrawal, Fazl Barez, Mrinank Sharma, Jesse Mu, Nir Shavit, and Ethan Perez.
Jailbreak defense in a narrow domain: Limitations of existing methods and a new transcript-classifier approach.
*arXiv preprint arXiv:2412.02159*, 2024.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.
Chain‐of‐thought prompting elicits reasoning in large language models.
*arXiv preprint arXiv:2201.11903*, 2022.
URL [https://arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903).
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Yao et al. (2025)
Yang Yao, Xuan Tong, Ruofan Wang, Yixu Wang, Lujundong Li, Liang Liu, Yan Teng, and Yingchun Wang.
A mousetrap: Fooling large reasoning models for jailbreak with chain of iterative chaos.
*arXiv preprint arXiv:2502.15806*, 2025.
- Yuan et al. (2024)
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu.
GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Zeng et al. (2025)
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He.
7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient.
[https://hkust-nlp.notion.site/simplerl-reason](https://hkust-nlp.notion.site/simplerl-reason), 2025.
Notion Blog.
- Zhao et al. (2024)
Xueliang Zhao, Wenda Li, and Lingpeng Kong.
Subgoal-based demonstration learning for formal theorem proving.
In *Forty-first International Conference on Machine Learning*, 2024.
URL [https://openreview.net/forum?id=pSnhA7Em1P](https://openreview.net/forum?id=pSnhA7Em1P).
- Zhou et al. (2022)
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Chen, Yi Tay, Donald Metzler, Quoc Le, et al.
Least-to-most prompting enables complex reasoning in large language models.
*arXiv preprint arXiv:2205.10625*, 2022.
- Zhou et al. (2025a)
Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang.
The hidden risks of large reasoning models: A safety assessment of r1.
*arXiv preprint arXiv:2502.12659*, 2025a.
- Zhou et al. (2025b)
Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li.
On the role of attention heads in large language model safety.
In *The Thirteenth International Conference on Learning Representations*, 2025b.
URL [https://openreview.net/forum?id=h0Ak8A5yqw](https://openreview.net/forum?id=h0Ak8A5yqw).
- Zou et al. (2023)
Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language models, 2023.

## Appendix A Use of Language Models

Language models are used in our study as a general-purpose assist tools and we use them for checking grammar mistakes and fixing Latex compile errors.

## Appendix B Limitations

Our experiments focus on text-based LRMs and a subset of benchmark tasks (HarmBench).
Evaluation relies on automated judges, which—although stricter than those used in prior work—may still under- or over-estimate refusal.
Moreover, we study refusal primarily through one mechanistic lens (activation directions and attention), leaving open the possibility of complementary pathways for safety failures.

## Appendix C Evaluation Details

#### Judges.

Unless otherwise noted, attack success rate (ASR) is evaluated using Gemini 2.5 Pro as the automated judge.
For robustness, we also report substring matching and DeepSeek-v3.1 judges in Section [5](https://arxiv.org/html/2510.26418v1#S5).
We adopt the JailbreakBench unified evaluation standard (Chao et al., 2024b) for consistency.

#### Prompts.

CoT Hijacking prompts follow the template in Section [4.1](https://arxiv.org/html/2510.26418v1#S4.SS1), consisting of optional role-setting, benign reasoning, and a harmful payload prefixed by a final-answer cue.
Full examples are available in our released code repository.

#### Models.

The main experiments use proprietary LRMs (Gemini 2.5 Pro, ChatGPT o4 Mini, Grok 3 Mini, Claude 4 Sonnet) and open models (Qwen3-14B, GPT-OSS-20B, S1-32B).
Where applicable, we standardize temperature to 0.7 and max generation length to 8k tokens.

## Appendix D Jailbreak Example of attacking GPT-5 ( Dangerous Content )

## Appendix E Additional Results on Refusal Components

These results are simply around refusal components between harmful and harmless instructions without CoT.

Figure: Figure 10: Distribution comparison of refusal components between harmful and harmless instructions without CoT. The violin plots show the density distribution of parallel component values across all layers and samples. Harmful instructions (red) exhibit higher mean values and different distributional characteristics compared to harmless instructions (blue), with harmful instructions showing more positive skewness indicating stronger expression on the refusal direction.
Refer to caption: Extra_Figs/distribution_comparison_1.png

Figure: Figure 11: Refusal components for harmful instructions (Qwen3-14B). Rows correspond to samples, columns to layers. Positive values (red) indicate strong refusal activations. Harmful inputs predominantly activate later layers (25–39).
Refer to caption: Refusal_Component_Figures/heatmap_harmful.png

Figure: Figure 12: Refusal components for harmless instructions (Qwen3-14B). Harmless instructions shift from positive values in middle layers (safety checking) to negative values in final layers (safety verification allowing compliance).
Refer to caption: Refusal_Component_Figures/heatmap_harmless.png

Figure: Figure 13: Difference heatmap (harmful–harmless). Red regions show where harmful instructions elicit stronger refusal components than harmless ones, especially in layers 25–35.
Refer to caption: Refusal_Component_Figures/heatmap_difference.png

## Appendix F Refusal Component Experiments Based on Jailbreak Success

Figure: Figure 14: Refusal component by outcome. Mean parallel components across layers for successful vs. failed jailbreak attempts. The activation is measured at the last token of the full generation (prompt + CoT + final response).
Refer to caption: Extra_Figs/jailbreak_layer_comparison_mean.png

## Appendix G Additional Results on CoT Length

### G.1 Algorithm for Template Generation

Figure: Algorithm 1 Generate templates and compute refusal components for different CoT lengths

### G.2 Qwen3-14B Results

Figure: Figure 15: Refusal component comparison across layers (Qwen3-14B) for different CoT lengths. Includes harmless, harmful, and stealthy harmful instructions. CoT lengths: 1k, 3k, 11k, 21k, 31k, 47k tokens.
Refer to caption: Extra_Figs/overlaid_layer_comparison_1.png

### G.3 GPT-OSS-20B Results

Figure: Figure 16: Refusal component comparison across layers (GPT-OSS-20B). Four template lengths: 1k, 3k, 5k, 8k tokens. Results show the same trend: longer CoT reduces refusal activation. Note that due to GPU memory constraints, the template length range is much narrower than in the Qwen3 experiments, yielding a less pronounced but consistent pattern.
Refer to caption: Extra_Figs/overlaid_layer_comparison_gptoss_.png

## Appendix H Attention Ratio Analysis

$$ $\text{AttnRatio}=\frac{\sum\limits_{t\in H}\alpha_{t}}{\sum\limits_{t\in P}\alpha_{t}},\quad\alpha_{t}=\text{attention weight on token }t$ (2) $$

Figure: Figure 17: Attention ratio trend across CoT lengths. Ratio declines from 0.190 (1k) to 0.158 (4k). Longer CoT reduces attention paid to harmful instructions.
Refer to caption: Extra_Figs/attention_ratio_trend.png

Figure: Figure 18: Layer-wise attention ratio patterns (1k–4k CoT lengths). The effect of CoT length is concentrated in layers 25–32, with layers 28 and 30 showing strongest differences.
Refer to caption: Extra_Figs/attention_layer_analysis.png

Figure: (a) Layer 25 head-level ratios.
Refer to caption: Extra_Figs/layer_25_head_analysis.png

## Appendix I New Figures

Figure: Figure 20: Refusal component comparison across layers (Qwen3-14B) for different CoT lengths. Only for harmful instructions. CoT lengths: 1k, 3k, 11k, 21k, 31k, 47k tokens.
Refer to caption: Extra_Figs/gradient_harmful_components.png

Figure: Figure 21: Refusal component comparison across layers (Qwen3-14B) for different CoT lengths. Only for stealthy harmful instructions. CoT lengths: 1k, 3k, 11k, 21k, 31k, 47k tokens.
Refer to caption: Extra_Figs/gradient_stealth_harmful_components.png

Figure: Figure 22: Refusal component comparison across layers (Qwen3-14B) for different CoT lengths. Only for harmless instructions. CoT lengths: 1k, 3k, 11k, 21k, 31k, 47k tokens.
Refer to caption: Extra_Figs/gradient_harmless_components.png

## Appendix J Attention Visualization (Method + Examples)

### Methodology

We analyze attention maps in Qwen3-14B on our jailbreak samples (thinking mode enabled), focusing on how the first harmful-response tokens (“target field”) attend over Prompt, Thinking, and Response regions.
As shown in Figures [23](https://arxiv.org/html/2510.26418v1#A10.F23) and [24](https://arxiv.org/html/2510.26418v1#A10.F24), when the model begins generating the first harmful-response token, most attention is paid on the benign reasoning context, with little attention allocated to the ’harmful instruction’—thereby achieving ’refusal dilution’.

Figure: Figure 23: Sample 0 (mean-pooled) attention from target-field tokens to the full context. Dashed lines mark Prompt, Thinking, and Response boundaries.
Refer to caption: Attention_Map_Figures/attention_heatmap_sample_0.png

Figure: Figure 24: Sample 0 (max-pooled) attention highlighting peak focus. Long benign reasoning receives strong attention while harmful-payload spans receive comparatively less.
Refer to caption: Attention_Map_Figures/attention_heatmap_sample_0_max.png

## Appendix K Complete Attention Visualization Results

This section provides the full set of attention heatmaps (mean and max pooled) for all 10 samples analyzed.

Figure: Figure 25: Attention heatmaps for Samples 0–2. Mean pooling (left) shows overall tendencies; max pooling (right) emphasizes peaks.
Refer to caption: Attention_Map_Figures/attention_heatmap_sample_0.png

Figure: Figure 26: Attention heatmaps for Samples 3–5.
Refer to caption: Attention_Map_Figures/attention_heatmap_sample_3.png

Figure: Figure 27: Attention heatmaps for Samples 6–8.
Refer to caption: Attention_Map_Figures/attention_heatmap_sample_6.png

Figure: Figure 28: Attention heatmaps for Sample 9.
Refer to caption: Attention_Map_Figures/attention_heatmap_sample_9.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses.
soundness: 1
strengths: - The paper proposes a simple jailbreak attack that can achieve high ASRs on strong proprietary reasoning models, raising safety concerns.
- The paper extends prior studies on understanding the mechanisms of refusal/harmful behaviors in base models to reasoning models, showing that reasoning models similarly exhibit a refusal direction.
summary: This paper introduces a simple jailbreak attack on reasoning language models called Chain-of-Thought Hijacking, which prepends harmless reasoning to a harmful prompt. The authors found that this method can achieve high attack success rates on several proprietary reasoning models. They additionally perform mechanistic analysis on the refusal behaviors of reasoning models by taking the difference in activations between benign and harmful prompts. They find that harmfulness/refusal is mediated by a single low-dimensional direction, similar to non-reasoning models in prior work. Lastly, they suggest that the success of Chain-of-Thought Hijacking is due to long reasoning causing attention to shift away from harmful tokens.
weaknesses: **1. Overall:**

While this work presents a range of experiments, I find it difficult to connect the dots and see how they support a central, significant claim. There are many individual experiments but insufficient details or motivation for them. For example, the authors introduced the concept of “refusal dilution” in section 5.4, but it is never elaborated or connected to section 6. Another example is the refusal direction experiments in section 5 - how are they related to your jailbreak method? 

Also, the jailbreak experiments are conducted primarily on proprietary models, whereas the mechanistic analysis focuses solely on a single open-source model, Qwen3-13B. Is the mechanistic analysis trying to explain why your jailbreak method succeeds? If so, then the jailbreak evaluation and mechanistic analysis should be conducted on the same suite of reasoning models (or at least on overlapping sets of models).

**2. Missing details and unclear experimental design:**
- Table 2: It is unclear why the three baseline methods were chosen. Are they considered state-of-the-art?
- Using only GPT-4o-mini as the evaluator seems insufficient. The ASR could be inflated since GPT-4o-mini is not a highly capable model. Have you verified the alignment between GPT-4o-based evaluations and human annotations?
- Dataset selection: Why are jailbreak results reported on HarmBench, while refusal direction results are on JailbreakBench? This design choice is unclear.
- You mention a “Seduction” pipeline for automated jailbreaks, but it is not described anywhere. It is also unclear which auxiliary LLMs were used to generate the attacks.
- Tables 4 and 5: Why was DeepSeek Judge used for ablation and substring matching for addition? I understand that substring matching may be the simplest approach to detect refusals, but the evaluation should be standardized across experiments.

**3. Incorrect or missing citations:**
- Several important citations in the introduction are incorrect, including: HarmBench, Mousetrap, AutoRAN (L38-39). The author names are completely different from the actual works. 
- Section 3 seems to be mainly prior work, see https://arxiv.org/abs/2502.12025. 

**4. Writing issues:**
- Inconsistent usage of \citep and \citet in related work first paragraph
- L170: textbfNatural
- L234-243: the two paragraphs seem to be repeating each other
- Table captions should be above the tables
- L398: Missing figure link

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: minor typoes:

line 170: "textbf" --> "\textbf{}"

line 376: Figure 17: Did this mean to say Figure 6? 
line 398: "Figures ??"
soundness: 3
strengths: The paper is well-scoped, and experiments are well designed and executed to back the author's claims. In particular, a 99% success rate is compelling. The overall narrative is coherent, starting from demonstrating a new phenomena (99% jailbreak success rate with scaled contexts) to explaining the underlying mechanisms.
summary: The paper demonstrates a new phenomenon, dubbed "chain of thought hijacking", in which extending the context leads to 99% attack success rates in jailbreaking a safety-tuned model. 

The authors provide a "mechanistic explanation" for this phenomenon, in which as context is scaled, attention is more distributed across the tokens, and "weakens" the refusal features in the model.

This simple trick leads to 99% attack success rate against contemporary models.

The authors then look at building a "mechanistic explanation" for the observed behavior.

Using prior work, extracting and leveraging a "refusal feature direction" can bidirectionally control for refusal -- subtracting the refusal direction leads to higher jailbreak success rates, while adding the vector leads to a significant drop. 

The authors make the connection to longer context lengths with what they call "refusal dilution" - as context is scaled with benign tokens, harmful tokens only make up a small fraction to which the model attends to, and no longer triggering the refusal features.

The authors finally localize attention heads that seemingly are relevant for refusal + "refusal dilution", and show that ablating a small set of heads can flatten the distinction between harmful vs. harmless prompts, significantly reducing refusal rates.

While nicely executed, my main reservation is that these observed phenomena are not new. In particular, Li et al (https://arxiv.org/pdf/2402.10962) demonstrate "instruction drift", akin to "refusal drift", in which as context is scaled, less attention is spent on the system prompts (ie, instructions), akin to attending to toxic tokens, and models no longer follow instructions as context is scaled (akin to models no longer refusing harmful prompts). With that being said, while the paper sheds light to a attack surface against language models, scientifically, there is not much new that is learned for the reader.
weaknesses: As indicated in the summary, my main criticism is that the reported phenomena is not new. Li et al. (https://arxiv.org/pdf/2402.10962) as demonstrated this phenomena already, and there are many parallels (instruction drift vs. refusal drift) stemming from the same underlying reasons/mechanisms (less attention being spent on system prompts/instructions vs. toxic tokens). Put differently, the paper carves out a neat narrative and sheds light to a new vulnerability of language models, but to be honest, scientifically there isn't much new to take away from reading the work.

It is also unclear how much of the content of the scaled context plays a role in successful attacks. In particular, the examples used in the paper relate to a reasoning/logic puzzle that is closely related to the target jailbreaking behavior. However, if the author's claims are correct, what is in the context should not matter so long as context is scaled and attention is diluted. I think a study should be added to ablate these two things. 

Lastly, presentation could be improved. Many of the figures in the main text are never referenced, making it a bit difficult to follow the text. ex: Figure 4, 5, 6, 7.. Also, the legends in all the figures are not legible.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Based on Figure 1 grey texts, you seem to attack from both input prompt and CoT (prepending benign CoT). Am I understanding it right? If that's the case, is there an ablation done on how each factor affects the ASR?
soundness: 2
strengths: The paper contrasts the attack vector against prior work H-CoT, which requires exposed safety reasoning. The experimental methodology also uses comprehensive evaluations across multiple frontier models (Gemini, ChatGPT, Grok, Claude), showing that their attack attains high success rate. I think the work is well structured, especially on how the mechanistic analysis builds incrementally. The mechanistic analysis connecting refusal directions, attention patterns, and causal interventions provides original insights into reasoning model safety.
summary: This paper introduces Chain-of-Thought Hijacking (CoT Hijacking), a novel jailbreak attack against large reasoning models (LRMs). The attack pads harmful requests with long sequences of benign reasoning followed by a final-answer cue, achieving state-of-the-art attack success rates. Through mechanistic analysis, the authors demonstrate that refusal behavior relies on a fragile, low-dimensional safety signal that becomes diluted as reasoning length increases. Attention shifts away from harmful tokens toward benign reasoning, weakening refusal features in later layers.
weaknesses: First of all, the paper frames puzzles as "benign reasoning," but this characterization is quite questionable in my opinion. Given that the paper's core claim is that CoT length dilutes safety signals through attention mechanisms, it should include an ablation study that takes well-known benign reframing attacks such as persuasion attack [1] and explore how length changes affect the ASR. In my opinion, this attack works *because of the prompt rather than a general length effect*. Furthermore, no further information is provided about the red-teaming automation pipeline "Seduction" (line 226).

Another evidence that makes me think that such CoT Hijacking works is due to prompting rather than length effects is because of the output. If you take a closer look at Appendix D example as well as Figure 1 output, while the model outputs bypasses the safety guardrail, they do not provide clear instructions on how to fulfill the malicious requests but trying to adhere to the engineered prompt requests about logic grid puzzles. 

In fact, the showcased outputs are not really fluent generation (e.g., "The hum of the security checkpoint was a low, oppressive drone" from Figure 1 doesn't make any sense.) I don't even see these examples as successful attack since no truly harmful instructions that fulfill the requests were provided by the model. In other words, the judge protocol is dubious. I would recommend using StrongReject evaluator, following the protocol of [2] with implementation from [3].

Lastly, the paper seems to frame the attack vector as something that reveals safety weaknesses on reasoning models, but I highly suspect the puzzle-like attack is generalizable to even the base model. If this is true, this doesn't demonstrate a reasoning-specific vulnerability introduced by extended CoT capabilities, but rather exposes the weaknesses in with handling puzzle-like prompts. In other words, without direct comparison to non-reasoning base models, the paper's central claim that reasoning creates new attack surfaces (line 034-035) remains unsubstantiated.

---

[1] Zeng, Yi, et al. "How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[2] NIST and UK AISI. Us aisi and uk aisi joint pre-deployment test: Openai o1. Technical report, National Institute of Standards and Technology and Department of Science Innovation and Technology, December 2024. Joint pre-deployment evaluation of OpenAI o1 model.

[3] URL: https://strong-reject.readthedocs.io/en/latest/api/evaluate.html#strong_reject.evaluate.strongreject_aisi

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: I think the authors have several decent contributions and insights in their work and would encourage them to invest a significant amount of time to polish the presentation of their work.

The authors can improve my opinion about the paper by addressing my concerns raised in the weaknesses section.
soundness: 2
strengths: The authors address a timely and practically important issue: the ease with which reasoning-focused models can be jailbroken. Their proposed method is conceptually simple, likely simple enough that many users could independently discover and exploit similar strategies. This makes it especially relevant for model providers, as defending against such benign but effective jailbreaks poses a serious and ongoing challenge.

The approach seems to work well on harmbench and on closed models.

It is commendable that the authors include a brief mechanistic interpretability analysis via projection onto a refusal-steering vector. This analysis provides an initial glimpse into the internal dynamics underlying the method’s success.
summary: The authors propose a new reasoning model jailbreak based on smuggling harmful request within a benign reasoning tasks.
weaknesses: The overall quality of the work feels closer to a class project report or blog post than to a polished research contribution. Much of the presentation gives the impression of filling space rather than conveying substance. The paper makes excessive use of two-row tables (Tables 1, 3, 4, 5) and includes numerous large but only marginally informative figures, many of which appear redundant. In addition, the visual presentation suffers from poor readability, particularly due to the extremely small font sizes in Figures 4 to 9, which makes it difficult to extract key insights.

The empirical evaluation is also too limited to support strong claims. The authors test only four closed-source models and omit all open-source reasoning models, even though these are highly relevant to the stated problem and would connect more directly to the subsequent analysis. The benchmark coverage is restricted to a single dataset, and only two baseline methods are included. This narrow scope leaves unclear whether the observed vulnerabilities generalize beyond the chosen setup. For instance, more sophisticated jailbreaks such as the policy-over-values attack (https://www.kaggle.com/competitions/openai-gpt-oss-20b-red-teaming/writeups/lucky-coin-jailbreak) would provide a valuable point of comparison.

The post-hoc analysis that attempts to explain why models are susceptible to such simple jailbreaks is a positive step but remains superficial. The paper would benefit from a more compact and insightful presentation of results, for example by merging Tables 1, 3, 4, and 5 into a single comprehensive table and moving some of the repetitive figures to the appendix. The resulting space could then be used to conduct a more thorough evaluation of the phenomenon, including additional models, datasets, and ablation studies. Such an expansion would elevate the work to the standard expected at a top-tier venue like ICLR. A good example of this type of in-depth analysis can be found in this recent study https://arxiv.org/abs/2406.05946, which investigates refusal in chat models.