AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety
  Alignment of Large Reasoning Models
Authors: name: Zihao Zhu, name: Xinyu Wu, name: Gehan Hu, name: Siwei Lyu, name: Ke Xu, name: Baoyuan Wu
Organization: The Chinese University of Hongkong,Shenzhen

Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in
complex problem-solving through Chain-of-Thought (CoT) reasoning. However, the
multi-step nature of CoT introduces new safety challenges that extend beyond
conventional language model alignment. We identify a failure mode in current
safety CoT tuning methods: the snowball effect, where minor reasoning
deviations progressively amplify throughout the thought process, leading to
either harmful compliance or excessive refusal. This effect stems from models
being trained to imitate perfect reasoning scripts without learning to
self-correct. To address this limitation, we propose AdvChain, an alignment
paradigm that teaches models dynamic self-correction through adversarial CoT
tuning. Our method involves constructing a dataset containing
Temptation-Correction and Hesitation-Correction samples, where models learn to
recover from harmful reasoning drifts and unnecessary cautions. Extensive
experiments show that AdvChain significantly enhances robustness against
jailbreak attacks and CoT hijacking while substantially reducing over-refusal
on benign prompts, achieving a superior safety-utility balance without
compromising reasoning capabilities. Our work establishes a new direction for
building more robust and reliable reasoning models.