Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model
  Reasoning
Authors: name: Honglin Lin; name: Qizhi Pei; name: Xin Gao; name: Zhuoshi Pan; name: Yu Li; name: Juntao Li; name: Conghui He; name: Lijun Wu
Published: 2025-10-05 07:59:24

Abstract: Reasoning capability is pivotal for Large Language Models (LLMs) to solve
complex tasks, yet achieving reliable and scalable reasoning remains
challenging. While Chain-of-Thought (CoT) prompting has become a mainstream
approach, existing methods often suffer from uncontrolled generation,
insufficient quality, and limited diversity in reasoning paths. Recent efforts
leverage code to enhance CoT by grounding reasoning in executable steps, but
such methods are typically constrained to predefined mathematical problems,
hindering scalability and generalizability. In this work, we propose Caco
(Code-Assisted Chain-of-ThOught), a novel framework that automates the
synthesis of high-quality, verifiable, and diverse instruction-CoT reasoning
data through code-driven augmentation. Unlike prior work, Caco first fine-tunes
a code-based CoT generator on existing math and programming solutions in a
unified code format, then scales the data generation to a large amount of
diverse reasoning traces. Crucially, we introduce automated validation via code
execution and rule-based filtering to ensure logical correctness and structural
diversity, followed by reverse-engineering filtered outputs into natural
language instructions and language CoTs to enrich task adaptability. This
closed-loop process enables fully automated, scalable synthesis of reasoning
data with guaranteed executability. Experiments on our created Caco-1.3M
dataset demonstrate that Caco-trained models achieve strong competitive
performance on mathematical reasoning benchmarks, outperforming existing strong
baselines. Further analysis reveals that Caco's code-anchored verification and
instruction diversity contribute to superior generalization across unseen
tasks. Our work establishes a paradigm for building self-sustaining,
trustworthy reasoning systems without human intervention.