R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised
  and Reinforcement Learning
Authors: name: Yongchao Chen, name: Yueying Liu, name: Junwei Zhou, name: Yilun Hao, name: Jingquan Wang, name: Yang Zhang, name: Chuchu Fan

Despite advances in reasoning and planning of R1-like models, Large Language
Models (LLMs) still struggle with tasks requiring precise computation, symbolic
manipulation, optimization, and algorithmic reasoning, in which textual
reasoning lacks the rigor of code execution. A key challenge is enabling LLMs
to decide when to use textual reasoning versus code generation. While OpenAI
trains models to invoke a Code Interpreter as needed, public research lacks
guidance on aligning pre-trained LLMs to effectively leverage code and
generalize across diverse tasks. We present R1-Code-Interpreter, an extension
of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and
reinforcement learning (RL) to autonomously generate multiple code queries
during step-by-step reasoning. We curate 144 reasoning and planning tasks (107
for training, 37 for testing), each with over 200 diverse questions. We
fine-tune Qwen-2.5 models (3B/7B/14B) using various SFT and RL strategies,
investigating different answer formats, reasoning vs. non-reasoning models,
cold vs. warm starts, GRPO vs. PPO, and masked vs. unmasked code outputs.
Unlike prior RL work on narrow domains, we find that Code Interpreter training
is significantly harder due to high task diversity and expensive code
execution, highlighting the critical role of the SFT stage. Our final model,
R1-CI-14B, improves average accuracy on the 37 test tasks from 44.0\% to
64.1\%, outperforming GPT-4o (text-only: 58.6\%) and approaching GPT-4o with
Code Interpreter (70.9\%), with the emergent self-checking behavior via code
generation. Datasets, Codes, and Models are available at
https://github.com/yongchao98/R1-Code-Interpreter and
https://huggingface.co/yongchao98.