ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
Authors: name: Jiazhan Feng; name: Shijue Huang; name: Xingwei Qu; name: Ge Zhang; name: Yujia Qin; name: Baoquan Zhong; name: Chengquan Jiang; name: Jinxin Chi; name: Wanjun Zhong
Published: 2025-04-15 18:10:22

Abstract: While reasoning models (e.g., DeepSeek R1) trained with reinforcement
learning (RL), excel in textual reasoning, they struggle in scenarios requiring
structured problem-solving, such as geometric reasoning, concise computation,
or complex equation solving-areas where computational tools like code
interpreters (CI) demonstrate distinct advantages. To bridge this gap, we
propose ReTool, which enhances long-form reasoning with tool-integrated
learning, including two key features: (1) dynamic interleaving of real-time
code execution within natural language reasoning processes, and (2) an
automated RL paradigm that allows policy rollouts with multi-turn real-time
code execution and teaches the model in learning when and how to invoke tools
based on outcome feedback. ReTool employs a systematic training framework,
beginning with synthetic cold-start data generation to produce code-augmented
long-form reasoning traces for fine-tuning base models. Subsequent RL training
leverages task outcomes as rewards to iteratively refine the model's tool use
strategy, enabling autonomous discovery of optimal tool invocation patterns
without human priors. Experiments on the challenging MATH Olympiad benchmark
AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with
400 training steps, outperforming text-based RL baseline (40% accuracy, 1080
steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5%
accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further
analysis reveals emergent behaviors such as code self-correction, signaling an
''aha moment'' in which the model autonomously masters adaptive tool use. These
findings highlight the promise of outcome-driven tool integration for advancing
complex mathematical reasoning and offer new insights into hybrid
neuro-symbolic systems.