ReTool: Reinforcement Learning for Strategic Tool Use in LLMs
Authors: Jiazhan Feng
Shijue Huang
Xingwei Qu
Ge Zhang
Yujia Qin
Baoquan Zhong
Chengquan Jiang
Jinxin Chi
Wanjun Zhong
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Agent
Primary area: foundation or frontier models, including LLMs

Abstract: While reasoning models trained with reinforcement learning (RL) excel in reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning, concise computation, or complex equation solving—areas where computational tools like code interpreters (CI) demonstrate distinct advantages. To bridge this gap, we propose ReTool, which enhances long-form reasoning with tool-integrated learning, including two key features: (1) dynamic interleaving of real-time code execution within natural language reasoning processes, and (2) an automated RL paradigm that allows policy rollouts with multi-turn real-time code execution and teaches the model in learning when and how to invoke tools based on outcome feedback. ReTool employs a systematic training framework, beginning with synthetic code-augmented long-form reasoning data for cold-start training. Subsequent RL training leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns without human priors. Experiments on challenging MATH Olympiad benchmark AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with 400 training steps, outperforming text-based RL baseline (40% accuracy, 1080 steps) in performance and efficiency.  Remarkably, ReTool-32B attains 72.5% accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further analysis reveals generalization to broader tool-use scenarios and emergent behaviors such as code self-correction, signaling an ''aha moment'' in which the model autonomously masters adaptive tool use. These findings highlight the promise of outcome-driven tool integration for advancing complex mathematical reasoning and offer new insights into hybrid neuro-symbolic systems.

Published as a conference paper ICLR 2026
RETOOL: REINFORCEMENT LEARNING FOR STRATEGIC TOOL USE IN LLMS
Jiazhan Feng': Shijue Huang Xingwei Qu', Ge Zhang', Chengquan Jiang', Jinxin Chi', Wanjun Zhong'!
Yujia Qin',
Baoquan Zhong',
'ByteDance Seed 'The Hong Kong University of Science Technology
(fengjiazhan, wan jun) (annnaayyddnneccoomm com joehsj31 00 11..... com
ABSTRACT
While reasoning models trained with reinforcement learning (RL) excel in reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning. concise computation, complex equation solving-areas where computational tools like code interpreters (CI) demonstrate distinct advantages. To bridge this gap, we propose ReToool, which enhances long-form reasoning with tool-integrated learning, including key features: (1) dynamic interleaving of real-time code execution within natural language reasoning processes, and (2) an automated RL paradigm that allows policy rollouts with multi-turn real-time code execution and teaches the model in learning when and how to invoke tools based on outcome feedback. ReTool employs a systematic training frame- work, beginning with synthetic code- augeeteed long-form reasoning data for cold-start training. Subsequent RL training leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns without human priors. Experiments on challenging MATH Olympiad benchmark AIME demonstrate Re ReTool's superiority: Our 32B model achieves 67% accuracy with 400 training steps, outperforming text-based RL baseline (40% accuracy, 1080 steps) in perfor- mance and efficiency. Remarkably, ReTooll-32B attains 72.5% accuracy in extended set- tings, surpassing OpenAl's ol -preview by 27.9%. Further analysis reveals generalization to broader tool-use scenarios and emergent behaviors such as code self-correction, signal- ing an "aha moment" in which the model autonomously masters adaptive tool use. These findings highlight the promise of outcome-drive driven tool integration for advancing complex mathematical reasoning and offer new insights into hybrid neuro-symbolic systems.
INTRODUCTION
Reinforcement learning (RL) has recently become popular paradigm for enhancing the reasoning capabil- ities of large language models (LLMs), enabling them to explore and refine long chains of thought (CoT)
(Wei et al. 2022 Yao al. 2023 Luong et al. 2024 Zhang al. 2024). Reasoning models such OpenAl ol (OpenAl et al. 2024) and DeepSeek RI (DeepSeek-Al Al et al.
2025 demonstrate strong performance in
pure text-based reasoning tasks by learning self-correct and engage more deliberate, analytical think- ing (Claude 2025 Team 2025dl a). These advances suggest early signs of metacognitive control, where models not only reason, but also monitor and revise their reasoning process.
"Equal contributions.
This work was done during an internship at Bytedance Seed.
I Corresponding author.

---

Published as a conference paper ICLR 2026
AIME2024
A1ME2025
50-
66.98
Cl-powered (I-(ReTool)
49.27
Cl-powered RL (ReTool)
Accuracy (%)
60
Aceuracy (%)
Text-bused
40-
36.67
40.00
40-
30-
800
200
609
800
1000
1200
Figure 1: AIME scores of ReTool and text-based RI. baseline based on the Qwen2.5-32B, Instrucct model.
Despite these advances, reasoning LLMs equipped with long chains of textual reasoning processes (Ouyang et al.) 2022) still show notable limitations in tasks that require precise numerical calculation symbolic manipulation, such as geometric reasoning, precise computation, or complex equation solving. In contrast, computational tools, such code interpreters (CI), can empower models with symbolic computation ca- pabilities that go far beyond pure text-based reasoning. Unlike textual CoT (Wei et al.] (023) methods that rely solely internal language patterns, code interpreters provide formal and executable interface for enumeration, verification, and precise computation. This not only enables exact numeric validation of intermediate steps- seps-ddrmatiicall reducing the ambiguity and compounding error often seen in textual rea-
soning (Chen et al., 2023: Wang et al. 2023b), but also allows models to expand their solution search space via programmable exploration.
Recent works have explored prompting and supervised fine-tuning methods (Chen et al. 2025b Pan et al. 2023 to equip L.MS with tool-use capabilities. However, these approaches are limited imitating the specifically-curate
data distribution, often failing to generalize beyond seen patterns or adaptively decide when and how to invoke external tools. As a result, models may misuse tools or fall back on brittle heuristics that are not robust across diverse problem settings. To overcome these limitations, RL offers a principled solution: it enables models to explore flexible reasoning trajectories and learn tool-use strategies guided by outcome-basee feedback. This paradigm not only incentivizes correct solutions, but also allows the model to discover nuanced behavioral patterns-sut such as how to recover from tool exccution mistakes via self-correction, decide when to effectively invoke tool execution during the long-chain reasoning process.
In this work, we embrace the RL paradigm and introduce ReTool, a Tool-augmented Reinforcement learning framework explicitly designed to guide LLMs towards optimal strategies for leveraging external computa- tional tools during reasoning. ReTool consists of two key components: First, we develop a data construction pipeline curate high-quality cold-start dataset that explicitly demonstrates when and how to invoke the code interpreter. This teaches the model an initial competency in tool usage and execution result analysis. Then, we apply tool-enhanced reinforcement learning train the model in discovering optimal tool manip- ulation reasoning strategy and adjusting its behavior through aitcomee-bssee rewards, going beyond what can be captured by supervised learning alone. During long-chain reasoning, the policy model rolls out by flexibly writing code blocks and achieving real-time execution results from a sandbox-style code interpreter to assist subsequent thinking.
We evaluate Tooo on the challenging MATH Olympiad benchmarks AIME2024 and AIME2025. Build- ing on Qwen2.5-32B - (Yang et al. 2024a), our model achieves 67.0% accuracy on AIME2024 with only 400 training steps, significantly outperforming the text-based RL baseline, which achieves 40.0% ac- curacy with 1080 training steps. These substantial gains highlight that explicitly modeling tool-use as part of the decision process not only pushes limits of model reasoning but also enhances training efficiency. Fur- thermore, when trained on -iiieee--RR--isiil------ strates further improvements, surpassing competitive baselines such QwQ-32B-Bm Prevriwww (Team] 2025d).
(DeepSeek-Al et al. (025), our model demon-
2

---

Published as a conference paper ICLR 2026
sl-32B (Muennighoff et al.J 2025), and OpenAI ol-preview (OpenAl 2024 This suggests that the RL training process inspires more efficient problem-solving strategies. We further conduct comprehensive analysis, including ablation study, extension to the web search domain, CI cognitive behavior through RL training. This analysis highlights several key findings: our model demonstrates enhanced code utilization capabilittes, enabling to employ more accurate and complex code snippets; It also learns to invoke tools appropriately, select tool adaptively, structure tool calls effectively, and iteratively refine reasoning through emergent code self-correction capabilities. Overall, our main contributions are summarized as follows:
We propose ReTool, novel reinforcement learning framework that integrates code interpreter execu- tion into the reasoning loop of LLMs. To equip the model with foundational capabilities for invoking the code interpreter. more, we design reinforcement learning framework that supports interlcaved code execution during rollout, enabling the model to iteratively explore, refine, and optimize its reasoning strategies through tool-augmented interactions guided by feedback from a sandboxed code interpreter.
curate high-quality cold-start dataset through developed pipeline. Further-
As shown in section 3.6, we conduct comprehensive empirical and behavioral analyses, and observe sev- eral key findings: (1) After RL training, the response length is reduced by approximately 40% compared to that prior to training, showcasing the potential reasoning token efficiency of tool-integrated reasoning; (2) Our approach can generalize broader tool-ust use scenarios like web search; (3) During RL training, the code ratio, code lines and correct code counts show increase trends, and the code invocation timing be- coming shifts earlier, indicating the improved code use capabilities and strategic tool usage development; (4) Emergent behaviors like code self-correction and adaptive tool selection can be observed during RI. phase, bringing more advanced tool-augmented reasoning patterns.
2 METHODOLOGY
In this section, we introduce ReTool. We begin with overview of the framework, followed by descrip- tion of the cold-start training process, including the data construction pipeline and supervised fine-tuning (section 2.2p. We then outline reinforcement learning pipeline, enhanced by code interpreter sandbox, to further enhance strategic tool usage development (section 2.3
2.1 OVERVIEW
Our approach consists of two stages: cold-start supervised fine-tuning followed by reinforcement learning with interleaved code execution rollout. Firstly, we collect data through our designed pipeline for cold- start supervised fine-tuning (SFT), which provides a robust initialization for the reinforcement learning (RL) phase. To enhance our model's 's tool utilization capabilities, we introduce specialized tool-using RL pipeline that enhances the model's ability to appropriately select and apply tools during the reasoning process.
2.2 COLD-START FOR TOOL-INTEGRATED REASONING FOUNDATION
We designed a pipeline to collect and curate high-quality data. Specifically, we begin by gathering ex- isting mathematical reasoning data from diverse sources, including open-source datasets such as Open- Thoughts (Team] 2025c). Subsequently, we implement dual-verification approach combining human ex- pert curation and Deepseek-RL (DeepSeek-Al et al. 2025) evaluation to filter invalid data. Through these steps, we collect a high-quality text-based reasoning dataset, denoted as Dinit-
Based on Pmit we further construct inde-inteerated reasoning data automatically. We first utilize a structured prompt template (detailed in Appendix Figure 99) for transformation, which modifies the original thinking process by replacing manual calculation steps that can benefit from code execution with the correspond- ing code snippets and their interpreter's execution results. Following this initial transformation, apply a
3

---

Published as conference paper ICLR 2026
Text:
Code:
Sandbox Feedback:
Final Result:
Question
Rollout with Interleaved Code Execution
V
Rollout
Policy LLM
Advantage
Policy
LLM
Reward
Advantage
Code Sandbox
Reward
(a) Text-based RL training
(b) ReTool RL training
Figure 2: Demonstration of text-based RL training process and ReTool's RL training process.
two-stage verification protocol. The first stage focuses on format verification, which improves readability and ensures consistent syntax that that enables the efficient detection of computational tool invocation trig- gers during subsequent reinforcement learning phases. The second stage entails answer verification, where we eliminate data samples whose final outputs do not align with the correct solutions to the mathematical problems. Finally, we collect a dataset Dei that consist of code-augmented long-form reasoning traces.
ReTool employs supervised fine-tuning learn when and how invoke the code interpreter from the afore- mentioned dataset Dei, thereby enhancing model's capability to appropriately utilize computational tools.
2.3
RETOOL: REINFORCHMENT LEARNING FOR STRATEGIC TooL USE
2.3.1
TRAINING ALGORITIM
We train ReTool based on PPO (Schulman et al. 2017). updates policy with the following objective:
021; CT) Touu (oe 0zt; CT)
[or
To(or q, 0.0ee CI) To (ot 7 Oce; CT)
(0) = Bigaat aa Glo)
min
Åe, clip
1, Ag
where Ty is policy model, No.. is reference model, To (o: O<e: CT) represents the rollouts with interleaved code execution and feedback from code interpreter. We modify PPO to better adopt tool integrated reasoning. During training, the policy LLM will collaborate with code sandbox to generate rollouts with multi-turn real-time code execution for solving given problems. We implement rule-based outcome reward to enable the model with the flexibility to autonomously explore and develop strategies for code usage awareness, code selection, timing of code invocation, and further diverse behaviors.
Reward Design To teach the model in learning when and how to invoke tools, implement rule-based accuracy reward to optimize the model. The accuracy reward evaluates response correctness. We require the model present final answers in a specified format (e.g., within (boxed()), enabling reliable rule-based verification. The reward is formulated as:
(1. 1-11 otherwise
is (a. a)
R(a, a)
(2)
where a and à represent the ground-truth answer and the predicted answer, respectively. We simplify the reward design aim to alleviate reward hacking and promote more diverse problem-solving behaviors based mere outcome feedback without considering code executability reward.
4

---

Published as a conference paper ICLR 2026
Rollout with Interleaved Code Execution To facilitate the integration of reasoning and executable code within the model, we propose a rollout approach that dynamically supports interleaved real-time code ex- ecution with natural language reasoning processes. As depicted in Figure 2 (b), our rollout process differs from the conventional approach, which typically generates only text-based reasoning (as shown in Figure 2 (a)). By contrast, our rollout approach integrates the collaboration of a policy LLM with an external code sandbox, enabling the production of hybrid content that combines text, code snippets, and real-time in- terpreter feedback. Concretely, we utilize a prompt template (Appendix Figure () to guide the model in interacting with the code sandbox by utilizing tags <code></ code> to explicitly mark the boundaries of generated codes. During the rollout process, policy model generate text-based reasoning t when a code termination trigger (</code>) is detected, the generation pause and the generated code ei is parsed and send to code sandbox environment for execution. Upon completion, the sandbox's 's output fi (successful results or еrror messages) filled within
tags and fed back the model, which continues generating the rollout until either providing a final answer o or producing new code snippet, ultimately producing hybrid reasoning trajectory it @ cj @ fi @ ..... o].
Notably, our approach returns both successful code execution results and interpreter error messages to the model. This dynamic feedback mechanism enables the model to iteratively explore, refine, and optimize its reasoning and tool usage strategies.
2.3.2 TRAINING DETAILS
Cold-start & RL For training, we employ the VeRL framework We adopt PPO as our RL method. We train model curated cold-start data for two epochs. Regarding hyperparameters, utilize the A optimizer with an initial learning rate of le-6. We define the expected maximum sequence length 512, and the KL coefficient set 0.0. We use Swen2.5-32B-Instruu (Qwen et al. 2025) as the main backbone. All experiments are conducted on NVIDIA H20 GPUs.
16384 tokens. For training, the mini-batch size is
feedback output from the loss computation. This sandbox- output masking approach blocks external tokens from in- terfering with loss calculations, ensuring training stability and preserving the model's inherently generated coherent reasoning sequences from disruption.
Interpreter Feedback Mask. We mask out the
KV-Cache Reuse. In order to reduce the memory cost during rollout, when each time the code termination trigger (</ code>) is detected, we will cache all the KV-cache before code execution and only calculate and append the KV-cache from the interpreter feedback (ciiteppreterx//inttpreet largely reduce the KV-cache for each rollout.
This will
Sandbox Construction. To accelerate the RL training process, we design asynchornous code sandbox environment. The sandbox pods function as workers in a pool, independently pulling tasks based on their current capacity, creating efficient load-balancing mechanism. This distributed asynchronous approach accelerates RL training by enabling parallel environment interactions across multiple threads, It prevents slower threads from creating bottlenecks and ensures optimal resource utilization, maintaining continuous throughput during the training process.
3 EXPERIMENT
3.1 EVALUATION SETUP
To ensure a stable evaluation, we repeat the evaluation set AIME2024 and AIME2025 32 times, GPQA (Diamond) (Rein et al.) 2023) times, MATH500 (Hendrycks et al., (022)) 4 times, GSMSK (Cobbe et al.,
2021 2 times, and report the overall average accuracy estimate pass@) @ The inference hyperparam-
5

---

Published as conference paper ICLR 2026
Table Main results on benchmarks in mathematic and STEM domains.
Model
AIME2024
AIME2025
GSM8K
MATH500
GPQA
Dwen2.5-Math-P5. 2aath-72BB-nnstuuu
30.0
95.9
85.9
Owen2.5-Math-77 Baat--72B--nstuut--II
40.0
95.8
88.1
Sky-TI
43.3
82.4
56.8
OpenAl op-preview
44.6
37.9
85.5
73.3
DeepSeek-RI. -eeesk----ZZe-oweeeeee
47.0
94.3
62.1
QWQ-ecB-Preview
50.0
33.5
90.6
54.5
sl-32B
56.7
93.0
59.6
ReTool D0wen2.5-32B-Inttmee
67.0
49.3
95.9
93.1
58.7
ReTool (DeepSeek-FF -1-eck-RR---iiiii D-----ttill-eeee Saeill--Qee--333
72.5
54.3
96.3
95.2
62.3
eters of evaluation lines, including -rrenh5--Mat---- 72B-Instruct (Yang et al. 2024b), Qwenn25-MMath772--Immrre
temperature 1.0 and top-p 0.7. We compare ReTool with competitive base-
(Yang
al. 2024b), Sky-TI Team 2025b), DeepSeek-RI-Zens Owen-32B 32B (DeepSeek-Al et al. 2025), QwQ
32B- (Team 2025d), s 31-32B (Muennighoff et al.] 2025). OpenAl ol -preview [ To
verify the effectiveness of our Re'Tool, we also compare the performance with RL without tool-using. i.e. Text-based RI. -rren..5-32BIInstuudd
And for the results of baselines, we report the avg@k by coping from corresponding literature source as pass@
3.2
MAIN RESULTS
As shown in Table Re'oool enables the LLM to flexibly leverage the code interpreter during the RL stage, leading to substantial performance improvements. Specifically, ReTool (0wen2.5-32B-Instrree achieves accuracies of 67.0% AIME2024, 49.3% AIME2025. 95.9% GSM8K, 93.1% MATH, and 58.7% on GPQA with only 400 training steps. This markedly outperforms most strong baselines with much larger parameter sizes, such Qwen2.5-Math-722. These findings indicate that the tool-integrated learning paradigm employed by Re'Tool enhances the model's reasoning capabilities significantly. Further- more, on AIME2024, ReTool (Qwen2.5-32B-Inse) Instrucct) surpasses the competitive baseline s1-32B by 10.3%. Similarly, on AIME2025, it achieves an 11 11.4% gain over OpenAl's ol op-preveew. When combined with a more advanced backbone, ReTool (DeepSeek- RI-DDstiill----- Sesill-Qwen-32BB further improves performance, achiev- ing scores of 72.5% on AIME2024 and 54.3% on AIME2025. These results suggest that more effective problem-solvingg strategies are discovered during the RL training process.
3.3 ABLATION STUDY
To further assess the effectiveness of ReToool, we conduct an ablation study and compare against several variants built on Owen2.5-322-Innstte Instrucct: (1) w/o RL: the cold-start model that still incorporates the code interpreter; (2) w/o Cl: a text-based RL method initialized with text-only cold-start SFT to ensure fairness; (3) w/o Training:
original base model. illustrated in Figure 3 (a), removing either the RL. stage (w/o RL) or the CI integration (w/o CD) leads to notable drop in performance
A1ME2024&20055, Further- more, our cold-start model achieves 40.9% accuracy on AIME2024, which is comparable the text-based RL baseline (40.0%) and significantly higher than the base model (26.7%). These findings demonstrate that our curated dataset successfully captures tool-usage patterns within executable reasoning traces, and that Cl-integrated training further enhances reasoning performance.
6

---

Published as conference paper ICLR 2026
AIME2024
AIME2025
GAIA
BrowseComp-ZH
Accuracy (%)
80
50
38.84
46.6
60
49.3
Accurucy
40
40
40.9
34,5
40
36.7
26.7
30
28.2
26.5
20
13
20
11.9
10
2.4
ReTool
Search-ol WebDuncer Retool-SFT ReJool-RL
Truining (a) Ablations on Owen2.5-32B-IInsile
(b) Results on Web Search Domain.
Figure 3: (a) Ablation study;(b) Results on the web search domain with same backbone on Qwen2.5-32B.
3.4 EXPANSION TO WEB SEARCH SCENARIO
To assess whether our strategy generalizes to broader tool-use scenarios, we further conducted experiments in the Web Search domain. Specifically, we applied ReTool with the Bing Search tool (following the MCP tool-use definition) on two widely used benchmarks: GAIA (Mialon et al. 2023) and BrowseComp- ZH (Zhou et al. (205)). As illustrated in Figure when using the same backbone (Qwen2.5-32B) Re'Tool consistently outperforms competitive baselines built on the same backbone, including WebDancer (Wu et al., 2025) and Search-ol (Li et al. 2025a). These findings highlight ReTool's generalizability and effectiveness beyond mathematical reasoning tasks, underscoring its adaptability across diverse tool-use environments.
3.5 "AHA MOMENT" OF CODE SELF-CORRECTION
Interestingly, our model exhibits an emergent ability to self-correct non-executable code, despite the absence of explicit training data for code self-correction. As shown in Figure 6 in Appendix, the model initially produced code that failed to execute due to the undefined function "greedy()". Upon receiving feedback from the interpreter, the model recognizee the error and responded with the reflection: "Oops, the functions need to be defined in the same scope. Let's correct that." " It then proceeded to generate revised, executable version of the code that included all necessary function definitions. This emergent behavior suggests that reinforcement learning can foster metacognitive capabilities, enabling the model to iteratively refine its generated code to address more complex problems.
3.6 COGNITIVE ANALYSIS
We present a comprehensive analysis, including the dynamics of code interpreter (CI)-related behaviors throughout the RL and the differences in code purpose before and after RL.
CI-related Behavior Evolution. To gain deeper insights into the RL process ReTool, we systemati- cally evaluated Cl-related metrics. Specifically, computed various metrics by analyzing model-generated outputs on the AIME2024 and AIME2025 datasets based on each saved checkpoint during RL training:
Response Length (Figure 4 (a)): We calculated the average response length and observed distinct trend: the response length initially declines sharply, later followed by a relatively gentle increase. We attribute the initial decline to the replacement of complex computational processes with more concise code, while the subsequent rise is likely due to the emergence of more diverse and complex code behaviors during RL training. Notably, the final average response length remains 40% shorter than that before RL. training (i.e., from 10k to 6k). This suggests that the Cl-powered reasoning approach potentially enhances efficiency of reasoning token utilization ratio by replacing intricate computational processes with code.
7

---

Published as conference paper ICLR 2026
(4) Nesponse Length
(b) Code Katlo
(0 Code Lines
1180
ADME2302
ARME33225
140
50-
AIME3125
ABME3024
184
280
128
360
289
24
Strpe
Steps
Strpe
(4) Froquency of Emerging Cade Self-Carrettonn
(e) Cade Pass Rate
(1) Code Invoration Timing
ADME22024
180
0.62
AIMEZI04
0.50
0.50
ABMER2124 Correc
AEMER2ES Correct Responscs
ARMER215 Inmervect Resparar
Steps
Stepo
Figure 4: Cl-related behavior evolution during RL training.
(b)): The ratio of responses that contain code are also calculated. Analysis reveals that throughout the RL training process, the average code ratios exhibit a total upward trend and end with covering nearly 98% percent of all questions. This suggests that the model's proficiency in code utilization improved progressively during the RL process, facilitating strategic tool usage development.
Code Ratio (Figure 4
Code Lines (Figure (c)): The lines of generated code reflects its complexity to some extent. Observa- tions show that the average code lines in responses exhibits consistent upward trend throughout training. By the end of RL training, the final average code lines is nearly fivefold higher than that before RL train- ing. This trend suggests that the model has learned more complex code strategies during the RL phase.
Frequency Emerging Code Self-Correction (Figure + (d)): We approximate the frequency code self-correction by detecting specific turning cues (e.g.. oops, wait, correcting) that appear between a failed code block and its immediate successor. As shown, the frequency of self-correction emerges early and peaks during the initial stages of training, then gradually decreases and stabilizes. This trend suggests that when the model's code generation capability still limited, it learns and leverages self-correct more frequently. As RL training, improves in producing correct code directly, thereby reducing the necessity for correction. This phenomenon reflects an emergent behavior: self-correction is prevalent at the outset but becomes less necessary the model gains competence.
4 Code Pass Rate (Figure (c)): The Cl-powered reasoning process involves generating intermediate code that may initially be incorrect, followed by iterative refinement based on interpreter feedback to produce executable code, so we report the average pass rate of last code in incorrect responses. Our analysis reveals that the code pass rate for correct responses remains consistently high, approaching 100%, while the code pass rate for incorrect responses exhibits a declining trend. This pattern suggests that code executability impacts the reasoning process and final result.
Code Invocation Timing (Figure (f)): We also calculate the code invocation timing, which is deter- mined by dividing the start position of code by the total length of the response. This metric reflects the timing of code invocation within the response. The results show that the code invocation timing advances during the RL training process, indicating that the model learns to determine the timing for tool usage.
Code Purpose Analysis. We also analysis the differences in code purposes before and after RL training, which reflects the types of code. We employ Claude4-Sonnet (Anthropic 2025) to classify the primary
8

---

Published as a conference paper ICLR 2026
purpose of code snippets based on their contextual information, then compute the frequency of code purposes that appear more than once, and the results are depicted in Figure 5 The word clouds reveal that calculation and verification are the dominant purposes of code in Cl-powered reasoning. After RL training, the code purposes in our model become more diverse, which demonstrates the metacognitive development of adaptive tool selection and enhances the generalizability of ReTool to broader range of problems.
4
BACKGROUND AND RELATED WORK
4.1
LLM REASONING
Recent advancements in large language models (LLMs) (Wei et al. 2022 Yao et al. 2023 Luong et al.
2024
OpenAl et al. 2024] Team 2025a DeepSeek-AL et al. [025] xAI [023] Claude [205 Team al.,
2023
Yang et al. 2024a) indicate significant progress toward cognitive abilities similar to human metacog-
nition through Ofhain-o-TThoulll (CoT) prompting, which enhances the reasoning capabilities of L.LMs by
leveraging step-by-step natural language reasoning [Wei et al. (2023); Wang et al. (2023a); WWng & Zhou 2024):Huang et al. (2025); Chen et al. (2025a). Building upon this foundation, recent research has shifted focus from train-time scaling to test-time scaling (Snell et al. 2024), where additional computational re- sources are allocated during inference to enable the generation of intermediate reasoning steps. Techniques such stepwise preference optimization (Lai al. 2024), Monte Carlo Tree Search (MCTS) (Xie al., 2024), and reinforcement learning (Luong et al. | [DeepSeek-A et al.) 2025) have been employed to im- prove multi-step and long-form mathematical reasoning. Complementing CoT, Program-of-Thougle (PoT) reasoning, introduced by Chen et al. (2023) and Gao et al. (2023), integrates external computational tools to simplify and validate complex reasoning steps, resulting enhanced accuracy.
4.2 TOOL INTEGRATED REASONING
Tool-integrated reasoning was first introduced to help LLMs solve computationally intensive mathematical problems with the integration of programming strategies (Chen et al. 2023 Yue al. 2023 Jin et al. [025]]
(Song et al.] 2025 Wang et al., 2024 Building on this line of work, Wang et al. (2023h) introduced an iterative framework that integrates textual reasoning with code execution, while Chen et al. (2025b) further advanced this direction by applying supervised fine-tuning on self-curated code-integrated CoT data. How- ever, this approach is inherently limited by its reliance on the specific data distribution, and cannot learn adaptive strategies for tool use -uch determining when and how to invoke tools through reinforcement learning (RL). A concurrent work (Li et al.] 2025b) applied RL to learn tool usage strategies on Qwen2.5- Math models Yang al., 2024b) at 1.5B and 7B scales, but the performance remained suboptimal. We further scale up this line of research and propose ReToool, a framework that leverages RL to strategically de- termine when and how to invoke the code interpreter. Our method outperforms Swen-Maah-7755
(Yang) et al. 2024b) and ol op-preveew (OpenAl 2024) significantly on AIME2024 and AIME2025. We also present
optinization
calculation
wint
calculation
probability calculation
geonstric calculation search
ver fin
ver firiiiation
processtng
calestation
(n) Refore R Retool RT. training.
(b) After Retool RI. training
Figure 5: Code purpose analysis.
9

---

Published as conference paper ICLR 2026
a comprehensive analysis of the learned tool-use behaviors and highlight several key findings regarding the model's cognitive patterns in code invocation after ReToool training.
5 CONCLUSION
this paper, we propose ReTool, a novel reinforcement learning framework that empowers LLMs to self- enhance their mathematical reasoning capabilities through effective Code Interpreter utiliizaiinn. Our prehensive experiments on various benchmarks demonstrate that ReTool not only achieves superior accuracy compared conventional text-based RL approaches, but also converges with significantly fewer training steps. Further analyses also demonstrate generalization to broader tool-use scenarios and emergent behav- iors such code self-correction. Through careful data curation and our specialized tool-using pipeline, ReTool enables models to develop sophisticated computationa intervention strategies, paving the way for more efficient and powerful tool-augmented reasoning in L.Mss.
ETHICS STATEMENT
This work does not involve human subjects, personal data, sensitive information. All experiments conducted on publicly available datasets in mathematics and STEM domains (e.g.. AIME, GSM8K, MATH, GPQA). The datasets were used strictly for research purposes and do not contain identifiable private infor- mation. Our methods focus on improving the reasoning capabilities of LLMS by integrating reinforcement learning with external computational tools. We affirm compliance with the ICLR Code of Ethics and confirm that no part of this research raises concerns regarding privacy, discrimination, or conflict of interest.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our work. The full methodological details of ReTool, including cold-start data construction, supervised fine-tuning, reinforcement learning setup, reward design, and evaluation protocols, are described in Sections 2 and 3 of the main text. Comprehensive ex- perimental results across multiple datasets and ablation studies reported in Tables 2 and Figures 3 Additional training details such as optimizer, learning rate, batch size, sequence length, masking strategy. and sandbox design are provided in Section 2.3.2 These details should enable independent researchers replicate our experiments. And the training data and the evaluation scripts are available in the supplementary material.
ACKNOWLEDGMENTS
We would like to thank Guang Shi, Mingxuan Wang, Renjie Zheng, Chen Dun, and Yun Jiang for their support on this work.
REFERENCES
Anthropic. Claude sonnet http:////////...aaaaoaooooooo//a
2025.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long thhinhtf- for reason- ing large language models, 2025a. URL URLt:stss://axxt wwwpaawwww....raaaaaaoaa orggabs//5033.... 09567
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for numerical reasoning tasks, 2023. URL https: //atps. org/ hts//bs/221111 /
12588
10

---

Published as a conference paper at ICLR 2026
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. An empirical study on eliciting and improving r1 rl-like reasoning models. arXiv preprint arXiv:2503.04544 04444444 2025b.
Claude. Claude 3.7 sonnet.
2025,
URL
https: wttp:://wwww.annrrpc.ccoomm /
-laude-3-7-Sonee
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 2021. URL BRLLattss://axiii wrg. hb///bbs//// /bs//110.. 2110.141661111..1111111111111111 14168
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li. Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao. Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo. Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang. Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong. Ying He, Yishi Piao. Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu. Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https: www.xxiw.....aa... hbsp//bs//5550 12948
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. Pal: Program-aided language models, 2023.
wwtp///i.orroooooooo... abs abs/221..
10435
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https:
//arxiv.org/ablt 21 13.00..330 03874
Shijue Huang, Wanjun Zhong, Deng Cai, Fanqi Wan, Chengyi Wang, Mingxuan Wang, Mu Qiao, and Ruifeng Xu. Empowering self-learning of lms: Inner knowledge explicitation catalyst. Proceedings of the AAAI Conference on Artificial Intelligence, 3901eee410))))32112211111133 24158, Apr. 2025. doi: 10. a609 v39123.34590. URL //op:::://js..o ww...aaaaaa.......o...aao....... ot..:///............/.ooo.....aa http::///..eeeeeeaaaaaaaaaaa...t

---

Published as a conference paper ICLR 2026
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. URL http:s/t://// org/ abs 2209/ 12186
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training umms to reason and leverage search engines with reinforcement learning, 2025.
URL / www.aviii.v...... http:///wwwww..oo//.////////////
09516
Xin Lai, Zhuotao Tian, Yukang Chen, Sengiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of 1lms, 2024. URL htttp:://axxv..orrt abs/
2406. 18629
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang. Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models, 2025a. URL https:/ /arxiv. org/
05366
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl, 2025b. URL https://
arxiv.org/ abs/25533/222// 2533..3333333333 23383
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning 08967
with reinforced fine-tuning, 2024.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023.
12983
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettle- moyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. peattps://arxe org/ /twp/b//////aaa///////////aaaaea abs/ 2501. 19393
OpenAl. Learning to reason with Ilms, September 2024. URL https://openai.com com.einai.com/idee.. Learning-to to- reason Beeon-wie with 1 1lms/
OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McK- inzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger. Christopher Hesse, Claudia Fis- cher, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica. Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang. Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas. Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo. Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao. Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pa- chocki, James Lennon, Jason Wei, Jcan Harb, Jerry Twore, Jiacheng Feng. Jiahui Yu, Jiayi Weng. Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hall- man, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach,
12

---

Published as a conference paper at ICLR 2026
Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe. Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Paviov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez. Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay. Steph Lin. Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda. Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng. Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peter- son, Tyna Floundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system http:://wwxvv..or.aaaa.o hbtp://b//22243 16720
card, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: pattps://arxiv.cem /ttpxv.ax.vio////..............a abs 2203/223..00 02155
Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3806- 3824, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023. findings-emmlp.248.. URL
org/ findings ceatings-emnllp. 248/
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin. Jian Yang. Jianhong Tu. Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https::/ org/ /bs/ 2412.15111 15115
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
htthst::: //apss:////xvvvv/ aawav.ivivv.//.aa http://wwwvv....a/aaa/aa htseg//bs/23111 12022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms, 2017.
/00//100...... 06347
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024.
org/ abs/2408.
03314
13

---

Published as a conference paper ICLR 2026
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning, 2025.
URL
ab5/2533.3250/0 05592
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:23/2.11800 2023.
Kimi Team. Kimi k1.5: Scaling reinforcement learning with 1lms, 2025a. URL https::/arxiv.com abs/2501 12599
NovaSky Team. Sky-tl: Train your ol preview model within $450. 2025b.
URL https:
7 7/novasky ai wtw.ggitub.i..oooo wtt.uu.b..//post//kkyyyttt
OpenThoughts Team. Open Thoughts. https://open-thoughtss.t ai, January 2025c.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025d.
URL https:
7/qwenim. haw.imm..itttte wwt.hht..........oooooo.ootottte qwq- 32b/
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. To- prompting: An empirical study of what matters. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pp. 2717-2733 Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/w1/2023. 1ag.66//////223...aoonngg... 153. URL https:
wards understanding
ort. 2023. awl-lclonnno 153/
Hongru Wang. Yujia Qin, Yankai Lin, Jeff Z. Pan, and Kam-Fai Wong. Empowering large language models: Tool learning for real-world interaction. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pp. 2983-299929999 2986, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314, doi: 10. 1011145/62277723661331 URL
3661381
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathemat- ical reasoning, 2023b. URL URL htRl/htts::// pettps://axxiv.orre org/ abs /
Xuezhi Wang and Denny Zhou. th-tugt reasoning without prompting, 2024.
URL https:
//axxiv.org abs/ / 2402. 10200
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language mod- els. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 24824 24837. Curran Asso- ciates, Inc., 2022.
http://ennnnsiiis...ceeeeeeeeeee neurips......... wwwpppeepp...eeeeeeeeeeeeeeeeeee files/ httpss//pppppp22////
aboa4 Paper- Conference. .df
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL
http:s//erxxxv.orr////////... 11903
Jialong Wu, Baixuan Li, Runnan Fang. Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Gang Fu, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou, Webdancer: Towards autonomous information seeking agency, 2025. URL UtLhttps: wttp:httt//sxiiiiii org abs ab55 22648
14

---

Published as a conference paper ICLR 2026
XAI. Grok. https://x. ai/ 2023. URL URL https://x.x ai/ Large language model.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning, 2024. URL
https::/aaxiv..o org/ 2200s/24400 2405.00451
An Yang, Baosong Yang, Beichen Zhang, Bimyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan. Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024b. https:///aa
org/ abs 2409 12122
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I.. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: deliberate problem solving with large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, 2023. URL https:
org 2ea.b/223339 230.//55533333
Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yinghao Ma, Yizhi Li, Ziyang Ma, Bill Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kaijing Ma, Minghao Liu, Morry Niu, Noah Wang, Quehry Que, Ruibo Liu, Sine Liu, Shawn Guo, Soren Gao, Wangchunshu Zhou, Xinyue Zhang, Yizhi Zhou, Yubo Wang. Yuelin Bai, Yuhan Zhang, Yuxiang Zhang, Zenith Wang, Zhenzhu Yang, Zijian Zhao, Jiajun Zhang, Wanli Ouyang, Wenhao Huang, and Wenhu Chen. Map-neo: Highly capable and transparent bilingual large language model series, 2024. URL https: /aatps://arxiv. 19327
Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese, 2025. URL
fttps://arxiv..or org/ abs 2504. 19314
15

---

Published as conference paper ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)
In preparing this paper, we employed large language models (LLMs) solely as writing assistants. Their role was limited to polishing grammar and improving fluency. They were not used for research ideation, methodological development, data analysis, or result interpretation. All scientific contributions, experimen- tal designs, and findings presented in this work entirely the result of the authors' own efforts.
A.2 RESULTS ON OTHER BASE MODEL
To further assess the effectiveness of Re ReTool, we conduct experiments using code-oriented backbone, 1rven255Cooder-7--Bee
(Hui et al. 2024). Following the same evaluation protocol as the main re- AIME2024, AIME2025, GSM8K, and MATH500. The
sults, we report the overall average accuracy results are presented T We observe that applying our ReTool framework through both supervised fine-tuning (SFT) and reinforcement learning (RL) consistently yields substantial improvements across all benchmarks when built on top of Qwen2.5-Coder- Co-ee--B--Instrueee Instruct... These findings highlight the generalizability and effectiveness of ReTool across diverse base models.
Table 2: Results on different base model.
Model
AIME2024
AIME2025
GSMSK
MATH500
10.0
86.7
66.8
+ ReTooll-SFT
14.69
17.29
90.67
77.2
Roool-RR
46.04
32.60
92.08
84.55
A.3 THE PRACTICAL HELPFULNESS OF THE GENERATED CODE TOWARD THE FINAL SOLUTION.
To more rigorously evaluate the practical usefulness of generated code for solving the target tasks, we con- ducted focused analysis of code snippets produced by our models before and after RL training. Specifically, we extracted all code snippets along with their corresponding contexts from the AIME2024 results and asked Claude4-Sone -onnnetS (Anthropic, 2025) assess whether each snippet contributed meaningfully to solving the overall problem. The analysis reveals that, prior to RL training, 406 out of 471 code snippets (86.20%) were judged to be helpful, while 65 (13.80% were not. After RL fine-tuning, the proportion of helpful code increased: 4,468 out of 5,010 snippets 18%) were rated as helpful, compared with 542 (10.82%) deemed unhelpful. These findings indicate not only a consistently high proportion of practically useful code but also a further improvement in code helpfulness following RL training.
A.4 CI-POWERED REASONING VS. TEXT-BASED REASONING.
We present a case study to illustrate the distinction between Cl-powered reasoning after reinforcement learn- ing (RL) training and conventional text-based reasoning prior to RL training, as illustrated in Figure [n the Appendix. When faced with the same question, text-based reasoning relies on a "laborious" text-only cal- culation process, which is prone numerical errors and often results in incorrect inference outcomes. In contrast, Cl Cl-powered reasoning substitutes this complex calculation process with concise code. This ap- proach not only ensures computational accuracy through the assistance of an external code interpreter but also enables the model to focus more effectively on holistic reasoning strategies.
16

---

Published as conference paper ICLR 2026
From an unlimited supply of I-cent coins, 10-cent coins, and 25.cent coins, ... Find the mumber of values of SNS between $10005 inclusive which greedy algorithm succeeds.
Question:
Response: Alright, let's tackle this problem step by using code ensure accumcy. The goal is determine for each value of (N () from to 1000 whether the greedy alguritim produces the minimum number of coins. If does, we count that V N y....
Scode>
pyytonn
def /esttttsssssiieeteeeeeee sis___pcciic____ NiN_NNNNNllll valuusle:s):
for N in N_values:
greedy_resud result greedy(N)
test test_resuie results test__seecffeeeeeeeeeeeeeeeeeee Dest_specffic_NNN__.O WilNivalle_tto_tee printitest_resuffe
<leode>
Tiacetaetee-rracctcck Traceback (mest recent call last):
line 11, in smodule> test, reaults - test Leco__spccii___ee__ N[N__N(NN_a______ yaluus to to_tes)) File htmp/mppr3ujj33 hempprrjj7////mpppp /ttpm////mmmmmmm3333/3 9k6o.py", line 5, in test, testttesseeeeee specific N greedy result = greedy(N) NameError: name 'greedy" not defined <ditterpeter>>
File htmp/mpprrsjj bujpmmmppp/////tttt httf/////mmmmm33kkooo
Oops, functions need defined the scope. Let's correct that:
<code>
python
def greedy(N):
denominations = [25, 10, 1]
deftest ces___secifi__NOW NNN_vvlues)- for in N_values greedy_reste result = greedy(N)
results test__sseee____eeeeeeeeee___eeee Spwcific__/N___nnn___ Nic__NNNNValee___aeeeeeaaeee _t_osttt___o_ooooo
printitest_resune results)
<lode>>
interppeter (42: (9. 6), (1, D), 10: 1). 1), 1000: (40. 40) <fnterpretee
Kanswer> (610) <lanswer>
Figure 6: The case of "aha moment" about code self-correction.
17

---

Published as conference paper ICLR 2026
Questtion: Let SNS be the geealest Tour digit positive inleger with propecty that is henev er ane of digih is changed to $15, the resulting sumber is divisible $78. Let SQS and SRS be the quodient and gemainder, respectiely, when SNS divited by $1000s. Find $Q+BS.
Text-hased Reesoning Before RL: Okay so I need to find the greatest four- resalling mumber divosible by 7. Then, I have to find Q and when divided by 1000, and then compote Q gltight,
Cl Cl-powered Reesoning After RL: liihht, let's tackle this pmoblmm stop by step. The goal find greatest four digit pos integer N changing any one of its dogits to meults mimber divisible 7. Then, Q bf ame quotient remaindor when V. N is divided by 1000...
digit
any one digits replaced
stop by step
Themfoo, each of these oq equdioons can be waraagge to:
1.N (1-A 1000 mnd7
compute values V. kmod 0. 1.00 lmoo 0. M. imad
2.N --- (1- 100 mod7
71. and II lmod 7 V
(1- 1)*1U
=.N 1 mod?
tython
The Sour expressions be equal mod7, Thersfoee, can tham equal to each other -(1 Ay- = (1- B)" 10100 mod
Compute med. 1000 1000 % med
values
Watt, let's check B, C, for A-9
mod. mod_10 10%
B+3A -2 modd=45P -2277 -2425 mod7=25 2792-25:214
med.
C=2A -1 mod7=2*9-114 1-17 mod7=17 772=117-141
D D46A -5 mod7=669 5549 mod7449 477-49949-0
print[[" mod 7 (mod_17)
So, C-,,C-3 Therefore, N-9430...
Let me compute each lhse modulo7:
<loode>
1.0000 mod7, so
100 10*C D.00 mod7
<interpreter-111 1000 mod 100 mod. 7 10 mod. 7 mod
But 100*B=2*5 mod7.
1<Sutesppeter>>
10*C=3*C =3
6 6-22 +3C-D ml mod7
We need to solve these equations for digits it h d where a from to and V. b, c, d are from Let's use code to Iterate through possible values and find the valid M V).
2. 2.1000*A =6A mod7, 100=2 mod7,
3. 1000P +2B +3 mod7 (since 100*B=2B, 1m3, 10"Cm3C)
ppython
Wait, no. 1etmm recompate each equaton properl.
In]:
Equationl: IBCD -0 mod?
for a range(9, 0,-1]
-100% 10"C
10:3,
Lel me tabulale [rum] wi:
return
BB-6-A mod7 (C-6-5A
DD-2A mod7 /aald?
1 1|5-4-1 mod?-] (6-5-1 mod7-1 [2-1-1 [Yes...
N find valid nn)
""Result*:
N//1000 R N 8 1000
-I 36241)
Diriding 1000 gives =5 and Q R 5 4 624 629
result::
print([NN (N), (O), R #R8 Q R (result)))
I| (hoxed (629)
<hodes
<intespreter-N 3694, 5. 694, + R 699 <fintrpreter>
Thus, the final answer is confirmed to be 699.
<Hisi er> (63)) <lanswer>
Figure 7: Case of Cl-powered reasoning vs. Text-based reasoning.
18

---

Published as a conference paper ICLR 2026
Template prompt for Reinforcement L Learning Rollout
Solve the following problem step by step. You now have the ability to selectively write ex ecutable thon code to enhance your reasoning process. The Pytbon code will be exocuted by an external sandbox, and the output (wrapped in aalleppeeecooupuu/tttteereeee reasoning and help you arive at the final answer. The Python code should be complete scripts, including necessary imports.
can be retumed to aid your
Code Format:
code snippet wrapped with <code>>
py thon
code snippet
</oode>
A nswer Format:
The last of your response should be: <answer>iboredd" be final answer goes here. ///nswwer>>
User uestion: (question)
ssistant:
Figure 8: Template prompt for ReTool rollout.
19

---

Published as a conference paper ICLR 2026
Template Prompt for Data Curation
ou are helpful Al assistant Initially, w when solving ques lio, you would need to think step by step, without the abiliyy to use code for calculation, Now, you have the capahility to write code to use the code interpreter for calculation. The code will be ex ecutedd by sandbox, and the result can be returned to enhamce your reasoning process. Y ou can now leverage code to enhance your calculation while still maintaining the reasoning process.
The thinking process can have multiple code snippets. Each code snippet is wrapped with:
<code>
py thonon
code snlppet
<code>, and should be execulable. The relumed result is w wrapped with <interpoeeer> execution resulls (lexll) [exxtt((<inteerpeeeee)..
Goal:
Modify the original thinking process to make it more accurate by replacing manual calculation steps that can benefit from code execution with the corresponding code snippets and their interpreter's execution results. The core reasoning logic from the original thinking process, including any unsuccessful attempts, should remain unchanged. Y ou should only replace the necessary manual calculation steps with code and interpreter's execution results, without altering the rest tokens of the thinking process. Wrap the revised thinking process within <revised thinking prcciingproeesseeeee and </revised ttinkingg_rocess>),
User uestion:
{question) }
Original Thinking Process (without code interpreter's support):
<onginal thinkinggnponte proclig__pocsseee (original response) </original thinking_processs process>
Details:
Ideelity sections here code execulion could speed up resoning process make calculation more accurate. Replace manual calculation steps with code snippels corresponding interpreter's execution results. K Keep the logical flow the reasoning process intact, including any failed exploration attempts were part of ini tial proces.
The code snippets should complete scripts, including necessary imports, and should not contain markdown symbols like
<code>
python
code snippet
</oode>.
O uutput in the code snippets must ex expliiilly call the print function.
Execution results should match the model's outgul exactly, with no extra or missing tokens.
7. If the Original Thinking P Proess does not include an <answer> section at the end, please add it in the Revised T hinking Process:
<answeet er> (boxed (" The final answer goes here.") " </answers
Revised Thinking Pr Process (W ith code suppprr)
Figure 9: Template prompt for data curation.
20


Reviews:
[{"review_id": "14NILeDVl6", "reviewer": "Reviewer_xeUG", "summary": "The paper proposes to enable a Python code execution tool during reasoning in LLMs by RL training with the final solution correctness posing as the reward. Tool calls are delimited via special markers and the rollout function hands over to a Python interpreter when it encounters them and the execution result is inserted into the context again with special delimiters, after which it continues with generating tokens.", "questions": "Why use PPO over GRPO? How was the value model parameterized?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper gets decent results compared to the baselines and the ablations seem to show that the LLM can use tools in a way that raises its success chance\n- The statistics on how often responses use the Python code execution tool and how long or correct the code is, is interesting, albeit calling it cognitive analysis seems a bit much", "weaknesses": "The primary contribution is an RL framework that integrates reasoning and tool use, by adding delimiter tokens to the tool use part and having a parser check whether it should hand over to a Python interpreter before continuing with the token generation. The reward is the final correctness, which is the same is in the standard RL reasoning paradigm. This is a straightforward approach that has been considered by many people and to distinguish this paper from a \"flag-planting\" paper, I'd suggest adding more analysis to improve our understanding on how far this method can go and where the limitations lie. For example:\n- the experiments are currently limited to Qwen2.5 32B models and it's unclear from the paper results whether it will work for other models that might have seen less reasoning like data during pretraining or differ in other aspects\n- the analysis in 3.6 shows the behavior of the ReTool model but doesn't show how general that behavior is, by comparing to how the reasoning RL model without tool use behaves or whether the behavior is consistent across different models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ne4wgiVLnO", "reviewer": "Reviewer_MF9u", "summary": "This paper introduces ReTool, a reinforcement learning (RL) framework designed to teach large language models (LLMs) how to strategically use a code interpreter (CI) to improve their reasoning capabilities. The method involves a two-stage process: a \"cold-start\" supervised fine-tuning on curated code-augmented reasoning data, followed by RL training with PPO to optimize tool-use strategies based on task outcome rewards. The authors demonstrate that ReTool significantly improves performance on challenging mathematical reasoning benchmarks like AIME, outperforming strong baselines and showing impressive training efficiency, while also fostering emergent behaviors like code self-correction.", "questions": "1. Could you provide more details about the cold-start dataset, $D_{CI}$? Specifically, what was its final size after the two-stage verification protocol, and what were the statistics of tool usage (e.g., average number of tool calls per sample)?\n2. The binary reward function ($+1/-1$) is remarkably simple yet effective. Did you experiment with more shaped reward functions, such as providing intermediate rewards for successful code execution or penalties for syntax errors? If so, how did they compare? If not, could you elaborate on your hypothesis for why the simple outcome-based reward is sufficient for learning complex behaviors like self-correction?\n3. In the PPO training details (Section 2.3.2), you mention setting the KL coefficient to 0.0. This effectively removes the trust region constraint that differentiates PPO from vanilla policy gradient methods. Could you explain the rationale for this choice and whether you observed any training instability as a result?\n4.  I am curious about the learning dynamics from an exploration-exploitation perspective. Could you plot and discuss the evolution of the policy's entropy during the RL training phase? An analysis of this trend would be insightful; for example, does the entropy decrease monotonically as the model becomes more confident in its tool-use strategy, or are there distinct phases where it might change as the model discovers new, more complex patterns?\n5.  The cold-start SFT phase is used to ensure the model adheres to the specified tool-use protocol (e.g., using `<code>` tags). With the advent of more powerful base models that exhibit strong in-context learning abilities (e.g., Qwen3-8B), do you think this SFT step is still essential? Could a sufficiently capable model learn the required format and syntax directly from the prompt during the RL phase, potentially simplifying the overall training pipeline?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The proposed two-stage approach, combining supervised learning for foundational skills with reinforcement learning for strategic optimization, is logical, well-motivated, and shown to be highly effective.\n2. ReTool achieves state-of-the-art performance on the challenging AIME benchmarks, substantially outperforming both its own 32B backbone and other strong, often larger, models. The reported efficiency (e.g., achieving high scores with only 400 training steps) is particularly impressive and highlights the effectiveness of the tool-integrated RL paradigm.\n3. The \"cognitive analysis\" in Section 3.6, which tracks metrics like response length, code ratio, code complexity, and invocation timing, offers valuable insights into *how* the model learns to use tools more effectively. The identification of emergent behaviors like code self-correction (the \"aha moment\") is a compelling finding that deepens our understanding of RL's impact on LLM reasoning processes.\n4. The motivation is well-established, the methodology is described in sufficient detail for understanding, and the figures (especially Figure 2 and Figure 4) are effective at illustrating the core concepts and findings. The ablation studies convincingly demonstrate the importance of both the CI integration and the RL training stage.", "weaknesses": "1. The performance gain from the RL stage is substantial (e.g., from 40.9% to 67.0% on AIME2024 in the ablation). Based on your cognitive analysis, could you provide more intuition on what you believe is the most critical strategic capability the model learns during RL that SFT on curated data fails to instill? Is it primarily about *when* to invoke the tool, or does it also learn more complex policies like using the tool for iterative verification or hypothesis testing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4P1lh4Kjq", "reviewer": "Reviewer_Bh1w", "summary": "This paper proposes a novel framework for integrating executable tools, specifically a code interpreter, into the reinforcement learning (RL) reasoning loop. The authors introduce a mixed trajectory approach, where textual inputs are converted into executable code that interacts with an interpreter. The key steps in the process are as follows:\n1. The authors first construct training data by combining text, code, and code execution results, using a template-based approach. These trajectories are used for supervised fine-tuning (SFT), enabling the model to start with a reasonable understanding of both generating code and its subsequent execution. This cold start phase is crucial for overcoming the initial knowledge gap that would otherwise hinder RL fine-tuning.\n2. After the SFT phase, the model is further refined using Proximal Policy Optimization (PPO) with a sparse binary reward signal, based on whether the final generated code is correct. This feedback loop integrates code execution into the generation process, where the model learns when and how to invoke the interpreter (i.e., when to call external tools). The execution step is embedded within the model's reasoning process, allowing it to improve its performance over time by adjusting its tool usage.\n\nThe authors evaluate their method on several benchmarks, including AIME2024/2025, GSM8K, MATH500, and GPQA, where the method demonstrates significant improvements compared to existing models, including those that do not use RL. Notably, the proposed method, with a Qwen2.5-32B backbone, achieves AIME2024=67.0% and AIME2025=49.3% in just 400 steps of training, outperforming larger models. Further improvements are seen when using a distilled model variant, DeepSeek-R1-Distill-Qwen-32B, which reaches 72.5%/54.3%", "questions": "1. Please report pass@k\n2. It is well known that RL results are often sensitive to seeds and minor config changes. Please provide >= 5 independent runs per key setting and report mean ± std\n3. The reported accuracy on AIME-2025 is noticeably lower than on AIME-2024. Could you clarify the causes?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It provides a practical, reproducible pipeline that many groups could adopt.\n2. Competitive results on challenging benchmarks\n3. The paper spells out the execution protocol, loss masking, async sandboxing, and caching—practical details that substantially reduce adoption friction.", "weaknesses": "1. It has limited algorithmic novelty. The optimization relies on standard PPO.\n2. Impact of RL on reasoning upper-bound (pass@k) is missing. I strongly encourage the authors to provide pass@k (k=32,64,128,256,1024) results in the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AFcSeFKnit", "reviewer": "Reviewer_yanT", "summary": "The paper explores RL-based strategies for tool use in LLMs. I'll admit that this is quite outside of my expertise so my assessment is mostly an educated guess. Tool use is obviously well-explored though the specific implementation of incorporating RL-based reward signals into tool use pipelines is the main contribution here. There is concurrent work (which the authors cite) though the authors claim to \"scale this up\" and that the previous approach doesn't work. This comparison to perhaps the most relevant related work is a bit vague, making it harder to assess the technical contribution.", "questions": "Can you clarify the main differences compared to the concurrent work you cite? See additional points among weaknesses above.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-- Results shown are fairly promising\n\n-- Contribution seems to have some novel elements (even if the related work section is quite limited, and comparison against concurrent work is confusing)", "weaknesses": "-- The evaluation is mostly \"internal\", i.e., comparing the model to different versions of itself, etc. There's some evaluation against some standard baselines, though it's hard for me to assess these, i.e., do they just show that the use of tools outperforms general purpose models (which seems known?). It's hard to know where the strong and really comparable baselines are.\n\n-- Experiments overall a bit thin by ICLR standards.\n\n-- Very thin related work section. The most relevant related papers seem to be swept under the rug a bit and described only vaguely.\n\n-- Parts of the evaluation seem cherry-picked or rely on anecdotal examples.\n\n-- Method itself is very simple, not exactly a criticism itself, but as a reader who doesn't know the topic it's hard to follow exactly what the technical contribution is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)