Agentic RL Scaling Law: Spontaneous Code Execution for Mathematical Problem Solving
Authors: Xinji Mai
Haotian Xu
Xing W
Weinong Wang
Yingying Zhang
Wenqiang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Agentic RL
Code Execution
LLMs
Math
Primary area: deep_learning
TL;DR: This paper presents ZeroTIR, revealing agent‑level RL scaling laws that tie training steps, code‑call frequency, response length, and accuracy, and surpassing ZeroRL and SFT baselines on challenging math benchmarks.

Abstract: Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards  enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial. We investigate RL from outcome-based rewards for Tool-Integrated Reasoning, ZeroTIR, training base LLMs to spontaneously generate and execute Python code for mathematical problems without supervised tool-use examples. Our central contribution is we demonstrate that as RL training progresses, key metrics scale predictably. Specifically, we observe strong positive correlations where increased training steps lead to increases in the spontaneous code execution frequency, the average response length, and, critically, the final task accuracy. This suggests a quantifiable relationship between computational effort invested in training and the emergence of effective, tool-augmented reasoning strategies. We implement a robust framework featuring a decoupled code execution environment and validate our findings across standard RL algorithms and frameworks. Experiments show ZeroTIR significantly surpasses non-tool ZeroRL baselines on challenging math benchmarks. Our findings provide a foundational understanding of how autonomous tool use is acquired and scales within Agent RL, offering a reproducible benchmark for future studies. Code is released at \href{https://github.com/yyht/openrlhf_async_pipline}{https://github.com/yyht/openrlhf\_async\_pipline}.

Agentic RL Scaling Law: Spontaneous Code Execution for Mathematical Problem Solving
Xinji Mai 12,
Haotian Xu Xing W
Weinong Wang?
Yingying Zhang Wenqiang Zhang 1.4,*
Cooleege of Intelligent Robotics and Advanced Manufacturing, Fudan University
2Xiaohongshu
East China Normal University
4 "Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University xjmai230m. fudan. edu. cn, (xuhaotian, wuxing, wwngweinong /0xi aohongshu. com
Abstract
Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial. We investigate RL from outcome-bauee -based rewards for Tool- pnto-Inneerated Rea- soning, ZeroTIR, training base LLMs to spontaneously generate and execute Python code for mathematical problems without supervised tool-use examples. Our central contribution is we demonstrate that as RL. training progresses, key metrics scale pre- dictably. Specifically, we observe strong positive correlations where increased train- ing steps lead to increases in the spontaneous code execution frequency, the average response length, and, critically, the final task accuracy. This suggests quantifiable relationship between computational effort invested in training and the emergence of effective, tool-augmented reasoning strategies. We implement a robust framework featuring a decoupled code execution environment and validate our findings across standard RL algorithms and frameworks. Experiments show ZeroTIR significantly surpasses non-tool ZeroRL baselines challenging math benchmarks. Our find- ings provide a foundational understanding of how autonomous tool use is acquired and scales within Agent RL, offering reproducible benchmark for future studies. Code is released at hhtts://gthub.cm//////oooopp async_ _pppline
1 Introduction
LLMSs have demonstrated remarkable capabilities across various domains. However, they often face challenges when tackling tasks that demand precise, multi-step reasoning and complex computations, particularly within the realm of mathemmatccs[12] The inherent nature of next-token prediction often leads LLMs to generate responses based on high likelihood rather than computational correctness, hindering their reliability for mathematical problem-solving. Existing approaches to augment the mathematical abilities of LLMs typically involve Supervised Fine- Fine-Tuning (SFT) on specific datasets integrating external tools in controlled manner[3]. SFT, while potentially effective, often necessitates extensive, high-quality trajectory data and may constrain the model's ' capacity to explore novel problem-solving strategies, potentially leading overfitting specific solution patterns and sacrificing generalizability. Tnte--ntege Reasoning (TIR) methods, such as the TIR capability usually depend specific prompt structures or
mentioned in the context of Qwen2.5-Man Math 4.
Equal contribution. Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Code DageeFrg
AIME 2025 Acarasy
Asarge Respener Lengls
ZeroTIR Training Dynamics
1.0
1200
1190
0.8
1000
Code Usage Freg
0.6
900
AIME 2025
6.4
700
600
0.2
500
0.00 100 200
300
400
500
400
Trinnin Steps
Figure Illustration of Agent RL Scaling Law during Zero-TIR training. The observed trends, including the non-monotonic scaling of code usage and the positive correlation between training steps, code usage, response length, and task accuracy, exemplify the core scaling dynamics.
predefined triggers to invoke tools like code interpreters. This paradigm differs fundamentally from fostering an agent that learns to utilize tools spontaneously based on the emergent needs of the task.
The advent of models like DeepSeek-RI [5] has highlighted the significant success of RL in scaling LLM reasoning capabilities directly from base models, which is also known as ZeroRL. This approach enables emergent abilities, such as self-correction and reflection, using only outcome-based rewards. Works like Open DeepResearch (6) have underscored the immense potential of tool invocation within LLMs. However, the application of agentic tool use, particularly code execution for mathematical tasks, has received comparatively less attention. We posit that for mathematical problems, precise and deterministic computation via code execution is often more advantageous than retrieving potentially noisy textual information from search engines, primarily due to the deterministic nature and rapid feedback loop of code execution @ g We also note contemporaneous work, such as the excellent contribution by TORL (9, which explores the potential of using code execution for mathematical tasks.
However, we believe that RL from a fine-tuned model with existing tooling capabilities obscures some important findings. Similar to RL from model after SFT, it is difficult to observe a relationship between response length and performance. The paper aims to provide a more comprehensive and clearer analysis to facilitate community research and reproduction of what we term Agent RL Scaling Law. We present exhaustive experiments conducted using mainstream community frameworks (Open-Reasoner-Zero [TO. OpenRL.F (1I) and popular RI. algorithms (PPO [12), Reinforce++ ), coupled with an environment server. We investigate how L.Mss, initialized from base models, can spontaneously learn to leverage a Python code execution environment through RL. Our central hypothesis is that the learning process for utilizing such a tool adheres to discernible patterns, which we designate as Agent RL Scaling Law.
We identify and characterize novel Agent RL Scaling Law that govern the autonomous acquisition of spontaneous code execution skills in ZeroTIR for mathematical reasoning.
We propose and implement an effective framework, ARL, for training base LLMs to spontaneously leverage code execution, which can quickly enabled on the community mainstream RL training frameworks.
We provide extensive empirical validation showing that ZTRL model trained with ZeroTIR significantly outperform non-instrumental ZeroRL baselines challenging mathematical benchmarks and SFT-based TIR methods.
By discovering the Agent RL Scaling Law and developing a robust framework to leverage them, our approach significantly advances the understanding and application of reinforcement learning for training capable I.I.M agents that autonomously utilize tools for complex reasoning tasks like mathematics.

---

Proept eigh--igh irtaere
cuuctly
irteger
betwees N and 2015
Ziero 72R Rg
Zaro AX
eigge thr each digne
Johi the we rocs cesuriin th the carter sight -ight smnppera formed only in desiil
comeitaiee fure suitrre thes
by that coalt wirderr and
digtt enticcle are 2. 4.O t. So. there are o choies for for dight
Let's wnite the hymees code ho sewv this probirs.
bey al thy by number to de otte difterane Extwaan
all thilatedde hell Lan'l the pa
dedd
ant ob cll be euth pia of dex 30 22, and 20 lica four day froe can nangge from
denccedy permurational of the digtt
y
Diertin shech tf dinablityy
Fd
P954 ngod 19t6 degt the number of sighn-dig vmsggrr 20
cnt Jor "uopeggin sero))
2964t
Pindly we medd to tind the difterero betvere and 2025, Co 27940- 2025 25023
mon siiihi by. IP neee for prrn poreosteee V chole Ripered
The craner
whe iepiene
porttosent www.bit 2C officeoo))
2.
1000 striemee betwee this mumber
Figure 2: Example responses to the same prompt, contrasting the code-augmented solution generated by ZeroTIR (left) with the reasoning-onke solution from ZeroRL (right).
2
Related Works
Our research builds upon advancements in LLM reasoning, TIR, and agent reinforcement learning.
2.1
Tote-Inte Reasoning
To overcome the limitations of purely internal reasoning, particularly in domains requiring calculation or factual lpok- [1). TIR approaches augment LLMs with external tools. These tools can range from calculators and search engines code interpreters. Much existing work implements TIR through SFT, training models on datasets containing tool invocation examples, or through carefully designed prompts and controlled mechanisms that trigger tool use under specific conditions, such as Sween.55MMth-TT tools according to predefined patterns or explicit instructions [7]. Our research diverges from these controlled approaches by focusing on spontaneous tool use. We investigate whether agent can learn autonomously, through ZeroRL, when and how to utilize a code execution tool based purely on maximizing task success reward, without being explicitly instructed or pre-trained on tool-use trajectories.
4 [6]. These methods typically guide or compel the model to use
2.2 Agent Reinforcement Learning
LLM Agents, capable of autonomous planning, decision-making and environmental interaction including tool use, represent significant evolution [18]. Reinforcement Learning provides a powerful paradigm for training these agents. Agent RL has been applied to train LLMS for information retrieval tasks, with frameworks like Search-RI [19] and R1-Searcher 20 teaching models autonomously query search engines during reasoning using outcome-based rewards 21 This principle is also being
applied to computational tools; Contemporaneous work like TORL g uses ZeroRL to train agents for code interpreter use in mathematics. A notable trend in Agent RL is the effectiveness of simpler, outcome-based rewards over complex process rewards or imitation learning, fostering exploration and the emergence of novel strategies. We situate our research within the Agent RL landscape, specifically focusing on the integration of code execution tool for enhancing mathematical reasoning of base model. Distinct from search-focused agents and building upon the ZeroRL paradigm, our primary contribution lies in the systematic identification and analysis of the Agent RL Scaling Law that govern the learning dynamics of spontaneous tool acquisition across various RL algorithms and frameworks, providing foundational understanding and reproducible benchmark for this specific type of Agent RL.
3

---

Feedback
Thinking Process
Prompt
Ep
Go
Ex
dx
1x+1
0k+1
ck+2
\boxed()
stop
2
python
python code
extract
Error Log
Environment
Or
verify
execute
Output
Figure 3: Detailed schematic of the interactive rollout process.
3
Methodology
ZeroTIR, trains base LLM to autonomously utilize Python code execution environment for mathe- matical problem-solving through reinforcement learning. This section details the core components of our framework, including the RI. formulation, the interaction mechanism between the LLM and the code environment, and specific design choices made during implementation and training.
3.1 RL for Spontaneous Code Execution
Our methodology utilizes reinforcement learning train the LLM agent, primarily employing policy gradient algorithms such as PPO and REINFORCE variants like Reinforce++. PPO operates as an actor-critic method, optimizing a policy network Te while concurrently training value network V to estimate state values, which aids in reducing the variance of policy gradient estimates. The policy update in PPO typically maximizes clipped surrogate objective function:
I. C.LLP = B. min (0) clip(r, - + +))+)
where re(0) xo(ae]s)) is the probability ratio between the current policy and the policy used Wo (Aeles) (8.)
to collect the data (Tesaa) Å, is the estimated advantage at timestep I, and is hyperparameter defining the clipping range. The value function Vo is trained concurrently, usually by minimizing the mean squared error against target values derived from sampled rewards.
crucial aspect of applying PPO in our setting involves handling the tokens inserted by the external code execution environment &code. The value function Va(st) should not be trained on states st corresponding these cettromment-geeeraatee tokens, as they are not products of the policy wa- Thus, the value loss computation must mask these states. Furthermore, the advantage estimation Åe, especially when using Generalized Advantage Estimation (GAE) with * 1, requires careful consideration. GAE computes advantages based on temporal difference (TD) errors:
T-11
ACAE Σ
(7A)
where & = (V Va(*t)
1=0
Here, 7 the discount factor and À is the GAE parameter balancing bias and variance. Since (o(5)1)) is unreliable or ill-defined if s141 consists of environment feedback tokens, terms involving these states can introduce noise or bias into ÂÇAE A common practice is to effectively mask the influence environment-inserte
of these tokens, potentially by setting A, tokens during the policy loss calculation, preventing spurious updates to Tg. However, we found that setting À to did not affect the results, here we set À to 1.
positions corresponding
REINFORCF-style algorithms, including Reinforce+++, operate differently, typically foregoing an explicit learned value function. The policy gradient is estimated directly using sampled trajectories.

---

The general form of the policy gradient objective is:
T- Σ t=
VoJ ETrwy
Vo log To (o (sc)
where the agent learns by adjusting in the direction suggested by the gradient. The advantage estimate Ag in this context is often calculated using the discounted Monte Carlo return G, minus baseline b(s)):
T
Å. = G. - b(st), where G. G.=E,
Tk
k=
The baseline b(8z) serves reduce variance and can be simple running average of returns or potentially more sophisticated estimates, but crucially, it doesn't rely on learned value function Vo in the same way PPO does. While the advantage calculation itself is less directly complicated by the value of environment states, careful reward attribution and ensuring only policy-generated actions (a, corresponding to LLM tokens) contribute to the gradient term V log ne (ar|s;) remain essential. Reinforce++ specifically may incorporate advanced baseline techniques or modifications to the update rule for improved performance compared to vanilla REINFORCE.
Irrespective of the specific algorithm, the overall learning objective remains the maximization of the expected outcome- based reward R(x, y), regularized by the KL divergence from a reference policy Tref to maintain stability:
max
(-Ie [R(r, v)] 33 [=o Ecode) | (-12;; Ecade))
Here, a is the input problem, is the full trajectory including interactions with the code environment Ecodes R(2, y) the outcome-based reward, Tref the reference policy, DK & the KL divergence. and 3 controls regularization.
3.2 Training Stability and Efficiency Techniques
Training stability and efficiency crucial in interactive RL settings like ZeroTIR. To address potential instability, such as training collapse observed in some ZeroRL frameworks, we employ two key techniques.
First, we introduce a replay buffer filtering mechanism to enhance stability and focus learning. Multiple responses generated for the same prompt are grouped, and their final answer accuracy (based on outcome rewards) is calculated. We filter out groups with accuracy above a high threshold 0.8 or below low threshold 0.2, prioritizing samples within the intermediate range where the learning gradient is likely most beneficial.
Second, we implement an efficient interaction mechanism for spontaneous code execution during rollouts, depicted schematically in Figure and detailed in Algorithm [T] This method leverages dynamic stop tokens (e.g.. "'python, "") to iteratively manage reasoning. code generation, interaction with the external code environment, and integration of execution feedback. This state-machine approach is significantly more efficient than generating complete sequences followed by post-hoc parsing for code extraction. This mechanism also facilitates managing tool interaction frequency by counting completed execution cycles (Neatls). For experimental control, particularly in initial runs managing computational resources, we enforce maximum call limit (Nmaz). ). Upon reaching this limit, notification ("Tool call count has been exhausted. You can no longer call the tool.") is injected into the context before the final generation resumption, ensuring the agent relies on internal reasoning thereafter.
3.3 Environment Interaction Frameworks
Effective Agent RL involving externl tools necessitates robust and scalable environment interaction. Integrating execution enviromments directly into RL training code often creates dependency issues and hinders modularity. Our initial approach addressed this by implementing the Python code execution environment as an independent, network-accessible service. This decoupled architecture enhances robustness compared to embedding the interpreter locally, as service failures do not crash the main training process. It also improves maintainability and allows the service be scaled
5

---

Algorithm ZeroTIR Rollout with Spontaneous Code Calls
Require: policy æ. prompt P, code env E, call budget N
1: C+ Pt-,, T+00, 0,
2: while true do
3: [.D])-....EEATTETI (EOS, /bozed[) "<pyhhnn
4:
C,T+C+ T +
5:
o E ( then return T
6:
end if
7:
i' = "pyton then
8:
k+ +1
9:
if > N then return T end
11:
(c, (.__)ag "...T..GEEEEEE GENGERAAT(EC,
C.T+C+ T r (==EBEEC(EXTRACTTC
13:
14:
C + FMT(r), T TOKENS(F)
15:
elsereturn T
end
17:
end while
Table I: Performance comparison key mathematical reasoning benchmarks.
Model
Params
Tool
AIME24
AIME25
MATH500
Avg.
Code Prop.
Qwen2.5 Ins.
13.3%
20.0%
75.8%
36.4%
0.0
Qwen2.5 Ins.
7B
/
16.7%
0.0%
76.4%
31.0%
0.0
Qwen2.5 Math Ins.
7B
13.3%
6.7%
83.2%
34.4%
0.0
Qwen2.5 Math Ins.
7B
V
20.0%
26.7%
78.0%
41.6%
95%
SimpleRI--Zero
7B
33.3%
6.7%
77.2%
39.1%
0.0
rStar-Math
7B
26.7% X
78.4%
52.6%
0.0
Eurus-2-PRIME
7B
X
26.7%
13.3%
79.2%
39.7%
0.0
TORL
7B
43.3%
30.0%
82.2%
51.8%
83%
ZTRL
7B
46.7%
30.0%
85.2%
54.0%
90%
independently using standard web serving technologies like Flask, Gunicorn for concurrency, Nginx for load balancing, and client-side rate limiting with libraries such as 'aiolimiter' to handle numerous concurrent requests stably. The RL framework interacts with this service via standard HTTP calls.
While the decoupled service provides stability and scalability, synchronous interaction can become bottleneck. To further improve training throughput, especially for large-scale experiments, implemented an enhanced interaction framework within OpenRLHF based on asynchronous rollout and pipelining. In this setup, each rollout actor manages an asynchronous queue, which is pre-filled with generated experiences. During training. data retrieval from queue occurs concurrently with the submission of new rollout task an actor. Experience generation proceeds asynchronously while the trainer performs parameter updates, and results are collected afterward to refill the queue. An adaptive mechanism handles potential data shortages in the replay buffer due to filtering by synchronously collecting completed rollouts when needed, preventing excessive data accumulation at the rollout actors. This asynchronous pipeline approach yielded significant speedups in our experiments, proving approximately 1.6 times faster than basic asynchronous rollout and over 4 times faster than the initial synchronous decoupled server interaction, enabling more extensive experimentation, particularly with larger models. For detailed design, please refer to our github repository.
4 Experiments
In this section, we detail the experimental setup designed to evaluate our ZeroTTR approach and validate the existence of Agent RI. Scaling Law for spontaneous code execution in mathematical reasoning. We describe the datasets, baselines, implementation details, and present the core results demonstrating the effectiveness of our method and the observed scaling phenomena.

---

4.1 Experimental Setup
Our experiments primarily utilize the Qwen 2.5 Base 7B/32B model, starting directly from pre-traine pre-trained weights to align with the ZeroRL philosophy. We implement our TIr approach using standard community frameworks OpenRLHF and Open- Bpen-Reasoner-eers Zero, and evaluate key RL algorithms including PPO and Reinforce++. The training dataset consists of OR7-57k [22 and deepmath [23 dataset containing verifiable mathematical problems. We call the model trained in this way ZTRI.. Model performance is evaluated on suite of standard mathematical reasoning benchmarks such as
MATH500(24), AIME24/25 25] 26], HMMT Feb. 24/25[27), emime [28), olymemath (29 and so on,
which are some of the most difficult mathematical data sets out there.
Key RL hyperparameters include rollout batch size of 128, with 16 samples generated per prompt. We use policy update step and 12 critic update steps per iteration. Micro-batch sizes for training and forward passes are set to Stability and efficiency techniques, including group-accuracy replay buffer filtering and dynamic stop-token based interaction (detailed in Section 3.2). are employed. The decoupled code execution environment (Section 3.3 handles all tool calls. For initial scaling law validation experiments, the maximum tool calls per trajectory were limited (N = 20) for efficiency. The The evaluation metrics include greedy decoding (temperature=0), majority voting, pass@k k, and the final performance measured under different top-p sampling settings (temperature=1).
4.2 Comparative Performance Analysis
Only Qwen base, beyond Qwen math. To evaluate the effectiveness of our ZeroTIR approach, denoted ZTRL, we present a comparative analysis in Table This table compares our method, trained on the Qwen 2.5 Base 7B model, against relevant baselines and oftte-of-the-at art models 30 32
The results clearly demonstrate the significant advantage of our ZeroTIR approach. Our 7B ZTRL model achieves strong average performance of 52.3% across AIME24, AIME25, and MATH500. Furthermore, ZTRL at 52.3% average significantly surpasses the performance of the Instruct model using prompted TIR, and other ZeroRL methods without tool integration like SimpleRL-Zero at 39.1% and Eurus-2-PRIME at 39.7%. Notably, our ZTRL model, trained from the general Qwen 2.5 Base, also outperforms the specialized Qwen 2.5 Math Instruct model when used with its integrated TIR capability, achieving a 52.3% average compared to 41.6%. We also compare our results with TORL [9) significant concurrent work applying ZeroRL with TIR mathematical reasoning. It is important to note that the reported TORL results utilized the math-specialized Qwen 2.5 Math Base model for training. As shown in Table D our ZTRL approach, despite starting from the general Qwen 2.5 Base model, achieves a slightly higher average score of 52.3% versus TORL's 51.8% on the AIME24, AIME25, and MATH500 benchmarks. This highlights the robustness and effectiveness of our specific ZeroTTR implementation in eliciting strong tool-augmented reasoning capabilities even without domain-specialized base. The high code usage proportion observed for our model, 89%, is comparable to TORL's 's 83% and correlates strongly with the achieved performance, consistent with the Agent RL Scaling Law discussed.
4.3 Analysis of hyperparameters
Table 2 confirms a strong monotonic relation between the interaction cap Nmaz and accuracy across all model scales. Raising the cap from zero four or twenty lifts average scores by as much as fifteen percentage points, though gains taper beyond four calls, underscoring agent reinforcement-leammig scaling law in which additional tool use yields better problem solving. Performance also grows with model size. Under identical hyperparameters the 32B model surpasses the 7B variant, which itself exceeds the 1.5B bascline. The code-usage ratio rises non-linearly, implying that larger models either resolve more tasks without code employ code more efficiently. At the 7B scale Reinforce++ and PPO reach similar final accuracies, yet Reinforce++ converges roughly three hundred steps sooner, attaining near-optimal performance around step four hundred while PPO needs more than seven hundred. DeepMath training offers a small but consistent edge over Orz-57k, so Reinforce++ with DeepMath was selected for 32B runs to combine efficiency and ceiling.
Table 4.3
shows that data choice still matters at high capacity. DeepMath delivers the highest Max score, sixty percent on HMMT Feb. 25, whereas Orz-trained models provide stronger majority robustness on CMIMC, fifty-three percent Maj versus thirty-three for DeepMath. Proof-focused curricula sharpen peak reasoning, while heterogencous contest data stabilises consensus. Decoding
7

---

Table 2: Detailed final performance comparison selected mathematical reasoning benchmarks.
Train Elv: Ite..
Eval Env. Iter.
hmmt feb. 25
hmm feb 24
olymenath
code ratio
aime25
aime24
emime
math
aV..
Method
Params
Algorithm
Datasel
ZTRL
1.5B
ppo
orz-57k
0
0
10.0%
3.3%
0.0%
0.0%
3.3%
2.0%
55.8%
10.6%
0.000
ZTRL
1.5B
ppo
orz-57k
2
2
3.3%
3.3%
0.0%
0.0%
3.3%
1.25%
60.6%
10.3%
0.073
ZTRL
1.5B
ppo
orz-57k
4
4
10.0%
20.0%
10.0%
0.0%
10.0%
5.0%
59,4%)
16.3% 2.161
ZTRL
1.5B
ppo
orz-57k
20
20
13.3%
13.3%
10.0%
0.0%
13.3%
7.75%
62.6%
17.2%
4.090
ZTRL
7B
ppo
orz-57k
0
4
26.7%
13.3%
13.3%
6.7%
10.0%
8.2%
80.6%
22.7%
0.143
ZTRL
7B
ppo
orz-57k
20
20
26.7%
50.0%
10.0%
20.0%
6.
13.5%
80.2%
31.0%
3,490
ZTRL
7B
Reinforce++ orz-57k 2
2
26.7%
30.0%
16.7%
13.3%
26.7%
12.3%
82.8%
29.7% 3,686
ZTRL
7B
Reinforee++ dcepmath 2
2
16.7%
36.7%
20.0%
10.0%
20.0%
13.5%
81.0%
29.6% 1.710
ZTRL
7B
Reinforee++ deepmath 4
16.7% 40.0%
16.7%
16.7%
20.0%
13.2%
80.6%
29.1% 2.417
ZTRI.
78
Reinforee++ doepmath 4 2
26.7%
36.7%
16.7%
23.3%
20.0%
12.7%
81.2%
31.3% 2.257
ZTRL
7B
Reinforce++ deepmath 4
4 26.
33.3%
20.0%
23.3%
20.0%
12.5%
82.0%
32.1% 2,470
ORZ
7B
PPO
orz-57k 0
0
10.0%
16.7%
0.0%
6.7%
10.0%
7.0%
82.2%
18.9% 0.000
DeepMath-7er
7B
deepmath
0
0
13.3%
23.3%
13.3%
6.7%
10.0%
7.2%
82.4%
22.3% 0.0000
ORZ
32B
PPO
orz-57k 0
30.0% 0
40.0%
20.0%
20.0%
30.0%
20.8%
90.6%
35.9%
0.000
ZTRL
32B
Reinforce++ deepmath 2
2
26.7%
53.3%
20.0%
16.7%
20.0%
16.7 7%
86.2%
34.2% 1.691
ZTRL
32B
Reinforce++ deepmath 2
4
26.7%
50.0%
16.7%
26.
23 3%
19.0%
87.8%
35.7% 1.994
ZTRL
32B
Reinforce++ deepmath 2
33.3%
56.7 %
20%
26.7 % 33.3%
17.5%
8%
39.3% 1,558
ZTRL
32B
Reinforce++ deepmath
4 4 30.0% 46.7% 20.0%
23.3%
36.7%
2188%
89.4% 38.2% 1.863
entropy drives a clear Max-Maj trade-off. With a four-call budget and DeepMath training, raising top-p from zero point seven to point zero increases Max by seven points on HMMT Feb. 25 yet lowers Maj and Avg on several sets. Reduced entropy boosts agreement with only modest loss of peak score. Lowering the training interaction cap from four two seldom harms and sometimes improves evaluation accuracy at cap four. On AIME25 Max rises from fifty to sixty-six percent, indicating that the large model quickly internalises useful code heuristics and that excessive training calls may overfit early exploration.
These findings show that very large models benefit less from extreme training interaction budgets, that dataset diversity and decoding entropy must be tuned jointly to balance peak and robust metrics, and that curriculum choice remains influential even at the 32B scale.
Table 3: Four training configurations (step 350) across five math-conted datasets. Train/Test interaction limit; orz = orz-57k; dm = deepmath; Max = max @ 32; Maj = maj@ 32; P@ = pass Avg- avg @ 32.
Aorrz, 4/4, p=0.7)
B(dm, 4/4, p= 1.0)
C(dm, 2/4, p=1.0)
D(dm, 4/4, p=0.7)
Dataset
Max P@1 1
Maj
Avg
Max
Maj
P@
Avg
Max
Maj
P@1 1
Avg
Max
Maj
P@
Avg
AIME25
63% 53%
35%
35%
50%
40%
28%
27%
66%
40%
30%
29%
56%
36%
31%
30%
AIME24
76%
60%
42%
41%
76%
63%
48%
48%
66%
63%
42%
42%
76%
50%
43%
43%
HMMT Feb. 25
53%
20%
19%
19%
60%
33%
25%
25%
50%
23%
22%
22%
50%
33%
22%
22%
HMMT Feh. 24
53%
33%
25%
24%
53%
33%
25%
24%
46%
26%
22%
22%
50%
40%
26%
26%
CMIMC
60%
53%
34%
34%
63%
33%
30%
30%
63%
33%
30%
30%
56%
33%
31%
31%
4.4 Analysis of Training Dynamies
Figure 4 presents key training dynamics, offering insights into the learning process and the Agent RL Scaling Law across different experimental settings. The evolution of code-related metrics is particularly revealing. Code Proportion. representing spontaneous code usage frequency, consistently shows an initial dip followed by a significant increase for TIR-enabled models, indicating that the agent learns the utility of the tool over time after overcoming initial generation challenges. Concurrently, the Code in Correct metric, tracking correct answers involving code, rises sharply in conjunction with the Raw Reward Avg, empirically linking effective, learned tool use directly to task success.
Response Length generally increases with training, especially for larger models, correlating with the inclusion of code and outputs, though this trend does not perfectly mirror reward improvements across all settings. The reward curves clearly confirm that enabling tool interaction leads to superior
8

---

300
200
200
20
80
20
Teares
Zome Cram
Caarr Cresn
Prppstio
Figure 4: Training dynamics comparing TIroTIR and ZeroRL in different model and RL algorithm.
performance compared to non-tool baselines, and further illustrate positive scaling with both increased maximum allowed interactions N Maz and larger model parameter counts.
Interestingly, while ablations show performance benefits from higher Naa values, the Code Count Avg across different settings often stabilizes between and 2 calls per response. This suggests that although a larger interaction budget is available and beneficial, the agents predominantly learn strategies involving few interactions. Combining these observations, learning via Zero'TIR, early attempts at multiple interactions might yield poor rewards due to lower code quality, leading the agent to converge towards highly effective single-call strategies. Indeed, our analysis indicates that over 90% of correct responses involving code utilize only a single code execution, with fewer than 10% employing two or more calls, even when permitted.
conclude that for base models
4.5 Joint scaling of training and inference- interactions
Figure shows the accuracy obtained when we train agents with 2, 4 interactions per problem and then test them with 1-16 interactions. On every benchmark, allowing more interactions at inference time yields higher scores, but the improvement becomes progressively smaller the budget reaches 8-16 steps. The curves also reveal a clear interaction between the two budgets: models that were trained with many interactions profit the most from a generous inference budget, yet they can be slightly worse than the low-budget model when only a single step is permitted at test time. In other words, heavy reliance multi-step reasoning during training can hurt in the one-shot regime, while paying off strongly once at least four inference steps are available.
The extent of these effects varies across datasets. On the harder AIME24/25 sets the gap between the 8-interaction model and the 2-interaction model widens to more than ten percentage points when sixteen inference steps are allowed, whereas HMMT24/25 the corresponding gap is around seven points. Taken together, the results suggest that training and deployment budgets should he aligned: application only afford but if larger inference budget is feasible, giving the model ample interaction capacity during training unlocks noticeably better final accuracy.
reasoning steps it is safer train with the same constraint,
Conclusion
This work investigated the autonomous acquisition of tool use, specifically spontaneous Python code execution for mathematical reasoning, by base Large Language Models trained via outcome- based Reinforcement Learning, ZeroTIR. Our central contribution is identification and empirical characterization of Agent RI. Scaling Law governing this learning process. We showed that training progression leads predictable dynamics tool usage frequency, code quality, response length, and task accuracy. Key findings include the positive scaling of performance with both model size and

---

30
35
Thaid www.rrrrieeeetttnenn.
(a) AIME24
(b) AIME25
Rerincione Trenn Trme staractioe Mirratioes
25
1.
Interence-Tine interactions
Interence-Pie Time Internations
(c) HMMT24
(d) HMMT25
Beeiiin-TTmm--iee Time- Intertnnesactions and snference-Timer Time--nteracction We use average 32 to compute the score for AIME24, AIME25, HMMT24 HMMT25. We use 500-step checkpoint for different
Figure 5: Training-
to perform the fair evaluation.
Theintay Time brieeaction
Theiing Time Inteactioon
eiring wewiein-mmmmtttii wnw.emmeeeeeeennn.. Hirenggt
caining-interactioo @
(b) training-intern interaction @4
(c) interactionnneeeee @
Figure 6: Training time interactions and performance (AVG@32) 32). Increasing the inference interaction number from to to improves all datasets. AIME24 shows the largest and nearly monotonic gains, and AIME25 is second. HMMT 24 and 25 rise steadily. CMIMC peaks around 4 interactions and then slightly declines but remains above The overall average flattens near 8 interactions, so using 8 interactions gives good balance between accuracy and cost.
the allowed budget for enviromment interactions, alongside insights into algorithm efficiency where Reinforce++ showed faster convergence than PPO in our experiments, and the deepmath dataset yielded strong results. Crucially, we revealed typical patterns of Tool-Integrated Reasoning usage that emerge during ZeroRL training. Our analysis of training dynamics and interaction counts suggests that while increased interaction potential improves results, models often converge to strategies favoring fewer, high-utility code calls, with majority of successful tool-using solutions employing only a single execution. Due to resource limitations, this work primarily focused on empirically demonstrating the existence and qualitative nature of these scaling law. rigorous quantitative analysis to determine the precise mathematical form of these relationships remains an important direction for future work. Further research should also explore removing constraints on interaction counts entirely and investigating performance in even more complex agentic environments. Overall, our findings advance the understanding of autonomous tool learning Agent RI. and provide reproducible framework for studying these critical scaling effects.
10

---

Acknowledgments
This work was supported by National Natural Science Foundation of China (No.62576109, 62072112, 12471280, 12101241) and grant from the Shanghai Municipal Education Commis- sion (N0.0024))).00..
We also thank Zhong-Zhi Zhi Li and Jian Hu for valuable advice and discussions.
References
[1] K. Wang, H. Ren, A. Zhou, 7. Lu, S. Luo, W. Shi, R. Zhang, I.. Song, M. Zhan, and H. Li, "Mathonder: Seamless code integration in ilms for enhanced mathematical reasoning," arXiv preprint arXiv:2310003311, 2023.
[2] L. Gao, J. Schulman, and J. Hilton, "Scaling Laws for Reward Model Overoptimization" arXiv preprint arXiv:2210.1072 2022.
[3] H. Wang, Li, C. Qu, W. Xu, F. Zhu, Y. Xin, W. Chu, and Lin, "AutoCode4Math: Learning Autonomous Code Integration for Math LLMs," arXiv preprint arXiv:2409. 14668, 2024.
[4] A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu. J. Tu. J. Zhou, J. Lin et al.. "Qwen2. 5-math tech- nical report: Toward mathematical expert model via self-imprrvement"" arXiv preprint 1211:2240,.21122, 2024.
[5] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., "Deepseek-rl: Incentivizing reasoning capability iims via reinforcement learning." arXiv preprint arXiv:2501. 2025.
[6] A. Roucher, A. V. del Moral, T. Wolf, L. Werra, and E. Kaunismäki, "smmolagents': smol library build great agentic systems." http:///hhwbbbooo/ggggggggg/////
2025.
[7] L. Gao. A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, Callan, and G. Neubig. "PAL: Program- Aided Language Models," Advances in Neural Information Processing Systems (NeurIPS), 2022.
[8] W. Chen, X. Ma. X. Wang, W. Weng, B. Sadler, H. Hajishirzi, and M. Ostendorf, "Program of Thoughts Prompting: Disentangling Computation from Reasoning Numerical Reasoning," Transactions on Machine Learning Research (TML.R) 2023, arXiv:2211.12588
[9] X. Li, H. Zou, and P. Liu, "Torl: Scaling tool tool-integrated rl," arXiv preprint arXiv:2503.23338 2025.
[10] J. Hu, Y. Zhang, Q Han, D. Jiang, X. Zhang, and H.-Y. Shum, "Open-reasoonerzzer: An open source approach scaling reinforcement learning the base model," GitHub repository, 2025, uRL.: http:////gghuub.oo//////// Reasoner- Reasoner-Zer) Version accessed relevant to user's experiments. [Online]. Available: https://gituu..cm//// Reasoner- Reasoner-Zert
[11] G. Sheng, C. Zhang, Z. Ye, Wu, W. Zhang, R. Zhang, Peng. Lin, and C. Wu, "Hybridflow: flexible and efficient rlhf framework," arXiv preprint MXXi:22409.19255 2024.
[12] Schulman, F. Wolski, P. Dhariwal, A. Radford, and Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv: 1707.06777.634 2017.
[13] Hu, "Reinforce+:: A simple and efficient approach for aligning large language models." arXiv preprint 2XXi:25011.03222. 2025.
[14] Wei, X. Wang, Schuurmans, Bosma, Xia, E. Chi, Q Le, Zhou et al., "Chain-of thought prompting elicits reasoning in large language models," Advances neural information processing systems, vol. 35, pp. 24 824-24 837, 2022.
[15] Z. Hou, Du, Y. Niu. Z. Du. A. Zeng. X. Liu, M. Huang, H. Wang. J. Tang, and Y. Dong, "Does RLHF Scale? Exploring the Impacts from Data, Model, and Method," arXiv preprint arXiv:2412.06000 2024.
[16] S. Yao, J. Zhao, Yu, N. Du, Shafran, K. Narasimhan, and Y. Cao, "ReAct: Synergizing Reasoning and Acting in Language Models," in Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
[17] T. Schick, J. Dwivedi-VV Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, "Toolformer: Language Models Can Teach Themselves to Use Tools." arXiv preprint 072.41230 2023.
11

---

[18] C. Qu. S. Dai. X. Wei. H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen, "Tool learning with large language models: A survey." Frontiers of Computer Science, vol. 19, no. p. 198343, 2025.
[19] B. Jin, H. Zeng, Z. Yue, J. Yoon, S. O. Arik, D. Wang, H. Zamani, and J. Han, "Search-r1:: Training Ilms to reason and leverage search engines with reinforcement learning," arXiv preprint arXiv:250.009516 2025.
[20] S. Sun, X. Jin, Y. Zhou, G. Dong, Y. Zhang, J. Liu, and Z. Dou, "RI-searcher: Incentivizing the search capability in Ilms via reinforcement learning," arXiv preprint arXiv:2503.05592, 2025.
[21] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, W. Saunders, and J. Schulman, "WebGPT: Browser-assisted Question-Answering with Human Feedback," arXiv preprint
urXiv:2112.093322222 2022.
[22] J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H.-Y. Shum, "Open-reasoner-zzeo: An open source approach to scaling up reinforcement learning on the base model," arXiv preprint arXiv:2503.24290. 2025.
[23] Z. He. T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang al., "Deepmath- 103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning," arXiv preprint arXiv:2504.1 1456, 2025.
[24] D. Hendrycks, C. Burns, Kadavath, A. Arora, S. Basart, E. Tang. D. Song, and Steinhardt, "Measuring mathematical problem solving with the math dataset," in NeurIPS Dataxets and Benchmarks, 2021. [Online]. Available:
[25] M. Jia, "Aime 2024 dataset card," http://ngggigiacceoooaaaaaaaaaaa 2024 2024-03.
version
[26] Mathematical Association of America, "American invitational mathematics examination (aime) problems and solutions," caautiions,"htippan http://mww.rrrr/uuuuuee--rrrrrno
accessed 9 May 2025.
[27] MIT Mathematics Tournament, "Hmmt fehruary problem archive," https:/www.hmmmttng//wwwww tournaments/testing accessed 9 May 2025.
[28] Carnegie Mellon Informatics and Mathematics Competition, "Cmime math - past problems," https:
accessed 9 May 2025.
[29] H. Sun, Y. Min, Z. Chen, W. Zhao et al., "Challenging the boundaries of reasoning: An olympiad-level math benehmark for large language models," arXiv preprint arXiv:2503.21380, 2025.
[30] W. Zeng, Y. Huang, W. Liu, K. He, Q. Liu, Z. Ma, Jnd ing with reinforcement learning is both effective and efficient," 2025, notion Blog.
J. He, "7b "7b model and 8k examples: Emerging reason-
[31] G. Cui, L. Yuan, Z. Wang, H. Wang. W. Li, B. He, Y. Fan, T. Yu. Q. Xu, W. Chen et al., "Process reinforcement through implicit rewards," arXiv preprint arXiv:2502.0144 012.002.01456 2025.
[32] X. Guan, I. I. Zhang, Y. Liu, N. Shang, Y. Sun, Y. 7hu, F. Yang, and M. Yang, "rstar-math: Small llms can master math reasoning with self-evolved deep thinking," arXiv preprint arXiv:2501.05579, 2025.
12

---

A Statistical Evidence of the Scaling Relationship
We sample ~500 training steps batch data and compute the association between code/tool-usage rate and task accuracy. We observe strong positive monotonic correlation with Spearman p 0.686 and extremely small p-values. A log-log linear fit between effective training steps and accuracy yields R 0.93, indicating a predictable trend as the training budget grows. We also observe characteristic trajectory from non-mat non-math, non- TIR base: the fraction of responses using code initially dips (0.142 - 0.002) before rising high plateau (> 0.98), alongside the accuracy increase after the inflection.
Table Summary statistics linking tool usage and accuracy.
Statistic
Value
Note
Spearman (usage vs. acc.) Log-log fit Ri (steps vs. acc.) Early usage dip > plateau
0.686
strong positive association high goodness of fit phase transition pattern
0.93
0.142 + 0.002 > > 0.98
Base-model rationale. Effective tool is a prerequisite for outcome-only training. We therefore adopt a base with better tool-utilization priors, which empirically reduces tool over-calling and hallucination and stabilizes the RL signal.
B Tool-Call Efficiency and "Laziness"
We analyze the number of tool invocations per example. Incorrect answers average 2.7 calls versus 1.9 for correct answers. As training progresses, tool calls become more efficient: 90% of correct code-using cases require single call. Increasing the maximum allowed interactions Nmax improves accuracy with diminishing returns beyond R4; competent models typically solve with 1-2 calls. Table 5: Tool-call efficiency analysis. Lower calls for correct solutions indicate non-brute-force behavior.
Metric
Value
Observation
Avg. calls (incorrect)
2.7
higher than correct lower, more efficient
Avg. calls (correct)
1.9
Single-call share (correct, code-using)
> >90
efficiency improves
Benefit of N Nmax
diminishing beyond ~4
accuracy gains taper off
C Extended ZeroTIR Training and Validation Results
During the extended experiment phase, we reran the ZeroTIR training process up to 1,500 steps and recorded the average@33 @32 accuracy on all three evaluation benchmarks (AIME24, AIME25, and HMMT25). The complete trend of training-ste progression and corresponding validation accuracy summarized in Figure
Key observations. Accuracy continues to improve or remain stable up to 1,500 steps, with no sign of degradation collapse. Both AIME24 and HMMT25 exhibit noticeable late-stage gains, indicating that the agent benefits from additional computation without overfitting to earlier exploration patterns. These results further support the stability and scalability of the ZeroTTR framework under long-horizon training.
D Cross-Dataset Generalization and Comparative Trends
To complement the single-run results presented above, we additionally tracked the performance of ZeroTTR across multiple datasets, iterations, and baselines. Figure visualizes the extended evaluation curve covering AIME24, AIME25, and the averaged five-dataset setting compared against non-thinking bascline (Claude-Optus4).
13

---

Evaluation-Curve
50
48
46
44
42
40
38
36
34
32
30
28
26
24
zero-tir-7b-ave
22
zero-tir-7b-aee
tirr-7b-aime25
20
18
zero-tir-bbaamme24
16
non thinkinggaimez
14
zenotirrbbbavooo---dttttttt-e-e
12
10
zero-tin7b-aiime25--ter
zero-tin7b-aimme24-ierr
0 100 200 300 400 500 600 700 800 90110001002200133044000111110000
Training-Stepp
Figure 7: Cross-dataset evaluation under prolonged training. The Zero7IR-7B series demonstrates stable and monotonic improvement across all datasets up to 1,500 steps, surpassing the non-thinking baseline and showing overfitting collapse.
Observations. The averaged five-dataset curve (blue) provides a robust indicator of global general- ization, with accuracy continuing rise smoothly through extended optimization. Both single-task and averaged variants of ZeRoTTRR consistently outperform the static baseline across the full training horizon. These results validate that ZeroTIR's improvement is not confined to specific benchmark but reflects a scalable and transferable training dynamic across diverse reasoning domains.
E Limitations
The strongest results here are math-centric. While the infrastructure is general and decoupled, broader evaluations across non-math tasks, alternative tools, and efficiency -oriented rewards are left for future extensions to assess the scope and boundary conditions of the observed scaling behaviors.
Performance under Long Training Steps
45
Accuracy (%6
35
30
20
15
AIME25
HMMT25
10
AIME24
200
400
600
800
1000
1200
1400
Training Steps
Figure 8: Smoothed validation accuracy curves for AIME24, AIME25, and HMMT25 across 1,500 training steps. Accuracy continues to improve or stabilize, showing no collapse and confirming late-stage gains.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and Introduction clearly state three contributions -ntibbiiiiiiiiiiiiiiiiiiiiin Agent RL Scaling Law, providing Zero'TIR framework, and empirically validating indand
Justification:
and these are borne out by the methodology in Section and the results in Section 4.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: The Conclusion candidly notes resource constraints and that precise mathe- matical form of the scaling law is left for future work, explicitly framing these limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The work is empirical; it introduces theorems and therefore requires no formal proofs.
4. Experimental result reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper?
Answer: [Yes]
Justification: Section 4.1 specifies datasets, model sizes, hyper-parameters, and links to an open GitHub repository, enabling faithful replication of the experiments.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results?
Answer: [Yes]
Justification: Code is publicly released GitHub, and all referenced datasets are publicly available, with citations provided for each.
6. Experimental setting/details
Question: Does the paper specify all the training and test details necessary to understand the results?
Answer: [Yes]
Justification: Training batch sizes, optimizer details, interaction limits, and evaluation protocols appear in Section 4.1 and Table 2, offering sufficient context.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about statistical significance?
Answer: [Yes]
Justification: Results reported point accuracies and averages.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources needed reproduce the experiments?
Answer: [Yes]
Justification: The manuscript omits hardware details are given.
15

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics?
Answer: [Yes]
Justification:
The study uses only publicly available data, releases code responsibly, involves human subjects, and follows standard ethical practice.
10. Broader impacts
Question: Does the paper discuss both potential positive and negative societal impacts of the work performed?
Answer: [No]
Justification: The manuscript does not contain Broader-Impact discussion of how tonomous tool-using LLMs might be beneficial or misused.
Safeguards
Question: Does the paper describe safeguards for responsible release of data models that have high risk for misuse?
Answer: [No]
Justification: Although the model is publicly released, the paper does not outline usage restrictions, filters, or other safeguards against malicious exploitation.
12. Licenses for existing assets
Question: Are the creators or original owners of assets properly credited and are the license and terms of use explicitly mentioned?
Answer: [Yes]
Justification: Prior models and datasets are cited.
13. New assets
Question: Are new assets introduced in the paper well documented?
Answer: [NA]
Justification: The work does not release a new dataset; the ZTRL model is fine-tuned derivative and not provided as a separate asset.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full instructions and compensation details?
Answer: [NA]
Justification: No crowdsourcing human-subject research is involved.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks participants and IRB approvals?
Answer: [NA]
Justification: The study does not involve human subjects and therefore requires no IRB review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is an important component of the core methods?
Answer: [Yes]
Justification: Sections and detail fine-tuning and reinforcement-leaminn of the Qwen 2.5 base model, making LLM usage central to the method.
16


Reviews:
[{"review_id": "0yc0Uaw7Z5", "reviewer": "Reviewer_Xq2s", "summary": "This paper investigates how large language models (LLMs) can spontaneously learn to use external tools—specifically a Python code execution environment—for mathematical reasoning through reinforcement learning (RL). The authors introduce a framework called ZeroTIR, which trains base models using outcome-based rewards without relying on supervised fine-tuning (SFT). A key contribution of the paper is the discovery and characterization of the Agent RL Scaling Law: as training progresses, the model's spontaneous tool usage, response length, and task accuracy all increase in a predictable manner. Extensive experiments using PPO and Reinforce++, along with a decoupled code execution environment, demonstrate the effectiveness of ZeroTIR. The proposed method (ZeroTIR) outperforms both non-tool ZeroRL baselines and prior SFT-based approaches across multiple challenging mathematical benchmarks.", "questions": "1.\tThe authors should provide more experimental results of state-of-the-art SFT-based TIR methods to demonstrate that ZeroTIR indeed outperforms these approaches.\n2.\tZeroTIR appears to be more of a benchmark rather than a novel method. The authors should conduct additional experiments on other open-source foundation models (such as LLaMA and Mistral) to verify whether ZeroTIR maintains its superiority on those models.\n3.\tFigure 4 shows the training dynamics of ZeroTIR and ZeroRL. However, the authors should present results over longer training steps for ZeroTIR to examine whether it eventually collapses.", "limitations": "see questions", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths: This paper addresses an important and under-explored problem: how large language model (LLM) agents can autonomously learn to use tools (such as code execution) without  supervised tool-use examples. The proposed Agent RL Scaling Law offers a new perspective on how tool-use behavior emerges as the training scale increases. It is both an empirical insight and a promising benchmark for future research in Agent RL.\nThe experimental results are comprehensive, covering multiple model scales (1.5B, 7B, 32B), reward formulations, datasets (ORZ-57k, DeepMath), and reinforcement learning algorithms (PPO, Reinforce++). The method is also benchmarked on competitive datasets such as AIME, MATH500, and HMMT.\nThe paper is well-structured and clearly written. The training pipeline, reward design, and algorithmic choices are easy to follow.\n\n\nWeaknesses: \nThis work lacks methodological innovation and is more of an empirical study. In fact, the experimental conclusions are already known — for example, DeepSeek-R1 also explores RL training based on base models rather than SFT-tuned models, although its results show that performing RL on SFT-tuned models leads to better performance than doing so on base models.\nThis work claims that ZeroTIR significantly outperforms both non-instrumental ZeroRL and SFT-based TIR approaches, such superiority is not clearly observed in Table 1.\nAll experiments in this work are conducted using the Qwen2.5 model series. While these models are strong, evaluating ZeroTIR on other open-source base models (such as LLaMA or Mistral), and comparing against RL applied after SFT on those models, would help strengthen the generalizability of the claims.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and careful review. Your opinions are very valuable to us. We address your concerns point by point below.\n\nFirst, ZeroTIR means Training the base model to integrate tool usage via reinforcement learning (RL), relying exclusively on output-based rewards, yields experimentally non-trivial results. The ZeroTIR setting is not an obvious conclusion like DeepSeek R1 to reach for the following reasons:\n\na. Unlike the methodology in ToRL or Qwen2.5 Math, which explicitly enforces code-based problem solving, our approach leaves open a critical question: can the model gradually acquire tool-utilization capability for mathematical problem solving when guided only by outcome rewards on base model without cold-start? A significant risk remains that the model may fall back on text-based chain-of-thought as its primary strategy.\n\nb. In contrast to ToRL, our experiments start from the \\textt{qwen25-general-base} model, whose initial code-usage ratio and pass rate at step 0 are merely 10 percent—far lower than the 30 percent reported in ToRL. With such a baseline, applying RL with only an Output Reward Model (ORM) offers no guarantee that the model will learn to exploit tools or deliver meaningful gains. We argue that this discrepancy deserves deeper investigation. \n\nTable: ACC and CODE-usage \n| Training-Steps   | 50    | 100    | 150    | 200    | 250    | 300    | 350    | 400    | 450    | 500|\n|---------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| ACC  | 0.105  | 0.342  | 0.366  | 0.374  | 0.404  | 0.422  | 0.396  | 0.400  | 0.398  |0.388|\n| CODE-Usage  | 0.142   | 0.002  | 0.046  | 0.616  | 0.848  | 0.755  | 0.979  | 0.984  | 0.987  |0.805|\n\nAs can be seen from the above table, the code ratio initially dropped from 0.14 to 0.002, then gradually increased, and finally reached a very high proportion of code usage. This phenomenon is probably hard to anticipate before any exploration is conducted since we only use ORM to guide the training process of RL.\n\n\n1. Comparison with RL-after-SFT and state-of-the-art SFT-based TIR\n\n1.1 We begin training from a base model rather than an SFT-tuned model. This choice is fairer: SFT quality depends on format, scale, and data construction, so validating RL algorithms directly on bases is common practice (SimpleRL [1], DAPO [2], ToRL [3]).\n\n1.2 The strongest SFT-tuned model currently available is Qwen2.5 Math TIR, which already reaches state-of-the-art math performance via massive SFT data. Other SFT-tuned models, including those based on LLaMA or Mistral, lag far behind Qwen2.5 Math TIR [4].\n\n1.3 To widen comparisons, we released ZTRL-0727-7B. Despite its smaller size, it consistently surpasses larger competitors; checkpoints and data are linked in the paper. On avg@32, ZTRL-0727-7B exceeds qwen3-238b-a22b-non-thinking by 5.3 pp on AIME25 and 12.08 pp on HMMT25, and is close to—or better than—the closed-source SOTA claude optus4-non-thinking.\n\n\nTable 1: The results of the ZTRL model on the avg@32.\n\n| Dataset   | ZTRL-0727-7B | Qwen3-238B-A22B-Non-thinking | Claude Optus4-Non-thinking |\n|-----------|--------------|------------------------------|----------------------------|\n| AIME25    | 29.99%       | 24.7%                        | 33.9%                      |\n| HMMT25    | 22.08%       | 10.0%                        | 15.9%                      |\n\n\nTable 2: The results of the ZTRL model on the Greedy.\n| Dataset   | ZTRL-0727-7B | TORL  |\n|-----------|--------------|-------|\n| AIME25    | 32.81%       | 30%   |\n| AMC23     | 80%          | 75%   |\n| MATH500   | 84.6%        | 82.2% |\n\n\n2. Other open-source foundation models\n\nTool-calling competence is a prerequisite. Other open-source bases (e.g.\\ LLaMA, Mistral) show weak tool performance and lack TIR training; recent studies [5,6] confirm that community models tend to over-call tools and cannot admit failure. Qwen2.5 models excel precisely because TIR data were injected during pre-training. For example, LLaMA 3.1 8B scores an F1 of 16.6 with 67 percent tool hallucination, whereas Qwen2.5 7B reaches F1 32.0 with 21 percent hallucination. Recent mid-training work [7,8] also reports that LLaMA bases are ill-suited to zero-RL, performing far below the Qwen family. For these reasons we chose Qwen2.5 for RL training.\n\n\n3. Longer training steps\n\nOur open-source framework trains stably beyond 1,000 steps; extended results are shown below.\n\n\nTable 3: The val results of the ZTRL-0727-7B beyond 1,000 steps. avg@32\n\n| Training-Steps   | 100    | 200    | 300    | 400    | 500    | 600    | 700    | 800    | 900    | 1000|\n|---------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| AIME25  | 12.18  | 25.00  | 28.22  | 28.12  | 28.02  | 28.12  | 29.79  | 32.81  | 29.99  |30.31|\n| HMMT25  | 8.54   | 18.95  | 21.14  | 20.83  | 20.62  | 22.08  | 24.16  | 21.25  | 22.08  |22.18|\n\nOnce again, thank you for your time. If these clarifications resolve your concerns, we would be grateful if you could consider raising your score.\n\n\nReferences\n\n[1] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild  \n[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale  \n[3] ToRL: Scaling Tool-Integrated RL  \n[4] Qwen2.5-Math Technical Report: Toward a Mathematical Expert Model via Self-Improvement  \n[5] When2Call: When (not) to Call Tools  \n[6] T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step  \n[7] Behavior Injection: Preparing Language Models for Reinforcement Learning  \n[8] OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling"}, {"review_id": "mt2fd8R2dT", "reviewer": "Reviewer_MhKq", "summary": "This paper discusses how existing RL for LM training strategies, specifically PPO and REINFORCE, can be readapted to enable LMs to issue tool calls in a multi-turn setting towards solving math problems. To make this work, the authors make a key change of incorporating execution logs into the inference process, but masking them out when the value (for PPO) or advantage (for REINFORCE) is being calculated to perform the gradient update. The authors also generally cover some additional details, such as special tokens, limiting tools calls, and speeding up training via parallelized, asynchronous calls. Finally, the experiments demonstrate improved performance on AIME24 and comparable numbers for AIME25 and MATH500, while also highlighting interesting trends around the frequency with which coding tool calls are issued during inference.", "questions": "- For Table 1, in the “Avg.” column, why is 52.3% bolded? Is there a reason for this, or should 52.6% for rStar-Math be bolded instead? Same goes for 83.2% under MATH500 instead of 82.2%.\n- You propose Agent RL as a framework that generally elicits spontaneous code execution behavior. Currently, the training framework is math specific. What infrastructure changes would it take to make this work for truly agentic evaluation settings, such as web navigation, code generation, or software engineering? The edits right now are also Python specific - how would you enable more language agnostic code execution?\n- Do the average scores account for the number of task instances per dataset? Or is this an average across the percentages achieved per benchmark?\n\nMinor edits\n- Line 190: I think the first sentence is incomplete.\n- You use the ZTRL acronym without saying what it means in Line 61 for the first time. Can you provide the full phrase that this abbreviation refers to somewhere?\n- I think the captions could be much more informative. In its current state, the results tables are quite hard to understand for me. For instance, what is “code ratio” in Table 2?\n- Feel free to ignore this suggestion, but I think Agent RL is a bit non-specific of a title. I think the nature of this work seems to lean heavily towards tool calling that involves the use of code execution to solve math problems. In this sense, the word “agent” feels a bit too general and potentially over-claiming in the worst case. It might be helpful to rephrase “Agent RL Scaling Law” in a way that defines the scope of this work more specifically.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- I like the contribution of this work, particularly the comparison between tool integrated and no-tool reasoning for math benchmarks. The angle of the LM spontaneously invoking tools, rather than being prompted to, is also a cool behavior to have elicited via training. I think the implications of this work are encouraging as RL for LM research efforts start to look beyond reasoning traces towards settings that incorporate execution environments.\n- I appreciate how the methodology was well grounded in prior work. I found the recap of REINFORCE and PPO to be quite helpful, and I really liked the level of detail that was captured. For instance, the authors highlight that it’s important the value function is not trained on execution outputs from tool calls, and that there were two ways to go about this (zero’ing out the estimated advantage versus the GAE parameter. The discussion on the challenges of getting the training infrastructure to play well with the execution environments (Section 3.3) was also well placed, and a good precedent for future explorations into more RL for tool calling / agent.\n\nWeaknesses\n- The performance gains on AIME24 are quite convincing, but less so for the other evaluations (e.g. AIME25, MATH500). For some settings, it is obvious that *RL methods are better than the base model (+ ~30-35%), but for others, such as MATH500, the gain is a lot less (~5%), or non-existent with AIME24 (both 26.7%). I guess it’s just a bit strange to me why this is the case, when the task formulations for AIME24 and AIME25 should be identical? \n- I think the clarity of the paper could be improved. I found Section 4.3 and 4.4 specifically to be a bit difficult to parse. There were a lot of statistics and trends jumbled together, and although the phenomena are all very interesting, I think they could be conveyed more clearly, with more explanation that attempts to explain why certain trends might emerge. For instance, Figure 4 is quite neat, but I think (1) the font should be larger, (2) the statistics (titles of each graph) are explained in terms of what they reflect, and (2) the direction of the trends are discussed more clearly.", "was_revised": true, "final_justification": "As mentioned in my response to the authors, I've decided to raise my rating from 3 to 4. I appreciate the authors' clarifications, and I think the highlighting of the concrete takeaways helped my understanding of this work. As this contribution is specific to the math domain, I think greater impact could be achieved by expansion to more agentic domains, but I still think this work is quite worthwhile and contributes useful infrastructure to the community.", "rebuttal": "Thank you very much for your time and careful review. Your review comments are very meaningful to us. We will address your concerns point by point as follows:\n\n1. for others settings the gain is a lot less  \nI understand your question as follows: we achieved a large improvement (+30–35%) on AIME25, but the gains on AIME24 and MATH500 are less obvious. First, based on Table 1, since our ZTRL model is trained from the Qwen2.5 Base model, the fair comparison baseline should be Qwen2.5 Instruct (whose training data volume is orders of magnitude larger than ours). Compared with Qwen2.5 Instruct, we obtain +26.7% on AIME24 and +3.8% on MATH500. In our newly released ZTRL-0727-7B, in order to broaden the comparison to more models, we compared against models much larger than ours. Experiments show that we consistently outperform models trained by other methods. The data and checkpoints can be found in the GitHub link in the main text for verification, and we will add these results to the paper. ZTRL-0727-7B exceeds qwen3-238b-a22b-non-thinking by 5.3% and 12.08% on AIME25 and HMMT25 avg@32, respectively, and is close to or surpasses the closed-source sota model claude optus4-non-thinking.\n\nTable 1: The results of the ZTRL model on the avg@32.\n\n| Dataset  | ZTRL-0727-7B | Qwen3-238B-A22B-Non-thinking | Claude Optus4-Non-thinking |\n|----------|--------------|------------------------------|----------------------------|\n| AIME25   | 29.99%       | 24.7%                        | 33.9%                      |\n| HMMT25   | 22.08%       | 10.0%                        | 15.9%                      |\n\n\n\nTable 2: The results of the ZTRL model on the Greedy.\n\n| Dataset  | ZTRL-0727-7B | TORL  |\n|----------|--------------|-------|\n| AIME25   | 32.81%       | 30%   |\n| AMC23    | 80%          | 75%   |\n| MATH500  | 84.6%        | 82.2% |\n\n2. Section 4.3 and 4.4 specifically to be a bit difficult to parse  \nIn Sections 4.3 and 4.4, the core insights we intended to present are these three points:  \n(1) Code usage dips at first (the model fumbles), then rises as it learns the tool’s value. “Code-in-correct” and reward curves climb together: effective code use drives success. Response length grows (more code/output) but does not perfectly track reward.  \n(2) Bigger + faster matters, but data and entropy still rule. Larger models (32B > 7B > 1.5B) perform better. Reinforce++ reaches the same peak as PPO about 300 steps sooner. Proof-heavy data (DeepMath) boosts best single-shot scores (Max), while diverse contest data (Orz-57k) boosts consensus metrics (Maj/Avg). Higher decoding entropy (top-p) trades higher Max for lower Maj, so dataset choice and entropy must be tuned together.  \n(3) Interaction cap helps—up to a point. Letting the model call tools more times (N_{\\text{max}} from 0→4→20) raises accuracy, but gains flatten after about 4 calls. Bigger models still end up using only 1–2 calls per problem because early multi-call attempts are low quality and get low reward.  \nWe will consider your suggestions and add more explanations, analyze in detail why these trends appear, provide more discussion, and revise the figures.\n\n3. bolded in table  \nYes, that is our mistake. We will correct it in the main text. Since the newly released ZTRL-0727 achieved a significantly improved new sota, detailed results can be found at the GitHub link in our paper. We will revise the tables as a whole.\n\n4. language agnostic code execution or other environment  \nIn fact, during implementation, we decoupled the environment from the agent training framework. By swapping the env service to a specific environment, cross-environment agent RL training can be achieved (e.g., for appworld, webshop benchmarks). The main infrastructure change needed is to wrap the corresponding benchmark as a server that conforms to our current interface standard, after which training can proceed. I understand this as a decoupled approach. We will update the code repository to better support related training.\n\n5. the number of task instances per dataset  \nThe average score is the mean of percentages across benchmarks. This is the standard practice in math evaluation, as seen in papers such as Qwen2.5 Math (Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement). To align with other papers, we adopt the mean of percentages across benchmarks [1][2].\n\n6. Minor edits  \nWe will modify the first sentence at Line 190, add the full phrase for ZTRL, and clarify that code ratio represents the proportion of responses containing code among all responses. We will add this explanation in the figure captions. We will reconsider the title “Agent RL Scaling Law.” In fact, we hope to define standard interfaces that adapt to most environments in the future, and we have already implemented preliminary functionality. Thank you very much for your careful review.\n\nOnce again, thank you for your time. If your concerns are alleviated, we would greatly appreciate it if you could consider raising your score. We sincerely thank you for your consideration.\n\nReferences:  \n[1] TORL: Scaling Tool-Integrated RL  \n[2] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild"}, {"review_id": "HFkCJhqC1z", "reviewer": "Reviewer_cNww", "summary": "In this work, the authors elicit tool-usage ability in reasoning LLMs without expert demonstrations and via verifiable RL alone. They observe that as training progresses, its dynamics is predictable and proficiency in tool (chiefly code) usage correlates strongly with proficiency in solving the task, which the authors deem a \"scaling law\". The authors report extensive evaluations and compare with relevant baselines, showing competitive results.", "questions": "Each question is preceded by the number of the line it refers to:\n- 104: Could the authors clarify which tokens correspond to actions, and which do not?\n- 113: *However, we found that setting $\\lambda$ to 0 did not affect the results, so here we set \\lambda to 0*. Could the authors elaborate? I cannot see how this statement follows from the discussion that precedes it.\n- 142: Does appending the notification ensure with 100% certainty that the agent will not attempt to call the compiler again?\n- 187: The authors here mention several metrics; but table 3, where these are reported, is not referenced in the text.\n- 190: \"Only Qwen base, beyond Qwen math\". This sentence ends abruptly. \n- 210: \"Table 2 confirms a strong monotonic relation\" It's honestly hard to read this monotonic relation from such a dense table. It would be helpful to substantiate it with a plot or, even better, with a correlation coefficient.\n- 221: \"Table 4.3\" Is it table 3 that is being referenced here.\n- Figure 4: Could the authors increase the font size in the axes and labels? It would be almost unreadable if the paper were printed. Also, since the author's main point is about the correlation between tool calling and task success, it would be helpful to report these metrics on the same plot.", "limitations": "There is no separate discussion about limitations, which are somewhat discussed in the conclusion and used as a basis to propose future work. The authors should discuss them more openly and in a separate section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Investigating bootstrapping of tool-usage without expert demonstrations is an interesting an clean idea.\n- The authors report comprehensive evaluation with different training setups and dataset, as well as relevant and competitive baselines.\n- The authors do show that their trained models can learn to leverage python without expert demonstrations, which at the sizes they consider is not obvious, and worth reporting.\n\n**Weaknesses**:\n- The \"scaling law\" presented in figure 1 is not a scaling law, which should be reported with a log-scaled x-axis. The dynamics observed by the authors is indeed predictable, but the data presented do not support the statement that it follows a scaling law in the strict sense of the term.\n- The authors' statement on correlation between tool usage and task success, while reasonable, is not substantiated by any further analysis (e.g. Spearman Correlation).\n- The performance gains compared to SOTA are ultimately modest. In terms of performance averaged over the benchmarks considered (table 1), the ZTRL method proposed by authors is only slightly better than the TORL baseline, and worse than rStar-Math.\n- A couple typos can be found in the text and the figures are poorly readable, suggesting that the paper was hastily written.", "was_revised": true, "final_justification": "The authors have diligently improved their paper with a statistical analysis of the correlation between tool usage and task success, and ran longer training experiments, showing more robust improvements over the baseline.", "rebuttal": "Thank you very much for your time and careful review. Your review is very important to us. It helped us correct many writing problems. Thank you very much for your effort. We will address your concerns point by point as follows:\n\n1. log-scaled x-axis  \nYour suggestion is correct. In fact, we have released the checkpoint of ZTRL-0727. This model substantially surpasses previous ones, and because it was trained for more steps, it exhibits a more pronounced scaling law phenomenon. We will add the log-log fitted curve and the coefficient of determination (R²) for ZTRL-0727 in the appendix.\n\n2. Spearman Correlation  \nWe agree with your suggestion. We will include statistical tests in the appendix. First, we use Spearman’s rank correlation coefficient to quantify the monotonic relationship between tool usage rate and success rate, analyzing the monotonic relationship between rule_rewards (accuracy) and code_ratio (code proportion) during training. The mean of rule_rewards is 0.3759, the standard deviation is 0.0677; the mean of code_ratio is 0.6629, the standard deviation is 0.3678. The result is ρ = 0.686, p = 7.06e-71, n = 500, indicating a strong correlation and high statistical significance (p < 0.001). We will further add other statistical testing methods in the appendix. \n\n3. performance  \n3.1 Our setting differs from rStar-Math. rStar-Math uses a much larger supervised/distillation/multi-stage RL pipeline and relies on Monte Carlo Tree Search. Our goal is to study the scalability of “Zero-RL + spontaneous tool use”. rStar-Math uses a Process Preference Model for step-level verification and a math-specialized base model as the backbone, while we only use the general-base model as both backbone and ORM, and rely on RL to help the model learn to use tools to solve complex math problems. The starting points differ and may not be suitable for direct comparison.  \n3.2 ZTRL-0727 further widens the gap on AIME25 and MATH500.\n\nTable 1: The results of the ZTRL model on the Greedy.\n\n| Dataset  | ZTRL-0727-7B | TORL   |\n|----------|--------------|--------|\n| AIME25   | 32.81%       | 30%    |\n| AMC23    | 80%          | 75%    |\n| MATH500  | 84.6%        | 82.2%  |\n\n4. which tokens correspond to actions, and which do not?  \nAction tokens are the special markers and parameter segments that trigger the external compilation/execution environment, including:  \n- All tokens between the start and end markers of <code_call> inserted in our template;  \n- Special control tokens parsed by the system as tool/action calls.  \nNon-action tokens are normal natural language reasoning/explanatory content.\n\n5. set \\lambda to 0.  \nThis is a typographical error; it should be set \\lambda to 1. Our expression was unclear: the actual reason is that rewards are extremely sparse and concentrated at the end of the sequence, and in our implementation we already perform “per-step equal distribution + group-wise normalization” of the advantage, so the temporal smoothing effect of GAE is almost negligible. Therefore, the results of \\lambda → 0.95 and \\lambda = 1.0 are nearly identical. Experiments verify this idea. Under this setting, the adv of environment feedback tokens equals 0, effectively adding an environment mask.\n\n6. call the compiler again?  \nStrictly speaking, it cannot be guaranteed 100% (an LLM might bypass the prompt), but since the correct answer is not provided, it will not be rewarded; such samples are penalized. This does not affect the training outcome.\n\n7. mention several metrics  \nThis was our oversight; we will remove it from the main text.\n\n8. Only Qwen base, beyond Qwen math  \nWe will revise this.\n\n9. substantiate it with a plot  \nWe agree. We will add a line/scatter plot showing the relationship between N_{\\text{max}} and accuracy, and report Spearman ρ.\n\n10. Table 4.3  \nThis is a typesetting error; it should be Table 3.\n\n11. the font size  \nWe will increase the font size and provide a high-resolution version in the appendix. We have also added a composite figure: tool usage rate and success rate are overlaid in the same chart (dual y-axis or normalized on the same axis) to illustrate the correlation more intuitively.\n\nOnce again, thank you for your time. If your concerns are alleviated, we would greatly appreciate it if you could consider raising your score. We sincerely thank you for your consideration."}, {"review_id": "3DClTZs27W", "reviewer": "Reviewer_96Y3", "summary": "This paper investigates the use of a Python interpreter to assist in solving mathematical problems. The research focuses on two main aspects: the effectiveness of problem-solving using this approach, and the relationships and scaling trends between various metrics when training models with this methodology.", "questions": "- The author could perhaps briefly analyze the differences in methodology between their approach and TORL, as well as the specific performance analysis results.\n- Why was a Python interpreter included in math problems? Is the issue of \"laziness\" a potential concern? Do these results align with expectations?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- This paper addresses an interesting and relevant problem, offering a solution that appears both reasonable and effective. \n- The authors conducted experiments and drawn several valuable conclusions that provide good insights.\n\nWeaknesses\n- From Table 1, it appears that across three datasets, the proposed method only surpasses TORL on AIME24. While I acknowledge the authors' contributions in other areas, there's still room for improvement in absolute performance. A deeper analysis of the underlying reasons for this performance gap might be beneficial.\n- In my view, the primary motivation for applying RL to mathematical problems, is to *enhance reasoning capabilities*. However, incorporating a code interpreter might encourage a tendency for the model to \"cheat\" or take shortcuts. For instance, in the case study presented in Figure 2, the model solves the problem using a brute-force algorithm, entirely lacking any mathematical optimization. In contrast, when I tested this problem with Gemini 2.5 Pro, the model was able to solve it elegantly using mathematical methods.", "was_revised": true, "final_justification": "My final score remains borderline accept (4).\nI think both questions raised are not fully addressed.", "rebuttal": "Thank you very much for your time and careful review. You have raised an interesting point, and we hope to continue discussing this with you in the next phase of the discussion. We will address your concerns point by point as follows:\n\n1. Absolute performance  \nWe trained for more steps and released ZTRL‑0727, which shows further gains on all datasets.  \nZTRL‑0727 widens the gap on AIME25 and MATH500.  \n\nTable 1: The results of the ZTRL model on the Greedy.\n\n| Dataset   | ZTRL-0727-7B | TORL   |\n|-----------|--------------|--------|\n| AIME25    | 32.81%       | 30%    |\n| AMC23     | 80%          | 75%    |\n| MATH500   | 84.6%        | 82.2%  |\n\nIn the newly released ZTRL‑0727‑7B, we also compare with much larger models. Results show we are consistently stronger; data and checkpoints are linked in GitHub and will be added to the paper.  \nOn the avg@32 metric, ZTRL‑0727‑7B surpasses qwen3‑238b‑a22b‑non‑thinking by 5.3 pp on AIME25 and 12.08 pp on HMMT25, and approaches or exceeds the closed‑source SOTA claude optus4‑non‑thinking.\n\nTable 2: The results of the ZTRL model on the avg@32.\n\n| Dataset  | ZTRL-0727-7B | Qwen3-238B-A22B-Non-thinking | Claude Optus4-Non-thinking |\n|----------|--------------|------------------------------|----------------------------|\n| AIME25   | 29.99%       | 24.7%                        | 33.9%                      |\n| HMMT25   | 22.08%       | 10.0%                        | 15.9%                      |\n\n2. “Cheat” or take shortcuts  \nOur model is only 7 B, a small backbone.  \n1. Tool use is itself part of strategic learning: human mathematicians also rely on calculators or code. RL aims to maximise task success and efficiency, not aesthetic elegance.  \n2. Not every problem is brute‑forced. We classify three solution modes (analytical, hybrid, brute enumeration). The latter two together are below 35 %. Many tasks still need symbolic reasoning to write correct or efficient code. Gemini, with larger pre‑training and math alignment, can solve elegantly without code; our setting is a smaller base, zero‑supervision RL, and limited resources. The two routes are not contradictory: the model can choose analysis when possible and code when computation is needed.\n\n3. Differences in methodology between our approach and TORL  \nParadigm: both works belong to the “zero‑RL + code tool” family and were done independently in the same period.  \n• We study the gap between a general base model and a math‑specialised base, and propose algorithmic changes needed during agent RL. TORL forces code execution; our training is spontaneous and only prompts “You can use code to solve your problem.” RL is used to let the model decide by itself. With longer training, the model shifts to code even without compulsion.  \n• TORL uses a math‑base model (effectively an SFT‑tuned backbone). Math‑base already underwent heavy TIR continued pre‑training, so its code‑ratio and pass‑ratio start around 0.2 and 0.8. The general base begins at 0.1 and 0.05, meaning far weaker TIR ability (though still better than llama3‑base).  \n• Performance: AIME24 contains more computational, small‑scale, sparse‑reward tasks, so tool calling gives a larger jump; MATH500 and AMC23 have many direct‑derivation or template problems, where TORL’s prompt is already well‑tailored. With longer training our ZTRL‑0727 further improves.\n\n4. “Laziness” as a potential concern  \nMath tasks often need exact arithmetic, enumeration, and combinatorial search. Learning to call external compute is crucial for real agents.  \nMitigation:  \n1. We impose a cost: each call adds tokens and potential failure.  \n2. As RL progresses, calls become more frequent but more efficient: over 90 % of correct answers with code need only one call; we do not observe blind multi‑call brute force.  \n3. Future work will add an explicit reward for reasoning quality or minimising calls to encourage “elegant” solutions.\n\nOnce again, thank you for your time. We hope to continue discussing this with you in the next phase of the discussion. If your concerns are alleviated, we would greatly appreciate it if you could consider raising your score. We sincerely thank you for your consideration."}]

Author rebuttal:

Meta-review: This work investigates the scaling behavior of tool-use during RL on an LLM without supervised tuning to show how tools should be used. It finds that LLMs spontaneously learn to use tools and that this ability correlates strongly with success.

Reviewers generally agreed that the findings in this work are interesting and somewhat novel, as most past work has involved supervised fine-tuning to teach tool usage prior to the RL phase. They did, however, express concerns that the contribution in the paper is primarily empirical and does not produce robustly supported scaling laws, per se, and that performance was not outstanding on some evals. In response, the authors produced substantial statistical evidence and a number of additional checkpoints with strong performance.

Overall, this work presents a somewhat mixed contribution of (a) a strong training recipe, with an emphasis on outperforming other models, and (b) empirical evidence of scaling relating tool call frequency and accuracy on various downstream benchmarks. While this makes it a slightly muddled story, and further work would be beneficial to establish more generalizable scaling *laws* in the common sense, its contributions are nontrivial, novel, and highly timely. Given this, we recommend acceptance of this paper. Please make sure to incorporate all the suggested revisions that emerged during the discussion period to strengthen the work.
Decision: Accept (poster)