Title: Agentic RL Scaling Law: Spontaneous Code Execution for Mathematical Problem Solving
Venue: NeurIPS 2025
Accepted: yes
Keywords: Agentic RL

Code Execution

LLMs

Math
Abstract: Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation. While Reinforcement Learning (RL) from outcome-based rewards  enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial. We investigate RL from outcome-based rewards for Tool-Integrated Reasoning, ZeroTIR, training base LLMs to spontaneously generate and execute Python code for mathematical problems without supervised tool-use examples. Our central contribution is we demonstrate that as RL training progresses, key metrics scale predictably. Specifically, we observe strong positive correlations where increased training steps lead to increases in the spontaneous code execution frequency, the average response length, and, critically, the final task accuracy. This suggests a quantifiable relationship between computational effort invested in training and the emergence of effective, tool-augmented reasoning strategies. We implement a robust framework featuring a decoupled code execution environment and validate our findings across standard RL algorithms and frameworks. Experiments show ZeroTIR significantly surpasses non-tool ZeroRL baselines on challenging math benchmarks. Our findings provide a foundational understanding of how autonomous tool use is acquired and scales within Agent RL, offering a reproducible benchmark for future studies. Code is released at \href{https://github.com/yyht/openrlhf_async_pipline}{https://github.com/yyht/openrlhf\_async\_pipline}.


Peer reviews:
comments: SUMMARY:
This paper investigates the use of a Python interpreter to assist in solving mathematical problems. The research focuses on two main aspects: the effectiveness of problem-solving using this approach, and the relationships and scaling trends between various metrics when training models with this methodology.

STRENGTHS AND WEAKNESSES:
Strengths
- This paper addresses an interesting and relevant problem, offering a solution that appears both reasonable and effective. 
- The authors conducted experiments and drawn several valuable conclusions that provide good insights.

Weaknesses
- From Table 1, it appears that across three datasets, the proposed method only surpasses TORL on AIME24. While I acknowledge the authors' contributions in other areas, there's still room for improvement in absolute performance. A deeper analysis of the underlying reasons for this performance gap might be beneficial.
- In my view, the primary motivation for applying RL to mathematical problems, is to *enhance reasoning capabilities*. However, incorporating a code interpreter might encourage a tendency for the model to "cheat" or take shortcuts. For instance, in the case study presented in Figure 2, the model solves the problem using a brute-force algorithm, entirely lacking any mathematical optimization. In contrast, when I tested this problem with Gemini 2.5 Pro, the model was able to solve it elegantly using mathematical methods.

QUESTIONS:
- The author could perhaps briefly analyze the differences in methodology between their approach and TORL, as well as the specific performance analysis results.
- Why was a Python interpreter included in math problems? Is the issue of "laziness" a potential concern? Do these results align with expectations?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors elicit tool-usage ability in reasoning LLMs without expert demonstrations and via verifiable RL alone. They observe that as training progresses, its dynamics is predictable and proficiency in tool (chiefly code) usage correlates strongly with proficiency in solving the task, which the authors deem a \"scaling law\". The authors report extensive evaluations and compare with relevant baselines, showing competitive results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Investigating bootstrapping of tool-usage without expert demonstrations is an interesting an clean idea.\n- The authors report comprehensive evaluation with different training setups and dataset, as well as relevant and competitive baselines.\n- The authors do show that their trained models can learn to leverage python without expert demonstrations, which at the sizes they consider is not obvious, and worth reporting.\n\n**Weaknesses**:\n- The \"scaling law\" presented in figure 1 is not a scaling law, which should be reported with a log-scaled x-axis. The dynamics observed by the authors is indeed predictable, but the data presented do not support the statement that it follows a scaling law in the strict sense of the term.\n- The authors' statement on correlation between tool usage and task success, while reasonable, is not substantiated by any further analysis (e.g. Spearman Correlation).\n- The performance gains compared to SOTA are ultimately modest. In terms of performance averaged over the benchmarks considered (table 1), the ZTRL method proposed by authors is only slightly better than the TORL baseline, and worse than rStar-Math.\n- A couple typos can be found in the text and the figures are poorly readable, suggesting that the paper was hastily written.\n\nQUESTIONS:\nEach question is preceded by the number of the line it refers to:\n- 104: Could the authors clarify which tokens correspond to actions, and which do not?\n- 113: *However, we found that setting $\\lambda$ to 0 did not affect the results, so here we set \\lambda to 0*. Could the authors elaborate? I cannot see how this statement follows from the discussion that precedes it.\n- 142: Does appending the notification ensure with 100% certainty that the agent will not attempt to call the compiler again?\n- 187: The authors here mention several metrics; but table 3, where these are reported, is not referenced in the text.\n- 190: \"Only Qwen base, beyond Qwen math\". This sentence ends abruptly. \n- 210: \"Table 2 confirms a strong monotonic relation\" It's honestly hard to read this monotonic relation from such a dense table. It would be helpful to substantiate it with a plot or, even better, with a correlation coefficient.\n- 221: \"Table 4.3\" Is it table 3 that is being referenced here.\n- Figure 4: Could the authors increase the font size in the axes and labels? It would be almost unreadable if the paper were printed. Also, since the author's main point is about the correlation between tool calling and task success, it would be helpful to report these metrics on the same plot.\n\nLIMITATIONS:\nThere is no separate discussion about limitations, which are somewhat discussed in the conclusion and used as a basis to propose future work. The authors should discuss them more openly and in a separate section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses how existing RL for LM training strategies, specifically PPO and REINFORCE, can be readapted to enable LMs to issue tool calls in a multi-turn setting towards solving math problems. To make this work, the authors make a key change of incorporating execution logs into the inference process, but masking them out when the value (for PPO) or advantage (for REINFORCE) is being calculated to perform the gradient update. The authors also generally cover some additional details, such as special tokens, limiting tools calls, and speeding up training via parallelized, asynchronous calls. Finally, the experiments demonstrate improved performance on AIME24 and comparable numbers for AIME25 and MATH500, while also highlighting interesting trends around the frequency with which coding tool calls are issued during inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- I like the contribution of this work, particularly the comparison between tool integrated and no-tool reasoning for math benchmarks. The angle of the LM spontaneously invoking tools, rather than being prompted to, is also a cool behavior to have elicited via training. I think the implications of this work are encouraging as RL for LM research efforts start to look beyond reasoning traces towards settings that incorporate execution environments.\n- I appreciate how the methodology was well grounded in prior work. I found the recap of REINFORCE and PPO to be quite helpful, and I really liked the level of detail that was captured. For instance, the authors highlight that it’s important the value function is not trained on execution outputs from tool calls, and that there were two ways to go about this (zero’ing out the estimated advantage versus the GAE parameter. The discussion on the challenges of getting the training infrastructure to play well with the execution environments (Section 3.3) was also well placed, and a good precedent for future explorations into more RL for tool calling / agent.\n\nWeaknesses\n- The performance gains on AIME24 are quite convincing, but less so for the other evaluations (e.g. AIME25, MATH500). For some settings, it is obvious that *RL methods are better than the base model (+ ~30-35%), but for others, such as MATH500, the gain is a lot less (~5%), or non-existent with AIME24 (both 26.7%). I guess it’s just a bit strange to me why this is the case, when the task formulations for AIME24 and AIME25 should be identical? \n- I think the clarity of the paper could be improved. I found Section 4.3 and 4.4 specifically to be a bit difficult to parse. There were a lot of statistics and trends jumbled together, and although the phenomena are all very interesting, I think they could be conveyed more clearly, with more explanation that attempts to explain why certain trends might emerge. For instance, Figure 4 is quite neat, but I think (1) the font should be larger, (2) the statistics (titles of each graph) are explained in terms of what they reflect, and (2) the direction of the trends are discussed more clearly.\n\nQUESTIONS:\n- For Table 1, in the “Avg.” column, why is 52.3% bolded? Is there a reason for this, or should 52.6% for rStar-Math be bolded instead? Same goes for 83.2% under MATH500 instead of 82.2%.\n- You propose Agent RL as a framework that generally elicits spontaneous code execution behavior. Currently, the training framework is math specific. What infrastructure changes would it take to make this work for truly agentic evaluation settings, such as web navigation, code generation, or software engineering? The edits right now are also Python specific - how would you enable more language agnostic code execution?\n- Do the average scores account for the number of task instances per dataset? Or is this an average across the percentages achieved per benchmark?\n\nMinor edits\n- Line 190: I think the first sentence is incomplete.\n- You use the ZTRL acronym without saying what it means in Line 61 for the first time. Can you provide the full phrase that this abbreviation refers to somewhere?\n- I think the captions could be much more informative. In its current state, the results tables are quite hard to understand for me. For instance, what is “code ratio” in Table 2?\n- Feel free to ignore this suggestion, but I think Agent RL is a bit non-specific of a title. I think the nature of this work seems to lean heavily towards tool calling that involves the use of code execution to solve math problems. In this sense, the word “agent” feels a bit too general and potentially over-claiming in the worst case. It might be helpful to rephrase “Agent RL Scaling Law” in a way that defines the scope of this work more specifically.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how large language models (LLMs) can spontaneously learn to use external tools—specifically a Python code execution environment—for mathematical reasoning through reinforcement learning (RL). The authors introduce a framework called ZeroTIR, which trains base models using outcome-based rewards without relying on supervised fine-tuning (SFT). A key contribution of the paper is the discovery and characterization of the Agent RL Scaling Law: as training progresses, the model's spontaneous tool usage, response length, and task accuracy all increase in a predictable manner. Extensive experiments using PPO and Reinforce++, along with a decoupled code execution environment, demonstrate the effectiveness of ZeroTIR. The proposed method (ZeroTIR) outperforms both non-tool ZeroRL baselines and prior SFT-based approaches across multiple challenging mathematical benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper addresses an important and under-explored problem: how large language model (LLM) agents can autonomously learn to use tools (such as code execution) without  supervised tool-use examples. The proposed Agent RL Scaling Law offers a new perspective on how tool-use behavior emerges as the training scale increases. It is both an empirical insight and a promising benchmark for future research in Agent RL.\nThe experimental results are comprehensive, covering multiple model scales (1.5B, 7B, 32B), reward formulations, datasets (ORZ-57k, DeepMath), and reinforcement learning algorithms (PPO, Reinforce++). The method is also benchmarked on competitive datasets such as AIME, MATH500, and HMMT.\nThe paper is well-structured and clearly written. The training pipeline, reward design, and algorithmic choices are easy to follow.\n\n\nWeaknesses: \nThis work lacks methodological innovation and is more of an empirical study. In fact, the experimental conclusions are already known — for example, DeepSeek-R1 also explores RL training based on base models rather than SFT-tuned models, although its results show that performing RL on SFT-tuned models leads to better performance than doing so on base models.\nThis work claims that ZeroTIR significantly outperforms both non-instrumental ZeroRL and SFT-based TIR approaches, such superiority is not clearly observed in Table 1.\nAll experiments in this work are conducted using the Qwen2.5 model series. While these models are strong, evaluating ZeroTIR on other open-source base models (such as LLaMA or Mistral), and comparing against RL applied after SFT on those models, would help strengthen the generalizability of the claims.\n\nQUESTIONS:\n1.\tThe authors should provide more experimental results of state-of-the-art SFT-based TIR methods to demonstrate that ZeroTIR indeed outperforms these approaches.\n2.\tZeroTIR appears to be more of a benchmark rather than a novel method. The authors should conduct additional experiments on other open-source foundation models (such as LLaMA and Mistral) to verify whether ZeroTIR maintains its superiority on those models.\n3.\tFigure 4 shows the training dynamics of ZeroTIR and ZeroRL. However, the authors should present results over longer training steps for ZeroTIR to examine whether it eventually collapses.\n\nLIMITATIONS:\nsee questions", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-06"}