AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent
Authors: Haipeng Luo
Huawen Feng
Qingfeng Sun
Can Xu
Kai Zheng
Yufei Wang
TAO YANG
Han Hu
Yansong Tang
Venue: ICLR 2026 Poster (2026)
Keywords: Mathematical reasoning
Large Language Models
Reinforcement Learning
Agent
Primary area: foundation or frontier models, including LLMs

Abstract: Large Reasoning Models (LRMs) like o3 and DeepSeek-R1 have achieved remarkable progress in natural language reasoning with long chain-of-thought. However, they remain computationally inefficient and struggle with accuracy when solving problems requiring complex mathematical operations. In this work, we present AgentMath, an agent framework that seamlessly integrates language models' reasoning capabilities with code interpreters' computational precision to efficiently tackle complex mathematical problems. Our approach introduces three key innovations: (1) An automated method that converts natural language chain-of-thought into structured tool-augmented trajectories, generating high-quality supervised fine-tuning (SFT) data to alleviate data scarcity; (2) A novel agentic reinforcement learning (RL) paradigm that dynamically interleaves natural language generation with real-time code execution. This enables models to autonomously learn optimal tool-use strategies through multi-round interactive feedback, while fostering emergent capabilities in code refinement and error correction; (3) An efficient training system incorporating innovative techniques, including request-level asynchronous rollout scheduling, agentic partial rollout, and prefix-aware weighted load balancing, achieving 4-5× speedup and making efficient RL training feasible on ultra-long sequences with scenarios with massive tool calls. Extensive evaluations show that AgentMath achieves state-of-the-art performance on challenging mathematical competition benchmarks including AIME24, AIME25, and HMMT25, substantially outperforming frontier open‑source models of comparable size.  Specifically, AgentMath-30B-A3B attains 90.6\%, 86.4\%, and 73.8\% accuracy respectively, surpassing OpenAI-o3-mini and Claude-Opus-4.0-Thinking while remaining competitive with OpenAI-o3, Gemini-2.5-Pro, and DeepSeek-R1-671B-0528. These results validate the effectiveness of our approach and pave the way for building scalable mathematical reasoning agents.

Published as a conference paper at ICLR 2026
AGENTMATH: EMPOWERING MATHEMATICAL REA- SONING FOR LARGE LANGUAGE MODELS VIA TOOL- AUGMENTED AGENT
Haipeng Luo 2.3 Huawen Feng" Qingfeng Sun
Can Xu
Kai Zheng
Yufei Wang" Tao Yang² Han Hu
Yansong Tanga
Shenzhen International Graduate School, Tsinghua University
Tencent Hunyuan *Pengcheng Laboratory
(1uchp24@ma 1s., Lang.yansongesse Jtsiiiittnnn edu.cn (bazzfeng, victorqsun, leocaxu, kaivenzanng, sarvvvanggetencettttoo trigorosyang, winstony otenseett
ABSTRACT
Large Reasoning Models (LRMs) like o3 and DeepSeek-RI have achieved re- markable progress in natural language reasoning with long chain-of-thought. However, they remain computationally inefficient and struggle with accuracy when solving problems requiring complex mathematical operations. In this work, we present AgentMath, an agent framework that seamlessly integrates language models' reasoning capabilities with code interpreters' computational precision efficiently tackle complex mathematical problems. Our approach introduces three key innovations: (1) An automated method that converts natural language chain- of-thought into structured tool-augmented trajectories, generating high-quality supervised fine-tuning (SFT) data to alleviate data scarcity; (2) A novel agen- tic reinforcement learning (RL) paradigm that dynamically interleaves natural language generation with real-time code execution. This enables models to au- tonomously learn optimal tool-use strategies through multi-round interactive feed- back, while fostering emergent capabilities code refinement and error correc- tion; (3) An efficient training system incorporating innovative techniques, in- cluding request-level asynchronous rollout scheduling, agentic partial rollout, and prefix-aware weighted load balancing, achieving 4-5x speedup and making effi- cient RI. training feasible on ultra-long sequences with scenarios with massive tool calls. Extensive evaluations show that AgentMath achieves state-of-the perfor- mance on challenging mathematical competition benchmarks including AIME24, AIME25, and HMMT25, substantially outperforming frontier open open-source mod- els of comparable size. Specifically, AgentMath-300R 30B-A3B attains 90.6%, 86.4%, and 73.8% accuracy respectively, surpassing OpenAl-o3-miní and Slaude-Opus- 4.0-Thinking while remaining competitive with OpenAl-o3, Gemini-2.5- and DeepSeck-RR -167 IB-0528. These results validate the effectiveness of our ap- proach and pave the way for building scalable mathematical reasoning agents.
INTRODUCTION
Large Reasoning Models (LRMs) such o3 and DeepSeek-R1 have made remarkable progress in natural language reasoning with long chain-of-thought (OpenAA et al. 2022 Team et al.
2025a DeepSeek-Al et al. 2025 XAI 2023 Claude, 2025 Team al. 2023 Wei et al.) 2022) However, when tackling mathematical problems that demand precise computation or intricate sym- bolic manipulation. including large-number arithmetic, complex equation solving, and geometric reasoning, pure text-based reasoning still has limitations: frequent computational errors necessitate redundant corrections, which in turn leads inefficiency and erroneous results.
Corresponding authors. This work was conducted during Luo's internship Tencent and was supported by the CTF- -TEe-eeneent Ph.D. Student Research Incentive Program (Tencent Hunyuan Special Fund)

---

Published as a conference paper at ICLR 2026
To enhance computational efficiency and aceuracy, recent work has explored incorporating external tools (i.e., code interpreters). delegating complex and prror-prone computational steps to external environments(Li et al. 2025e] Zhou et al. 2025al Lin & Xu 2025 Zhang et al. 2025b] Chen al. 2023 Gao et al. 2023 Gou et al. 2023b). For instance, models like o3 and 04-mini have signif- icantly improved mathematical reasoning accuracy through tool invocation. Nevertheless, existing approaches still face three critical challenges. First, high-quality tool-use data remains extremely scarce. While methods like START al. a (2025b) generate tool-augmented trajectories via prrmpt engineering, they suffer from delayed code computation and code result distrust; CoRT(Li et al., 2025a) employs manual annotation which is effective but lacks scalability; under supervised learn- ing, models struggle to learn autonomous debugging from code execution failures. Second, the po- tential for continuous performance improvement and tool-use strategy optimization through agentic RL remains unexplored. Third, competition-leve mathematical problems typically involve ultra- long reasoning chains with extensive tool invocations (i.c., 96k tokens, 96 tool calls), making tradi- tional batch-synchronous RL training frameworks inadequate for large-scale agent learning, while the rollout time for ultra-long sequences causes severe long-tail effects.
To address these challenges, propose AgentMath, tool-augmented agentic framework that seamlessly integrates model reasoning with code execution for efficient and reliable mathemati- cal problem-solving. AgentMath comprises three core components: First, we propose automated tool-augmented trajectory synthesis method that transforms pure-text long chain-of-thought data into structured training samples containing code execution and authentic feedback. Through code injection, execution verification, and nulti-dimensional refinement, this effectively alleviates data scarcity. Second, we design novel agentic reinforcement learning paradigm that supports dy- namic interleaving of natural language generation and code execution during reasoning. Through multi-turn interactive feedback, models autonomously learn optimal tool invocation strategies. Ex- periments reveal that model accuracy continuously improves with increasing tool invocations, ex- hibiting emergent code self-correction capabilities. Third, to support large-scale agentic RL train- ing SSchulman et al.) 2017 Li et al., 2025e; Feng et al., 2025b] Mai al. 2025), we develop an efficient training system incorporating key techniques such as request-level asynchronous rollout scheduling, agentic partial rollout, and prefix-aware weighted load balancing. These innovations improve training efficiency by 4-5x, effectively supporting reinforcement learning in scenarios with ultra-long sequences and extensive tool invocations.
Experimental results demonstrate that AgentMath achieves state-of-the-arr performance on challenging mathematical competition benchmarks including AIME24, AIME25, and HMMT25 (Balunvvaa et al.] 2025), outperforming frontier open-source tool-augmented models and pure-text reasoning models of comparable scale. Specifically, AgentMath-3003 33B-h achieves accuracies of 90.6%, 86.4%, and 73.8% respectively, surpassing OpenAT-o3-mint and Claude- Opus-4.0-Thinking while remaining competitive with Gemini-2.5-P and DeepSeek-R1-Op ---0588
Our main contributions include: (1) We propose an efficient automated tool-augmented data syn- thesis pipeline that effectively alleviates data scarcity issues. (2) We design a novel agentic rein- forcement learning paradigm achieving dynamic integration of natural language reasoning and code execution, enabling models to autonomously learn tool-use strategies through multi-turn interactive feedback. (3) We develop an efficient asynchronous training system that provides a scalable solu- tion for ultra-long sequences, multi-turn interaction agent reinforcement learning. (4) We achieve state-of-the oftthh--rttttttt-e- performance multiple challenging mathematical competition benchmarks, paving the way for building more complex and scalable mathematical reasoning agents.
2
METHOD
2.1 OVERVIEW
This section presents AgentMath, a tool-augmented agent framework designed to enhance complex mathematical reasoning by tightly integrating the emergent reasoning capabilities of Large Lan- guage Models (LLMs) with the precise arithmetic and symbolic computation facilitated by exter- nal code execution environment. The architecture operates in two stages: (i) supervised fine-tuning (SFT) curated, synthetic tool-invocation trajectories to establish initial competence in invoking tools appropriately, and (ii) large-scale reinforcement learning (RL) driven by outcome feedback to incentivize exploration and mastery of optimal, self-corrective tool-use strategies.
2

---

Published as a conference paper at ICLR 2026
Stage 1: Agentic Trajectory Generation via Code Injection
Stage 2: Multi-Faceted
Stage 3: Self Correctonn Capability Injection
Quality Refinement
Foundation Corpus
Format Condistency Cornection
LLM--bosed celf-correction
Succeeci
Code: Executability Verification
Agentic Trajectory Generation
Sandbox Correction
N-grom Overlop Filtering
Invalid
Sondbox Execution
Dropped
Success
Necessory Code Tnjection
Fragment Replocement
Initial Agentic Trajectories
Result Consistency Venfication
Valid
Final High-quality Detoset
This diagram outlines a three-stage pipeline for creating high-quality tool-augmented trajectories for training agents, including Agentic Trajectory Generation via Code Injection, Multi- Faceted Quality Refinement and Self-Correction Capability Injection. This automated process trans- forms pure-text reasoning into verified, executable agentic trajectories.
Figure I:
Problem Formulation and Interaction Protocol: We formulate tool-augmented mathematical rea- soning Markov Decision Process (MDP). The LLMM-baedd policy generates interleaved reason- ing segments and executable code blocks through interaction with a sandboxed execution environ- ment. Each trajectory consists of action-obserration pairs, where state transitions result from the policy's conditional generation and deterministic code execution. We define a structured markup protocol for agent-envirooment communication: <think>> denotes natural language reasoning, <code'> delimits executable code blocks, and <interpreter>> encapsulates execution feedback. This bidirectional exchange mechanism incorporates execution results into the generation context, enabling adaptive strategy refinement and planning. See Appendix A.2 for more details.
2.2 TOOL-DRIVEN DATA SYNTHESIS
The scarcity of high-qualiie training data that captures both complex reasoning patterns and strate- gic tool utilization remains fundamental bottleneck in developing code-enabled agents. This work introduces a three-stage automated manthesis-and-refineee CoT into agent-style demonstrations with executable code invocations and authentic interpreter feed- back, yielding a compact and efficient instruction dataset for SFT, as shown in Figure [
pipeline that transforms pure-text long
Stage Agentic Trajectory Generation via Code Injection. We assemble large-scale cor- pus from public mathematical reasoning sources (i.e., AM-Thinking, Guha al. 2025)), which distill responses from DeepSeek DeepSeek-R1-0525 R1-0528. To prevent evaluation contam- ination, we apply n- n-gram overlap filtering against benchmark datasets (i.e, AIME24/25, HMMT25), yielding high-quality pure-text reasoning dataset D
et al., 20255
Direct Manual annotation of agent trajectories is both costly and susceptible to noise. To address this, we propose an efficient code-injection strategy that leverages a powerful teacher model (i.e., DeepSeek- ParSSeek-V3(Deeeee V(eeppSSee-AAA et al.) (2025)), guided by carefully crafted prompts presented in Ap- pendix A.6.2 Each extensive Chain-of-Thought (CoT) sequence Tiest Drext systematically partitioned into multiple segments. Subsequently, each segment undergoes transformation via the injection function Finjects which substitutes computationally intensive reasoning steps Seale with ex- ecutable code blocks and their corresponding exccution outputs:
aagent Funeea( (Tex)); where Finject
Tiext + (Tiex with Scale = (c, oai)))
where denotes the injected code segment, Scale represents the replaced computational step, and Osim the sauulae-i execution result. This injection targets complex operations (exponential computations, matrix manipulations, equation solving) while preserving elementary calculations in
3

---

Published as a conference paper at ICLR 2026
textual form to maintain the model's understanding of tool invocation rationale and prevent over- dependence. Code blocks are delimited by <code>> </code>> tags, with execution results enclosed
in <interpreter>> </interpreter>> tags referring to (Feng et al. 20256).
Stage 2: Multi-Faceted Quality Refinement. Automatically synthesized trajectories can contain formatting issues, code defects, and logical inconsistencies. We apply four complementary proce- dures to ensure high quality and effectiveness:
(i) Format consistency correction: We employ regeneration for complex cases to enforce strict adherence to the <oode>-<interpeetee structural compliance.
normalization and teacher model
(ii) Code executability verification: Each embedded code snippet is executed within controlled sandbox environment. For any failures, we initiate a bounded resampling loop generate equivalent but executable alternatives. If execution remains unsuccessful within a predefined compute budget, the block is reverted its original textual step Seale to preserve logical soundness.
(iii) Environmental feedback alignment: Simulated outputs Osim from the teacher are systemati- cally replaced with ground-truth execution results Orcal E(c), where denotes the interpreter environment. A dedicated verifier model (i.e., Qwen3-32B) is employed to perform this judgment, guided by specific judge-promy detailed in Appendix [A.6.3] then assesses contextual consistency. Incoherent samples are removed or downgraded to text-only variants to maintain narrative integrity.
(iv) Tool-usage rationality assessment: Heuristic constraints on code complexity metries (i.e., line count, abstract syntax tree depth) are enforced to eliminate instances of unnecessary code invocation, thereby reinforcing necessity-awar tool utilization patterns.
Seff-Correction Capability Injection. Beyond correct tool invocation, a robust agent need also recover from erroneous tool feedback. We sample trajectories that were excluded during refinement due to execution failures, and for each failed program Crail with error output Oeror {Craa), we prompt the teacher model to generate a structured self-correction trace (diagnose the error - repair the code re-execute -> continue reasoning). The detailed prompt can be found in Appendix(A.6.] A small fraction of these negative-to-positive corrections is injected to strengthen debugging robust- ness. The final instruction DSFT combines validated, tool-augmented trajectories with diagnostic correction traces and serves as the foundation for SFT.
2.3 AGENTIC REINFORCEMENT LEARNING
We present agentic reinforcement learning (RL) framework that advances code- onde-integrated rea- soning capabilities beyond supervised fine-tuning (SFT). This stage pursues two objectives: (i) to quantify the incremental gains of RL over an SFT baseline, and (ii) to elucidate how RL reshapes tool-usage strategies under interleaved natural language generation and program execution. Addi- tionally, the detailed construction of the RL data is described in Appendix A.5.2
2.3.1 AGENT-SPECIRS REINFORCEMENT LEARNING
Our framework employs Group Relative Policy Optimization (GRPO)(Shao et al., 2024) as the core optimization algorithm, which obviates critic models while enhancing training efficiency through group-wise trajectory sampling and reward normalization described in Appendix A.3 We em- ployed multi-stage RL training, as detailed in Section 3.5 Following DAPO (Yu et al. 2025), we incorporate dynamic sampling, asymmetric gradient clipping, token-level loss computation, and KL divergence removal. We introduce three system innovations tailored for code-integrated agents:
Agentic trajectories with interleaved code execution. During rollout, trajectories are constructed loop (See Appendix A.2). yielding hybrid traces com- posed of chain-of-thought reasoning, inline code snippets, and real-time interpreter feedback. Tool invocations are bounded by per-instance cap T, enabling fine-grained control over agent- environment interactions and promoting sample efficiency.
through catpttt--ppuue--eett--t--eeeee
Loss Masking for Policy Gradient Updates. To focus learning on the agent's decision-making process, the advantage signal is applied exclusively to tokens within <think> and <code>> seg- ments. Tokens generated by the environment, specifically within <interpreter>>, masked during
4

---

Published as a conference paper at ICLR 2026
Agent Worker Pool
Agent Loop
Agjer
Actor (Felicy LLM)
Model Response
Anolyze + Dacids (hool coll?)
Fortall Length
Loop Continues
longh??
Yes
Agentic Trijectory
Escoodi vax tcel co
&
and
Append History
Tcol
Execute sondboved)
Agantic Optinizer
Request-Leve Async Scheduler
Deceding
Early-stop stop Partial Processing
Speed
Tedl Latency 1
Mox Length
Figure 2: The diagram of agentic reinforcement learning. depicts the structure and workflow of our agentic reinforcement learning system with core functions including Agent Loop, Asyn- chronous Scheduler, and Partial Rollout, along with key performance improvement. Based on the Asynchronous Scheduler, the Agent Loop continues rumning by default. It will stop early only when conditions are met: either the content length exceeds the max length (i.e., 32k) or the number of tool calls exceeds the maximum constraint.
optimization, ensuring that gradient updates are driven by the agent's own actions rather than deter- ministic environmental responses.
Adaptive Batch Construction with Filtering and Backfilling. For each problem instance, we sample G trajectories. Batches are filtered exclude problems where trajectories yield either uniformly correct or uniformly incorrect answers, which offer limited learning signal. To maintain consistent batch sizes, we backfill by randomly sampling additional filtered instances from the same pool, thus avoiding inefficient resampling loops while preserving distributional diversity.
2.3.2 REWARD DESIGN
Our reward function integrates answer correctness with tool-usage efficiency. The accuracy component Race provides binary feedback based on mathematical equivalence, validated via the math_verify library:
1, if isseequiaalet a),
Race
0, otherwise,
where à denotes the predicted answer and represents ground truth. Conditioned on correctness, the tool-usage reward R incentivizes efficient computational resource utilization:
R
min Rauxx a + 3 Newde) if Neode > 0,
where a represents the base tool-usage reward, / scales with invocation count, and Rmaxx caps the maximum tool-usage reward. The composite reward function becomes:
Rioaa
Race + I(Bsse 1) Rioo--
2.4 SCALABLE AGENTIC RL INFRASTRUCTURE
Agentic Reinforcement Learning for complex mathematical reasoning poses significant infrastruc- tural challenges. Empirical analysis reveals that, during RL training with the temperature set to 1.0, complex problems yield trajectories averaging 24k tokens and involve approximately 27 tool invocations. This combination of long-context generation and high-frequency external interactions produces heterogeneous computational workloads. Traditional synchronous batch rollouts exhibit substantial inefficiencies due synchronization overhead and resource underutilization.
5

---

Published as a conference paper at ICLR 2026
To address these challenges, design high- pereh-eerfrmmmee, scalable training system tailored to Agentic RL. Through an asynchronous decoupled architecture, an Agentic Partial Rollout algorithm, and prefix-aware load balancing, the system mitigates performance bottlenecks induced by long- tail effects and concurrent tool invocations, achieving ~ 5x improvement in end-to-end training throughput, shown in Figure 2
2.4.1 DECOUPLED AND ASYNCHRONOUS SYSTEM ARCHITECTURE
The architecture is founded the principle of decoupling GPU-intensive model inference from EPU/IO-intensive agent logic and environment interactions.
Distributed code execution sandbox cluster. We deploy distributed cluster of isolated worker pods to serve concurrent tool invocations at scale. This design offloads CPU-bound code execution from the training loop while enabling dynamic load distribution. Parallelization reduces tool-call latency from 175 to 1.2 s and removes inference blocking, substantially improving GPU utilization.
Request-Level Asynchronous Rollout Scheduling. Static batch-synchronous processing is placed with a coroutine-driven, request-level asynchronous scheduler. Each trajectory rollout is treated as independent long-running request, with the inference engine (server) and agents (clients) fully decoupled via asynchronous communication. When requests suspend for tool invoca- tions, the inference engine immediately processes other ready requests. This fine- mne-grained scheduling eliminates head-of- of-line blocking and maximizes GPU parallelism across heterogeneous workloads.
2.4.2 AGENTIC PARTIAL ROLLOUT
Agentic RL suffers long-tail latency from both sequence length and tool-invocation counts. We introduce an Agentic Partial Rollout mechanism that decomposes each trajectory into budget- limited segments:
T= (1) @ 7 (2) .... 1.....r D.... ((N),
where 2 denotes sequence concatenation. Each segment is constrained by a maximum generation length Tey and maximum number of tool invocations Tsey-
At each training iteration, the scheduler samples from an unfinished pool and a set new tasks P, generating segment per task. Segment generation terminates when: (i) an EOS token is pro- duced; (ii) segment length reaches Leegi (iii) tool invocations reach Tagi or (iv) cumulative trajectory metrics reach global limits Lgbobal T glohal. This segmentation prevents individual trajectories from monopolizing resources and smooths computational load, yielding a 2.2-2.5x speedup. Algorithm outlines the procedure.
2.4.3 PREFIX-AWARE WEIGHTED LOAD BALANCING
Partial rollouts alleviate long-tail latency but introduce requests with long prefixes, increasing KV- cache memory and prefill cost. Therefore, We design a Prefix-Awart Weighted Load Balancing strategy that assigns dynamic weights based on prefix length and routes requests to the least-loaded inference engine.
Each request R, with prefix length Lj receives weight
L
Wj
+ Whases
Truas
where Lonse (i.e., 16k tokens) normalizes length and Whese quantifies prefill overhead. For M engines with loads Wa. new request R, is routed to
SM S1....,
k* arg min Wk. and W +- WR- + W.y k))
To maximize KV-cache reuse, we implement sticky sessions via -tr-Leastt-Reeetly-Ueeeee caching, ensuring consecutive segments from the same trajectory preferentially route the same engine, thereby avoiding redundant context transfer and recomputation. This combination of dy- namic weighting and sache-affinity scheduling maintains load balance under heterogeneous traffic patterns while maximizing system throughput.

---

Published as a conference paper at ICLR 2026
3
EXPERIMENTS
3.1 SUPERVISED FINE-TUNING (SFT) DATA CONSTRUCTION
Stage Foundational Data Curation and Filtering. We aggregate raw corpus from multi- ple public math reasoning datasets (i.e., AM-Thinking, OpenThoughts, and AceReason). After problem-level deduplication, we apply N-gram (N = 4) and MinHash LSH algorithms eliminate overlaps with all evaluation sets, including AIME24, AIME25 and HMMT25 with 0.6 similarity threshold. To further prevent data leakage, we compute semantic similarities between training data al.) (024a). filtering out the top-5 most similar
and evaluation sets using gte-large IZoang samples. We then annotate each problem with difficulty scores from O to 10 using Qwen3-30B, retaining data with scores above 5, which yields 392k samples. Finally, using DeepSeck-R1-OS25 to generate solutions and removing instances with incorrect answers, we obtain 346k data.
Stage 2: Tool-Augmented Data Synthesis. We first decompose the problem-solving process into discrete reasoning segments, each with a fixed length of 3k tokens and perform tool-augmented synthesis for each segment using the Prompt presented in Appendix AA.6. via DeepSeek-V3-0324, During synthesis, we filter out samples with synthesis format error, tool execution failures, or incor- rect final answer, yielding 302k high-fidelity, tool-augmented solution trajectories.
Stage 3: Self-Correction Data Generation. To incorporate self-correction mechanisms, we sample 30k instances from trajectories with unsuccessful code execution and leverage the Self-correction Prompt presented in Appendix [ to guide DeepSeek- V3-0322 in generating correction pro- cesses, producing 14k valid self-correction trajectories.
Through this comprehensive pipeline, we construct a 316k tool-augmented synthetic training set with an average of 8.3 tool calls and an average sequence length of 16.9 16.9K tokens per sample.
3.2 REINFORCEMENT LEARNING (RL) DATA CONSTRUCTION
For RL. collect problems from multiple public high-quality RL datasets (i.e., DeepScaler, Skywork-OR1, Retool, POLARIS). We apply the same deduplication strategy as for SFT data, en- suring no overlap with evaluation sets through N-gram (N=4), MinHash LSH,, and semantic similar- ity computation with 0.6 similarity threshold. To identify challenging problems, we use AgentMath- 8B-SFT to perform 8 inference attempts on all data, filtering out problems that are solved correctly 8 times, culminating in a final set of 42k high-difficulty RL training data. This focuses training on hard instances that push the model's strategic capabilities and maximize potential gains from RL.
3.3 TRAINING SETTINGS
Base Models. The experiments utilize four base models from the Qwen3 series: Qwen3-1.7B. Base and Qwen3-8B- --BBsBe are pre-trained models without post-training: Qwen3-30B-AM 1-------------sttttte 2507 (Non-Thinking mode, 30B total parameters, 3B activated) and Owen3-235B-A2255 Innttuctt 2507 -Non-Thinking mode, 235B total parameters, 22B activated) are instruction-tun tuned Mixture-of- Experts (MoE) models without long chain-of-thought training.
SFT Training. We employ the Llama-Factory framework, training for 6 epochs with learning rates of 6 10-5 (1.7B, 8B, A3B models) and 2 10-5 (A22B model), using cosine decay with 10% warmup, batch size of 512, and maximum sequence length of 32k tokens.
RL Training. RL training is built on the verl 0.5. devo framework (SenennorrSSheng al. 2024). initializing from the best SFT checkpoint and using VLLM Wwon et al.) (023) as the inference engine, with a 128-node Sandbox cluster for large-scale code execution. We use a constant learning rate of 1 x 10 a batch size of 64, and temperature of 1.0, performing 8 rollouts per problem. Training progresses through three stages, dynamically adjusted to maintain length truncation and tool-call excess rates below 10%: maximum response length increases from 48k to 72k to 96k tokens, with corresponding tool invocation limits of 48, 72, and 96 calls, and partial rollout counts of 2, 3, and 4, ensuring each segment rollout remains within 24k tokens and 24 tool invocations. For reward design, we set a = 0.1 and 3 = 0.01, and constrain the maximum tool reward to Rmmxx to Appendix I Due computational constraints, AgentMath- -atthhhhhB------ 235B-A22B is trained solely via SFT. More details for data synthesis, SFT, RL, and evaluation are provided in Appendix A.5
(For details refer
7

---

Published as a conference paper at ICLR 2026
3.4 MAIN RESULTS
Table 1: Performance of AgentMath on AIME24/25, and HMMT25. Our model (highlighted in blue) is compared against other leading models, with accu- racy (avg@32) as the evaluation metric. Due to space limitations, use DS, QW2.5, and QM2.5 to de- note DeepSeek-RI. -L, Qwen2.5. and Qwen-2.5 2wee-2.55-Mtth, respectively. For more detailed and comprehensive performance table, refer to Table in the Appendix.
In this section, we comprehensively eval- uate AgentMath by comparing it against the advanced reasoning models on three challenging math competition benchmarks: AIME24, AIME25, and HMMT25. Results are presented in [ with more extensive model comparisons available in Appendix Table 4 To ensure robust evaluation, we perform 32 independent inference runs per test sample, using avg@32 as the pass @ metric. We use consistent configuration: 96K maximum sequence length, 96 maxi- mum tool calls, and code interpreter output is limited 1024 tokens, and we set 0.6 tem- perature, and 0.95 top-p.
Medel
Rase Model
Toal Use
ATME3
AIMESS
RMMIT1
Pepeisurr meddl
Open Molooloioo wire whhrods
56.7
Opennt ao-ooid
65.2
16.4
Upenal oi-
02.4
2.5 Gl
$2.5
A
91.5
58.0
17.5
Opea
X
67.3
86.3
$3.0
4.D Thanioer
$8.3
Frunter Moskia (1B
TiRL-1 5B
GMT 5-1.5B Buse /
26.7
36.7
Di-Diulll-o Owas 55
QMP 5-119 Bkso X
18.8
71.8
15.3
Owee:
Qwend- 1.1 Thankong
Qwaat-- 1.
52.0
23.3
The results show that AgentMath signifi- cantly outperforms existing tool-augmented and text-only frontier reasoning models across all three benchmarks at comparable parameter scales. Among small-scale models (1B 2B), AgentMath-11778 attains 59.6%, 48.1 1,, and 40.2% accuracy on AIME24, AIME25, and HMMT25 respec- tively, substantially surpassing both the tool-augmented CoRT-1.5B (43.1%, 30.2%, 20.1%) and the text-only OpenReasoning- 1.5B (55.5%, 45.6%, 31.5%). At the medium scale (7B ~ 8B), AgentMath-SBB achieves 89 8%, 84.7%, and 71.3%. signif- icantly outperforming the tool-augmented CIR-Qwen-NNTS-BB and text-only DS- 0528- 5228-Qwen3-8B (86.0%, 76.3%, 61.5%)
0W225:1 10225-1.---000 Ua--aamret
13.0
41.3
27.3
OpesRowening
Queet-11 Bose
40.2
Fronter Modkle (1I MB1)
TIRI 7B
QMP 5-78 Bost
43.3
50.0
GWW23-18 dass
V
46.7
30.0
21.5
Sempte UK
Bast
2677
GW2.5
51.9
dooe Ma QB
QMT 5-78 Bost
13.3
rus mantl Ow
QNT 5-7 Bas
X
Altzal bete-
DSS Dol-- Quen--1 Tis Theatll Quenn
53.9
48.3
29A
Sky ORI 778
X
10.3
35.7
Oeve 6th Thikkin
Quwen' --th Rour
417
Opeakcartize
QW..3-1B 1- lesinact Quead Bass
8
54.7
78.1
63.5
DS 4523 Qu
63.5
Foenlar Mosklo 368 - 128)
ZenoTR: 328
QC.5 555-328-ooe
56.7
533
750
STA
Owr 178
66.7
47.
APM-
CW Imtract
56.7
328 QW2.5 -istini
chta2s Aaes---aa--- 32B
504
52.3
13.5
Quead SuB-
A3B Aelelaaaad
Queas 30B-
A38- Bunc
72.9
61.3
43.0
Distii 11B
Que- 178
78.1
460
sTmL----mt 33I
TIS Detll Quien 1711
V
K1.7
611
45.
Ske
Disil 118
For larger-scale models (30B 32B),
Quecd S0B.
Thinkhy 2007
Owent 30B-
8727
85D
71A
AgentMath-30B-ASB reaches 90.6%,
Agon 30B.
30B A1t--berram--70
9065
96.4
738
Pronter Medell (-322))
86.4%, and 73.8%, exceeding the tool-
augmented 3TILL-3-TOOL-32B (81.7%.
A22l baa---a------
DS-0718
Deeppeck--V VS-Boae
Azzis Baa
R
79,2
19.5
55.4
44A
64.2%. 45.4%) and text-only Qwen3-
QweeS-208B
Thenking
Ghwad-22358
30B-A3B- 11B--A3--Thinkng-2225 Thinking-2507 (87.7%, 85.0% 74.3%).
2507
Qwens--2358
Deigg V.S
A
91.1
875
770
839
AgrntMan- -.T APT
Qward-22a0g
11.7
Notably, the Mixture-of offEExetts model SgentMath-33B-A23 with 3B active param-
eters and 30B total parameters outperforms most dense 30B models on AIME24 and AIME25, ap- proaching the performance of DS-671B-0528 This demonstrates that our approach achieves com- petitive performance with substantially larger models while maintaining computational efficiency.
At ultra-large scale (>32B), across the three benchmarks, surpassing DS-67 IB-0288 (91.4%, 87.5%. 77.0%) and achieving performance on par with AgentMatth33B-AA33 outperforms OpenAl-o3-ming and -oaude--ppu----hhinkii on three bench- marks and competitive with OpenAl-o3 and Gemini-2.5-Pro. Furthermore, AgentMath-23511 A22B-SFT exceeds OpenAl-o3, approaching OpenAl-o4-mini and Gemini-2.5-PPo Notably, due to computational constraints, AgemtMath-235i--A228 is trained solely via SFT.
Wtw.Maaahh-25B-222B---- achieves 93.4%, 90.8%, and 81.7%
Compared to proprietary models,
These results validate the effectiveness of our tool-augmented data synthesis method and large- scale reinforcement learning training strategy, yielding consistent improvements in math reasoning capabilities across diverse model scales. We provide the case study of AgentMath in Appendix A.8
8

---

Published as a conference paper at ICLR 2026
AREEL
Trak B9
Tron mip
Tran
(a) Acc on AIME24 and AIME25
(b) Training response length
(c) Training length clip ratio
(d) Code ratio
Tool call counts.
(i) Code clip ratio
Figure 3: Evolution of key metrics during multi-stage RL training: (a-c) accuracy AIME24 and AIME25, response length, length clip ratio; (d-f) code ratio, tool call counts, code clip ratio.
3.5 COGNITION ANALYSIS
Table 2:
Performance comparison between AgentMath and Text-Based Model in SFT and RL stages.
Tool-Augmented Synthetic Data vs. Text- Based Data. assess the effectiveness of tool- augmented synthetic data method, we conduct experiments addressing two core questions: the comparative advantage of tool-augmented syn- thetic data over text-based data SFT, and the performance
Models
AIME24
AIME25
Text- Hit--aaed-SFF--OO
57.1%
49.2%
AgentMath-SFTT-20
60.5%
53.3%
impact of tool augmentation and efficiency during RL. We employ Qwen3- 8B-Base the backbone model and identi- cal 20k data. As shown Table 2 in SFT stage, AgentMath-SFT achieves accuracies of 60.5%
Text-Based-RL AgentMath-RL
68.7%
57.5%
76.2%
67.5%
on AIME24 and 53.3% on AIME25, surpassing the text-based baseline by 3.4% and 4.1%, validat- ing our method of converting computation-inte intensivv steps into executable code. The benefits are further amplified in RL: detailed Figure 4 and Table 2 AgentMath- requires only 4400 steps to reach 76.2% (AIME24) and 67.5% (AIME25), a 4.0x efficiency improvement over the ~ 1600 steps needed by the Text-Based-SFFT model achieve inferior results (68.7% and 57. 57.5%). Notably, it matches the Text-Based model's final performance in just 100-200 steps. Additionally, inference efficiency improves substantially, as indicated Figure with sequence lengths reduced by 44k tokens (~14%) and slower length growth, attributable to precise code execution replacing verbose manual calculations. Collectively, AgentMath demonstrates superior accuracy, training ficiency, and inference scalability, confirming the power of interleaving natural language reasoning with computational tools. See Appendix A.7.1 for more details.
Multi Stage RL Training. Following the supervised fine-tum mie-tuning phase, observed that the model frequently generated responses exceeding 32k tokens for complex mathematical problems, with the most challenging instances surpassing 64k tokens. To effectively balance training efficiency with model capacity, we developed an adaptive, multi-stage RL strategy that progressively unlocks the model's potential by dynamically expanding the sequence length and tool-call budget. This process is triggered automatically when truncation rates for either response length or tool usage exceed 10%, incrementally increasing the context length from 48k to 72k (at step 120) and finally to 96k (at step 280), while the tool-call limit expands from 48 to 72 (step 140) and then to 96 (step 320), illus- trated in Figure 3e and 3f The training progression, detailed in Figure 3 reveals significant trends: generated trajectory average lengths increased from 24k to 30k (Figure 3b). tool average invocation frequency rose from 27 to 31 calls per problem (Figure Be). and code utilization improved markedly from 70% to 95% (Figure 3d). indicating enhanced proficiency in multi-step reasoning. Conse- quently, accuracy on the AIME24 benchmark rose from 78.4% to 89.8% (+1 (+11.4%) and on AIME25 from 72.2% to 84.7% (+12.5%) (Figure 3a with consistent improvements following each capac-

---

Published as a conference paper at ICLR 2026
AgentMaith
AgentMaath
Aucurars
AINESI
400
Traie
1090
Train Akga
1.30
1990
(a) AIME24 Accuracy
(b) ATME25 Accuracy
Figure 4: Performance Comparison of AgentMath vs. Text-Based Model in the RL phase AIME24/25. Both models were initialized from their best SFT checkpoint trained on 20k data.
ity expansion. Notably, the model exhibited emergent code self-correction capabilities shown in Appendix AA8.2 Figure g These results, along with the performance AgentMath with differ- ent backbones detailed in Table confirm the efficacy of strategy. The experiments establish three key insights: (1) expanded capacity is crucial for facilitating deeper reasoning chains; (2) The composite reward effectively guides the model's tool-call decisions; and (3) the stable training der extreme configurations (96k tokens, 96 tool calls) underscores the robustness of the AgentMath framework and its asynchronous training infrastructure. See Appendix A.7.2 for more details.
Synthetic Data Refinement and Scaling Law. As detailed in Table 3 we conduct systematic evaluation of AgentMath's 's data synthesis pipeline, revealing that progres- sive multi-dimensional refinement is criti- cal for performance. The initial unrefined synthetic data yielded suboptimal results
Table 3: Performance improvements on AIME24/25
through progressive refinement steps.
Models Refinement Steps
AIME24
AIME25
Initial Unrefined C1-Synthetic Data (20k)
35,3%
25.7%
Format consistency correction
47.4%
40.1%
Code executability verification
52.8%
44.8%
(AIME24: 35.3%; AIME25: 25.7%), pri- marily due to formatting inconsistencies and non-executable code. By systematically ap- plying refinements,includng format consistency, code executability verification, and environment feedback alignment, performance substantially improved to 58.6% on AIME24 and 50.8% on AIME25. The subsequent integration of self-correction mechanism, combined with supervised fine-tuning using selective feedback masking guided by code execution results, culminated in final accuracies of 60.5% on AIME24 and 53.3% on AIME25, underscoring the necessity of each refine- ment stage. Furthermore, scaling the coll-augmented dataset from 2k to 300k (Figure yielded sig- nificant performance gains, improving accuracy from 27.2% to 78.4% on AIME24 and from 21.1 to 72.2% on AIME25. This combination of rigorous quality control and effective data scaling ef- fectively mitigates data scarcity tool-augmented mathematical reasoning, establishing robust foundation for nigh-peeffrmancc reasoning agents. Further details are provided in Appendix A.7.3
Environmental feedback alignment
56.3%
48.3%
Self-correction capability injection
58.6%
50.8%
SFT selective feedback masking
60.5%
$3.3%
Owing to space constraints, a comprehensive analysis of the AgentMath framework's training eff- ciency and the impact of the partial rollout segment count is deferred to Appendix A.7.4
4 CONCLUSION
This paper introduces AgentMath, tool-augmented agent framework that seamlessly integrates lan- guage model reasoning with the precision of code interpreters to tackle complex mathematical prob- lems. Extensive evaluations show that AgentMath achieves state-of-the the-art performance on chal- lenging mathematical competition benchmarks, including AIME24, AIME25, and HMMT25, Re- markably, AgentMath-30B-A4 with only 3B active parameters achieves 90.6%, 86.4%, and 78.9% accuracy, outperforming OpenAI-o3-me and Claude- Claude-Opus-4.0 itive with OpenAl-o3, Gemini-2.. 2-5---2..-PPo and DeepSeck-RI R16611 Furthermore, the essential role automated aol--uementeed data synthesis and a scalable asynchronous training infrastructure in enabling effective and efficient agentic learning for mathematical reasoning.
while remaining compet- work highlights
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Major Key Project of Pengcheng Laboratory (Grant No. PCL2025A12) and the CiE-Tencent Ph.D. Student Research Incentive Program (Tencent Hunyuan Special Fund). Sincere thanks to Dr. Wenfeng Deng, Team Leader Wang and other colleagues for their valuable guidance and insightful suggestions throughout this research.
REPRODUCIBILIT STATEMENT
To ensure the reproducibility of AgentMath, we provide comprehensive details on algorithms, data synthesis procedures, Agent RL training, and experimental configurations throughout the main paper and appendices. The data synthesis pipeline, multi-dimensional quality refinement mechanisms, and the design of synthesis and judge prompts are detailed in Sections 2. and [.2] and Appendices A.2
and A.6 The RL training framework and implementation details are presented in Section 2.3 and
Appendix A.3 Optimization strategies for RI. training efficiency and the design of Algorithm are
described in Section 2.4 and Appendix A.44 We systematically evaluate the individual contributions
of coll-augmented data synthesis, stage RL training, scaling laws of synthetic data, and multi- dimensional quality refinement through ablation experiments in Section 3.5 and Appendix AA.7 Case analyses provided in Appendix [A.8] Detailed experimental settings, including SFT and RL training data, training strategies, evaluation settings and main results, are documented in Sections We hope these will help researchers better understand and
3.1 3.1. 3.2 3.3 3.4ann and Appendix A..5
reproduce our work.
REFERENCES
team. a-m
-m- m team/ a- models 1s
June 2025.
URL https://github. com/
Mayank Agarwal, Ibrahim Abdelaziz, Kinjal Basu, Merve Unuvar, Luis A Lastras, Yara Rizk, and Pavan Kapanipathi. Toolrm: Outcome reward models for tool-calling large language models. arXiv preprint (rXXi:2509.1966 2025.
Wasi Uddin Ahmad, Somshubra Majumdar, Aleksander Ficek, Sean Narenthiran, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Vahid Noroozi, and Boris Ginsburg. OpenCoddeeaasoning-JI A Simple Test Time Scaling Approach via Self-Critique, 2025. URL https:t/arxxxv..oo
abs/ / 221//557..077 09075
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models. URL hhttps://kkunpp: github.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310. 10631, 2023.
Fei Bai, Yingqian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Towards effective code-integrated reasoning, 2025. URL
https://arxiy: org. /ttp::///axxivoor//aa///s///// 2505. 24480
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovié, and Martin Vechev. Math- arena: Evaluating Ilms on uncontaminated math competitions, February 2025. URL https:
//matharena.at1
Qikai Chang, Zhenrong Zhang, Pengfei Hu, Jiefeng Ma, Yicheng Pan, Jianshu Zhang, Jun Du, Quan Liu, and Jianqing Gao. Thor: Tool-integrated hierarchical optimization via for mathematical reasoning. arXiv preprint urXiv:2509. 13761, 2025.
Yupeng Chang, Yi Chang, and Yuan Wu. Ba-lora: Bias-alleviating low-rank adaptation to mitigate catastrophic inheritance in large language models. arXiv preprint arXiv:2408. 04556, 2024.
11

---

Published as a conference paper at ICLR 2026
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, 2023. URL https:
//arxiv.org/aaaaa.vv. abs hbtg///b////aa /2w...211.... 12588
Yang Chen, Zhuolin Yang. Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Acereason-nemotive Advancing math and code reasoning through rein- forcement learning. arXiv preprint arXiv:2505. 16400, 2025a.
Zhipeng Chen, Yinggian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang. and Ji-Rong Wen. An empiri- cal study on eliciting and improving r1 rl-kike reasoning models. arXiv preprint arXiv:2503.0454 2025b.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang. Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, al. An empirical study on eliciting and improving rl-like reasoning models. arXiv preprint mwxiv:2503.0454 2025c.
Claude. Claude 3.7 sonnet.
2025.
URL fttps://www.anthrop...
cLaude-3-7-sonee
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An. Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. htLh:tt::: /ttp://aavvvvvvaa/ org/ abs/ 2501 12948
Guanting Dong, Hangyu Mao, Kai Ma. Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang. Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.1984. 2025.
12

---

Published as a conference paper at ICLR 2026
Dong Du, Shulin Liu, Tao Yang, Shaohua Chen, and Yang Li. Ulorl: An ultra-long output reinforce- ment learning approach for advancing large language models' reasoning abilities. arXiv preprint arXiv:2507.1976, 2025a.
Weihua Du, Pranjal Aggarwal, Sean Welleck, and Yiming Yang. Agentic-rl: Distilled dual-strategy reasoning. arXiv preprint arXiv25507.05077, 2025b.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https:
// //github. www. www/hugg./occcttccc huggi ng face/fac/aaee open-rl
Tiantian Fan, Lingjun Liu, Yu Yue, Jiaze Chen, Chengyi Wang, Qiying Yu, Chi Zhang, Zhiqi Lin, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Bole Ma, Mofan Zhang. Gaohong Liu, Ru Zhang, Haotian Zhou, Cong Xie, Ruidong Zhu, Zhi Zhang, Xin Liu, Mingxuan Wang, Lin Yan, and Yonghui Wu. Truncated proximal policy optimization, 2025. URL httttt://///xiiiit org/
abs/ 2506. 15050
Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, al. A comprehensive survey self-evolving agents: A new paradigm bridging foundation models and lifelong agentic systems. arXiv preprint arXiv:25004 25000:2000007077,, 2025.
Chen Feng, Ziquan Liu, Zhuo Zhi, Ilija Bogunovic, Carsten Gerner-Beuerle and Miguel R. D. Rodrigues. PROSAC: Provably safe certification for machine learning models under adversarial attacks. In The 39th Annual AAAI Conference on Artificial Intelligence (AAAI) [Oral]. 2 2025a. doi: 10.1669/aaii.v39/1 3393.333000
Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Beunree and Miguel D. Ro- drigues. Noisy but valid: Robust statistical evaluation of LLMs with imperfect judges. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. URL.
net m
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms, 2025b. URL htt.:ttss//axiv.ooott http:/orggbbb////..o// 11536
Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, and Xiao-Ming Wu. Tasl: Continual dialog state tracking via task skill localization and consolidation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1266-1278 2024.
Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S Yu, Xu Chu, and Xiao-Ming Wu. Recurrent knowledge identification and fusion for language model continual learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 27396-22 27741,,,,,3, 2025c.
Zihao Feng, Xiaoxue Wang, Bowen Wu, Hailong Cao, Tiejun Zhao, Qun Yu, and Baoxun Wang. Toolsample: Dual dynamic sampling methods with curriculum learning for rl-based tool learning. arXiv preprint arXiv:2509. 2025d.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang. Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. URL https://arxiv.orrt abs/ 20s/25055.4200
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turn: Unlocking long-horizon agentic search with large-scale asynchronous rl, 2025a. URL http://///aaww..o///a/b/////////
07976
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-atte aided language models, 2023, URL https: /tt.xiii.o.or///////........
abs/ 2bs/2211. 2211 2211.1044 10435
13

---

Published as a conference paper at ICLR 2026
Wei Gao, Yuheng Zhao, Dakai An, Tianyuan Wu, Lunxi Cao, Shaopan Xiong, Ju Huang, Weixun Wang, Siran Yang, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng, and Wei Wang. Roll- packer: Mitigating long-tail tail rollouts for fast, synchronous rl post-training, 2025b. URLhttpa
//arxiv. wrg.iviorrraooooooo abs/ / 2509.2..2 21009
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, arXiv preprint arXiv:2305.11738, 2023a.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A iool-integrtted reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.00. 77452, 2023b.
Etash Guha, Ryan Marten, Sedrick Kch, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Erie Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang. Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL /tttps://arrtt wwt.
abs 25066 04178
Ruohao Guo, Afshin Oroojlooy, Roshan Sridhar, Miguel Ballesteros, Alan Ritter, and Dan Roth. Tree-based dialogue reinforced policy optimization for red-teaming attacks. arXiv preprint D:Xi::2510.02286, 2025.
Chuzhan Hao, Wenfeng Feng, Yuewei Zhang, and Hao Wang. Dynasearcher: Dynamic knowl- edge graph augmented search agent via multi-reward reinforcement learning. arXiv preprint arXiv:2507.17365 2025.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang. Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report. arXiv preprint 2rXiv:2505.22312. 2025a.
Tao He. Hao Li, Jingchang Chen, Runxuan Liu, Yixin Cao, Lizi Liao, Zihao Zheng, Zheng Chu, Jiafeng Liang, Ming Liu, et al. Breaking the reasoning barrier survey on Ilm complex reasoning through the lens of self-evolution. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 7377-7417, 2025b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2003.0380 2021.
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405. 11143, 2024.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner
open source approach to scaling up reinforcement learning on the base model, 2025. htt /ttp:::///a/vvvi/a org/ abs//503.... 2tt.://////.03222222332 24290
Yunjie Ji, Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, and Xiangang Li. AM-Thinking-vl: Advancing the frontier of reasoning at 32B scale. https:
www.iv..rr/aaa//50..... 08311 2025. arXiv:2505.00 10000500811111 [cs.CL].
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training 1ims to reason and leverage search engines with reinforcement learning, 2025a. URL htl.httss:. http//ts//////teee httpppp://aaxii...// .ogg 250.//503.055 09516
14

---

Published as a conference paper at ICLR 2026
Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, and Jing Bai. Reveal: Self-evolving code agents via iterative generation-verfiiatioo arXiv preprint arXiv:2506. 2025b.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of 1lms, 2024. URL https:://aas:
org/abs/ 2ttaaabb/22...22.2222 18629
Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, Keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang, Xiang Wang, Junyang Lin, et al. Cort: Code-integrated reasoning within thinking. arXiv preprint arXhv:2506.09820 2025a.
Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools. arXiv preprint arXiv:2503. 25..20053...,5, 2025b.
Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, Hongxuan Lu, Tianrui Qin, Chenghao Zhu, Yi Yao, Shuying Fan, Xiaowan Li, Tiannan Wang, Pai Liu, King Zhu, He Zhu, Dingfeng Shi, Piao- hong Wang, Yeyi Guan, Xiangru Tang, Minghao Liu, Yuchen Bleanor Jiang. Jian Yang, Jiaheng Liu, Ge Zhang, and Wangchunshu Zhou. Chain-of-agenn End-to agent foundation models via multi-agent distillation and agentic rl, 2025c. URL htttt:://aaaat /ttpss//axxvv.. http:/awwivv....aa.aaaa abggg// 13167]
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhancee large reasoning models, 2025d. URL https:
abs 2255100536
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl, 2025e. URL https:
//arxiv.orr//ab//25000 23383
Yang Li, Zhichen Dong, Yuhan Sun, Weixun Wang, Shaopan Xiong, Yijia Luo, Jiashun Liu, Han Lu, Jiamang Wang, Wenbo Su, Bo Zheng, and Junchi Yan. Attention illuminates Ilm reasoning: The preplan-and-anchor rhythm enables fine-grained policy optimization, 2025f. URL https:
waaxiv.org/abs//51111 13554
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- QuannQuan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Heng Lin and Zhongwen Xu. a5Xiv:2508. 19201, 2025.
Understanding tool-integrated reasoning.
arXiv preprint
Jiashun Liu, Johan Obando-Ceron Han Lu, Yancheng He, Weixun Wang, Wenbo Su, Bo Zheng, Pablo Samuel Castro, Aaron Courville, and Ling Pan. Asymmetric proximal policy optimization: mini-critics boost I1m reasoning, 2025a. UtLpttts:://axxi wtg/ abb wttp://ggaaaa....a/a/////// httpaa/b.///11.....3 01656
Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, and Manling Li. Let's try again: Eliciting multi-turn reasoning in language models via simplistic feedback. 2nd Al for Math Workshop@ ICML. 2025, 2025b.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language mod- els. arXiv, 2025c. URL WRR.https://aat org abs. 24864
15

---

Published as a conference paper at ICLR 2026
Wei Liu, Siya Qi. Xinyu Wang. Chen Qian, Yali Du, and Yulan He. Nover: Incentive training for language models via verifier-fre free reinforcement learning, arXiv preprint arXiv:2505.66022 2025d.
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, Shengyi Huang, Johan Obando-Ceron, Siran Yang, Jiamang Wang, Wenbo Su, and Bo Zheng. Part i: Tricks or traps? deep dive into for Ilm reasoning, 2025e.
URL https: org/ abs/ / 220/22588.00 08221
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances Neural Information Processing Systems, 37:54463-54455 2024.
Han Lu, Zichen Liu, Shaopan Xiong, Yancheng He, Wei Gao, Yanan Wu, Weixun Wang, Jiashun Liu, Yang Li, Haizhou Zhao, Ju Huang, Siran Yang, Xiaoyang Li, Yijia Luo, Zihe Liu, Ling Pan, Junchi Yan, Wei Wang, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. Part ii: Roll flash accelerating rlvr and agentic training with asynchrony. 2025a. URL httts:://axxii//
abs/ 0tss/2500.1133
Keer Lu, Chong Chen, Bin Cui, Huang Leng, and Wentao Zhang. Pilotrl: Training language model agents via global planning-guiee guided progressive reinforcement learning. arXiv preprint arXiv:2508.00344 2025b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint (xXi:233880055.8 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jian-Guang Lou, Shifeng Chen, Yan- song Tang, and Weizhu Chen. Wizardarena: Post-training large language models via simulated offline chatbot arena. Advances in Neural Information Processing Systems, 37:111544-11150, 2024a.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yan- song Tang, and Weizhu Chen. Arena learning: Build data flywheel for Ilms post-training via simulated chatbot arena. arXiv preprint wxXiv2247.110621 2024b.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta. Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepScaleR: Surpassing -P-Preveww with 1.5b model by scaling RL. /ttpss///peattyyraaoooo-....... not not.oooooooo..iiooooo.oooooo
2025a. Notion blog; accessed 2025-09-24.
Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna Qiu, and Yuqing Yang. Agent lightning: Train any ai agents with reinforcement learning. arXiv preprint arXiv:2508.00 03680, 2025b.
Minh-
Minh-Thang Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, et al. Towards robust mathematical reasoning. In Proceedings of the 2025 Conference Empirical Methods in Natural Language 2025.
Processing, pp.
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced mueetunnng, 2024. (RRLtttt:///arae org/ abs/2401.088 08967
Xinji Mai, Haotian Xu, Zhong-Zhi Li, Xing W. Weinong Wang, Jian Hu, Yingying Zhang, and Wen- qiang Zhang. Agent rl scaling law: Agent with spontaneous code execution for mathematical problem solving, 2025. URL https: aaaii.iiiv...... org/
Zhiyu Mei, Wei Fu. Kaiwei Li. Guangju Wang, Huanchen Zhang, and Yi Wu. Realhf: Opti- mized rlhf training for large language models through parameter reallocation. arXiv preprint arXiv:2406. 14088, 2024.
16

---

Published as a conference paper at ICLR 2026
Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. Real: Efficient rlhf training of large language models with parameter reallocation. In Proceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025. misys.org, 2025.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-e math- ematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504. 16891, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL
a 0atpp//51.11.......1.111 19393
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, William Saunders, and John *et al.* Schulman. WebGPT: Browser-assisted Question-Answerigg with Human Feedback. arXiv preprini arXiv:21 2022.
Xuan-Phi Nguyen. Shrey Pandit, Revanth Gangi Reddy, Austin Xu, Silvio Savarese, Caiming Xiong. and Shafiq Joty. Sfr-deepresearch: Towards effective reinforcement learning for autonomously reasoning single agents. arXiv preprini (.Xii:25099062883, 2025.
OpenAl. Learning to reason with Ilms. http://ww...cccoaoeeeeeemeeeeeer g-to-reuxon-wihh-llmss 2024.
OpenAl. Introducing openai 03 and o4-mini, 2025. URL httpp:://ppeaa..ccoo Introducing-03 and- 04 -mini/
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Eliz- abeth Proehl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas. Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart An- drin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lemnon, Jason Wei, Jean Harb, Jerry Twore. Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kon- draciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk. Olivia Watkins. Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan
17

---

Published as a conference paper at ICLR 2026
Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agar- wal, Santiago Hernandez. Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Pupay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Tay- lor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai ol system card, 2024.
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston, Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116617-116677 2024.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, (6:53328-537744 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. 2023. URL /Lttlt://aaaatttt.. org/ httpg//abs/23300 047761
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
ByteDance Seed, :, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang. Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xi- aochen Zuo, Chi Zhang, Ruofei Zhu, Zhecheng An, Zhihao Bai, Yu Bao, Xingyan Bin, Jiangjie Chen, Feng Chen, Hongmin Chen, Riwei Chen, Liangqiang Chen, Zixin Chen, Jinsong Chen, Siyan Chen, Kaiyuan Chen, Zhi Chen, Jin Chen, Jiecao Chen. Jinxin Chi, Weinan Dai, Ning Dai, Jiahui Dai, Shihan Dou, Yantao Du, Zhengyin Du, Jianhui Duan, Chen Dun, Ting-Han Fan, Ji- azhan Feng, Junda Feng. Ziyuan Feng, Yuwei Fu, Wenqi Fu, Hanjie Fu, Hao Ge, Hongyi Guo, Mingji Han, Li Han, Wenhao Hao, Xintong Hao, Qianyu He, Jerry He, Feng He, Wen Heng, Zehua Hong, Qi Hou, Liang Hu, Shengding Hu, Nan Hu, Kai Hua, Qi Huang, Ziyue Huang, Hongzhi Huang, Zihao Huang, Ting Huang, Wenhao Huang, Wei Jia, Bin Jia, Xiaoying Jia, Yuhua Jiang, Haobin Jiang, Ziheng Jiang, Kaihua Jiang, Chengquan Jiang. Jianpeng Jiao, Xiao- ran Jin, Xing Jin, Xunhao Lai, Zheng Li, Xiang Li, Liyi Li, Hongkai Li. Zheng Li, Shengxian Wan, Ya Wang, Yunshui Li, Chenggang Li, Niuniu Li, Siyu Li, Xi Li, Xiao Li, Aoyan Li, Yuntao Li, Nianning Liang, Xinnian Liang. Haibin Lin, Weijian Lin, Ye Lin, Zhicheng Liu, Guanlin Liu, Guanlin Liu, Chenxiao Liu, Yan Liu, Gaohong Liu, Juncai Liu, Chundian Liu, Deyi Liu, Kaibo Liu, Siyao Liu, Qi Liu, Yongfei Liu, Kang Liu, Gan Liu, Boyi Liu, Rui Long, Weiqiang Lou, Chenwei Lou, Xiang Luo, Yao Luo, Caiping Lv, Heyang Lv, Bole Ma, Qianli Ma, Hongzhi Ma, Yiyuan Ma, Jin Ma, Wenehang Ma, Tingting Ma, Chen Mao, Qiyang Min, Zhe Nan, Guanghan Ning, Jinxiang Ou, Haojie Pan, Renming Pang. Yanghua Peng. Tao Peng, Lihua Qian, Lihua Qian, Mu Qiao, Meng Qu, Cheng Ren, Hongbin Ren, Yong Shan. Wei Shen, Ke Shen, Kai Shen, Guangming Sheng, Jinlong Shi, Wenlei Shi, Guang Shi, Shuai Shuai Cao, Yuxin Song, Zuquan Song, Jing Su, Yifan Sun. Tao Sun, Zewei Sun. Borui Wan, Zihan Wang. Xiaohui Wang. Xi Wang, Shuguang Wang, Jun Wang, Qinlong Wang. Chenyuan Wang, Shuai Wang. Zihan Wang. Chang- bao Wang, Jiaqiang Wang, Shihang Wang, Xuwu Wang, Zaiyuan Wang. Yuxuan Wang. Wenqi Wang, Taiqing Wang, Chengzhi Wei, Houmin Wei, Ziyun Wei, Shufa Wei, Zheng Wu, Yonghui Wu, Yangjun Wu, Bohong Wu, Shuang Wu, Jingqiao Wu, Ning Wu, Shuangzhi Wu, Jianmin Wu, Chenguang Xi, Fan Xia, Yuqiao Xian, Liang Xiang, Boren Xiang, Bowen Xiao, Zhen Xiao, Xia Xiao, Yongsheng Xiao, Chao Xin, Shulin Xin, Yuwen Xiong, Jingjing Xu, Ziwen Xu, Chenyin Xu, Jiayi Xu, Yifan Xu, Wei Xu, Yufei Xu, Shikun Xu, Shipeng Yan, Shen Yan. Qingping Yang, Xi Yang, Tianhao Yang, Yuehang Yang, Yuan Yang, Ximing Yang, Zeyu Yang, Guang Yang, Yifan Yang, Xuesong Yao, Bairen Yi, Fan Yin, Jianian Yin, Ziqiang Ying, Xiangyu Yu,
18

---

Published as a conference paper at ICLR 2026
Hongli Yu, Song Yu, Menghan Yu, Huan Yu, Siyu Yuan, Jun Yuan, Yutao Zeng, Tianyang Zhan, Zheng Zhang, Yun Zhang, Mofan Zhang, Wang Zhang. Ru Zhang, Zhi Zhang, Tianqi Zhang, Xinyi Zhang, Zhexi Zhang. Sijun Zhang, Wenqiang Zhang, Xiangxiang Zhang, Yongtao Zhang, Yuyu Zhang, Ge 7hang, He Zhang, Yue Zhang, Renjie Zheng, Ningxin Zheng, Zhuolin Zhengg Yaowei Zheng. Chen Zheng, Xiaoyun Zhi, Wanjun Zhong, Cheng Zhong, Zheng Zhong, Bao- quan Zhong, Xun Zhou, Na Zhou, Huan Zhou, Hang Zhu, Defa Zhu, Wenjia Zhu, and Lei Zuo. Scedl.5-thinking: Advancing superb reasoning models with reinforcement learning, 2025. URL bittp:://axiiv org/ abs / 2504. 13914
Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, al. rstar2-agent: Agentic reasoning technical report. arXiv preprint wrwiv:2508..007770... 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0388 2024.
Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. Nemo-aligner: Scalable toolkit for efficient model alignment, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 2409.19256 2024.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for IIms via reinforcement learning. arXiv preprint arXiv:2505.01441, 2025.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optiltin can be more effective than scaling model parameters, 2024. URL https::/arxiv.cr
abs/2408 03314
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in Ilms via reinforcement learning, 2025.
org/ abs / 2503. 05592
ByteDance Seed Team. Seed-oss open-source models. Bytteaanc-SBed//seeeee
https://githul. com/
2025a.
Gemini Team, Rohan Anil, Sebastian Borgeaud, -aan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint ///1v:23/2.//105, 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu. Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuchai Pan, Y. Charles, Yang Li. Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1lms, 2025a. URL https: httpp::s//arxvv... /wtpiv.vvvv./////.......... abs/ 12599
NovaSky Team. Sky-tl: Fully open-source reasoning model with ol o1-preview performance in $450 budget. http::/moooskyiiiiiiliuoooooooso githuu..oo/oosss/kkylll 2025b. Accessed: 2025-01-09.
Qwen Team. Qwen3 technical report, 2025c. URL http:/ss//aarc...og/////////3/3/
19

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025d. URL htttps://qwenmm.co github. wttpi///////gaeeee happp///www332000 (
Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, et al. Hunyuan-turbon Advanc- ing large language models through mamba-transformer synergy and adaptive chain-of-thought. arXiv preprini arXiv:2505./5431, 2025b.
TinyRI Team. Superdistillation achieves near-rl performance with just 5 URL https:// hteegaaag..cee.ccoooooo//////010
Yuchen Tian, Ruiyuan Huang, Xuanwu Wang, Jing Ma, Zengfeng Huang, Ziyang Luo, Hongzhan Lin, Da Zheng, and Lun Du. Evolprover: Advancing automated theorem proving by evolving formalized problems via symmetry and difficulty. arXiv preprint arXiv:2510.00732, 2025.
Haozhe Wang. Long Li, Chao Qu, Fengming Zhu, Weidi Xu. Wei Chu, and Fangzhen Lin. Learning autonomous code integration for math language models, 2025a. URL hottttt/////aiii org//
abs/2502. 00691
Hongru Wang, Yujia Qin, Yankai Lin, Jeff Z. Pan, and Kam-Fai Wong. Empowering large lan- guage models: Tool learning for real-world interaction. In Proceedings of the 47th Interna- tional ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '24, pp. 2983-2986, New York, NY, USA, 2024, Association for Computing Machinery. ISBN 1145/
9798400704314. doi: 10.114573626677..661338. URL
Hongru Wang, Cheng Qian, Wanjun Zhong. Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Ote: Optimal tool calls via reinforcement learning- arXiv preprint arXiv:2504. 2025b.
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi. Renrui Zhang. Linqi Song. Mingjic Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in 1lms for en- hanced mathematical reasoning, 2023a. hRtthttss///axiii/
abs 2210./2311.o. 03731
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.0893: 2023b.
Weixun Wang, Shaopan Xiong. Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large- scale learning: An efficient and user-friendly scaling library. arXiv preprint arXiv:2506.00 06 122, 2025c.
Zihan Wang, Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, 2025d. URL htttt::t///att http///axvvv/..////vv//t../a..tv org/ / 2504 20073
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022. URL. /ttppss:/pproceeedngg... neurisss.eeerppp File/ /ttp://0005111444411111111111232
Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, and Li Du. Autotir: Autonomous tools integrated reasoning via reinforcement learning. arXiv preprint arXiv:2507.2 208866 2025.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-RI: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025. URL https: //ap::://iiee htt.gtttbbcccc.p... http:taw..cooo//oo/////////ieaao
GitHub repository.
20

---

Published as a conference paper at ICLR 2026
Xiaobao Wu. Sailing by the stars: A survey reward models and learning strategies for learning from rewards. arXiv preprint arXiv:2505.02696, 2025a.
Xiaobao Wu, Sailing ai by the stars: survey of learning from rewards in post-training and test-time scaling of large language models. arXiv preprint arXiv:2500.006666 02686 2025b.
xAI. Grok. https://x.... ai/| 2023.
ai/ Large language model.
LLM-Core-Team Xiaomi. Mimo: Unlocking the reasoning potential of language model - from
pretraining to posttraining, 2025. URL SttL:tt:: http///axii... aww.aaaiivvvr........ wwtp:///ggaaaaaa/aoooo/.. 2305. 07608
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yer Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning, 2024. URLhttps:: //arrvv....iv/// owg.orgg//oo htrg//bs/// www.///4002 20000/2445.000 00451
Zhenghai Xue, Longtao Zheng. Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint S5li::2509002479, 2025.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei 7hang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-matt technical report: Toward mathematical ex- pert model via self-improvement, 2024b. / httpt::://// /ttp://aaavvv...a/aaa org/ abs /4099. 12122
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: deliberate problem solving with large language models. In Pro- ceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya- Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open- source Ilm reinforcement learning system at scale, 2025. URL httpt: https:://axivv.oo org/ abs/ 2503. 14476]
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, 2023. URL
httts::/arxiv. wwtp://orggbb///eee 05653
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Re- inforcing multi-tur reasoning in Ilm agents via turn-level credit assignment. arXiv preprint 1rXiv:2505.11821 2025a.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint Xriv::200..1 18892, 2025b.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for lms:: A survey. arXiv preprint 27X.v22509.02547, 2025a.
21

---

Published as a conference paper at ICLR 2026
Kechi Zhang, Ge Li. Jia Li, Huangzhao Zhang, Jingjing Xu, Hao Zhu, Lecheng Wang, Yihong Dong, Jing Mai, Bin Gu, et al. Computational thinking reasoning in large language models. 2025b.
arXiv preprint arXiv:2506.02060
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. Nemoorron-tessaacch-toooool!!!! Tool-using language models with rein- forced reasoning. arXiv preprint (rxxi:2555500022. 2025c.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407. 196699997..9,, 2024a.
Yao Zhang, Hongxiao Zhang, Jiacheng Zhang, Jingcheng Zhao, Rui Yan, Xiaoqing Liu, Jiahuan Wang, Min Zhang, Houfeng Wang, and Zhengguang Guo. rStar-Math: Small LLMs master math reasoning with self-evolved deep thinking. arXiv preprint urXiv:2403.01707, 2024b.
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Arik. Chain of agents: Large language models collaborating on long-context tasks. Advances Neural Information Processing Systems, 07:112208-132222 2024c.
Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, et al. Memento: Fine-tuning I1m agents without sie-ttnnng Ilms. Preprint, 2025a.
Yixiao Zhou, Ziyu Zhao, Dongzhou Cheng, Zhiliang Wu, Jie Gui, Yi Yang, Fei Wu, Yu Cheng, and Hehe Fan. Dropping experts, recombining neurons: Retraining-free pruning for sparse mixture- of-experts 1lms. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp.
15169-15186, 2025b.
Yixiao Zhou, Yang Li, Dongzhou Cheng, Hehe Fan, and Yu Cheng. Look inward to explore out- ward: Learning temperature policy from Ilm internal states via hierarchical rl. arXiv preprint arXiv:2602.13035, 2026.
Zilin Zhu, Chengxing Xie, Xin Ly, and slime Contributors. slime: An IIm post-training framework for scaling. https:://i thttb.. com/THUDM//l1 2025. GitHub repository. Corre- sponding author: Xin Lv.
22

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 RELATED WORK
Mathematical Reasoning in LLMs.
Large Language Models (LLMs) have made remarkable
progress mathematical reasoning Wei et al. 2022 Yao et al.,
2023 Luong et al. 2024 OpenAl
et al. 2024 TTem et al. 2025a DeepSeek-A1 et al. 2025 xAI
2023] Claude _ 2 Team et al.
2023 Yang al. 2024a bi He al. 2025b Fang et al. 2025
Zhang et al. [202a Wu 2025b a)
The introduction of Chain-of Thought (CoT) x Wei et al.
2022;
Yao et al. 2023) prompting enabled
models to decompose complex problems into intermediate reasoning steps, substantially enhanc-
ing their problem-solving capabilities. Subsequently, research has shifted from singular focus on model scaling towards optimizing the reasoning process itself([nell et al.] (024). This paradigm shift has spurred the development of Large Reasoning Models (LRMs) trained with advanced meth- ods like Reinforcement Learning (Schulman et al.] 2017. Li et al. 2024 Shao et al. 2024). Direct
Learning(Schulman
Preference Optimization (Raailov et al. 2023 Lai et al. 2024| Pang et al. [2024]] Luo et al.
2024b
Feng et al., 2024 2025c Zhou et al. all 2025b] 2026), and Monte Carlo Tree Search (Xie et al. 2024
Wang et al. 2023b Guo et al. 2025). -tate-of-the-art models such as OpenAl's ol and DeepSeek-
R [T(OpenAl et al. 2024: DeepSeek-AL al. 2025] Team [025d] Team et al. 2023 2025a b) exhibit human-like cognitive planning on long-chain reasoning tasks, pushing the frontiers of mathematical performance. Despite these advances, reasoning purely within natural language is constrained by inherent limitations: complex arithmetic and symbolic manipulations are prone error, and self- correction is often inefficient. It may cause hallucinations and risks(Feng et al. 2025a 2026). These shortcomings fundamentally limit their accuracy and efficiency on competition mathematical problems.
Tool-Augmented LLM Reasoning. Tool-augmented reasoning has emerged as a promising solu-
tion to the limitations of text-basee approaches([ et al. 2025e Zhou et al. 2025a Lin & Xu 2025
Zhang et al.
2025b Luo et al. 2024a] Jin et al. 2025b; Liu et al. 2025b; Luo et al. 2023
Azer-
bayev et al. 2023 Yu et al. 2023). ). Program-of-Thoree Thought (PoT) Chen et al. (2023); Gao et al. (2023);
Yue et al. ( Jin et al. (2255a); Wang et al. (2024) 20 Cobbe et al. [202 by; Hendrycks et al.
(2021) ; Lightman al. (2023); Tian et al. (2025); (Shao et al. 2024) OpenAl (2024 (222)) y Wang
ct al. 2025byg Chang et al. (2024); Gou et al. (2023a) et al. (2024 Qu et al. (2025); [Song et al.
(oo (2025 Li et al. (2025d); Schick al. (2023) :: /hhnn al. (2024b cp pioneered delegating compu- tational steps to external code interpreters, enhancing numerical accuracy. ToRA ToRA(Gou et al. 2023b)
subsequently developed code-integrated reasoning frameworks tailored for mathematical problems, demonstrating the efficacy of specialized tools in complex computations. START(Li et al. 2025b) generates tool-augmented trajectories via prompt engineering, though random code insertion often yields inefficient utilization. STILL3 et al. (0255) relies on prompt-based data construction, and CoRT {Li et al. employs high-quality human annotations but faces scalability constraints. These approaches predominantly depend on supervised fine-tuning, preventing models from learn- ing debugging strategies from execution failures or adaptively mastering tool invocation timing and methods. While Retool(Feng et al. 2025b) combines data rewriting with reinforcement learning for optimization, improvements over existing LRMs (i.e, DeepSeck-RI-DD RI-Disiilll--ue Solll--wen--3200 (DeppSeeeeck Al et al. 2025)) remain marginal. Current tool-augmented methods thus face three critical chal- lenges: scarcity of high-quality data, inadequate policy learning, and inefficient training on long sequences. AgentMath mitigates these limitations by automating tool-augmented data synthesis and employing reinforcement learning to enable autonomous exploration of optimal tool-use strategies, including tool invocation and code self-correction.
Agentic Reinforcement Learning. Reinforcement learning (RL) offers a powerful framework for cultivating autonomous. decision-making agents from LLMs (Dong et al.
2025 Zhang et al. [025cc]
Xue et al. 2025| Li et al.) 2025c] [Singh et al.] [025] [Shang et al. 2025
Ng Nguyen et al. [025] Wei
et al.] Luo et al.) 2025b] Hao et al. | 2025 Agarwal et al. [202 [u et al. [022bb] Zeng et a
2025a] Liu et al. (202dd] Du et a [225b]] Chang et al. 2025 Feng et al. [202dd]]]] Li et al. 2025d
Song et al. 20255 : Nakano et al. [0022] Mai et al. 2025; Wang et al. 2025a] Du et al. 2025a Gao
et al., 2025a Mei et al. 2025| Wang et al. 2025c Liu al. 2025a Li et al. 20251; Lu et al.
2025a; Gao et al. [2025b] Liu et al. 2025e Hu et al. 2024 Shen et al. 2024; Wang et al. 2025d
Zhu et al. [2025] Mei et al. [2024] Fan et al. 2025). In information retrieval. models like Search-RI
and R1-Searcher (Li et al. 2025d] Song et al. 2025) Nakano et al. 2022 have demonstrated how
outcome-based rewards successfully guide agents to query search engines. In mathematical
23

---

Published as a conference paper at ICLR 2026
reasoning, recent work has explored RL for emergent tool use. ToRL (Li et al. 2025e) utilizes RL to train an agent to operate a code interpreter without predefined patters,, while concurrent work on the scaling laws of agentic RL has revealed that simple, outcome- based rewards often foster greater exploration and policy innovation than complex process-based rewards rewards Lightman et al. 2023 Wang al., 2025b) leverages RI. to teach models strategic tool
et al. 2023b). Similarly, ReTool(Feng call, significantly outperforming SFT basclines and uncovering cognitive patterns in code-invocation decisions. Nevertheless, existing RL methods face a critical bottleneck when applied to competition- level mathematics. These problems can generate exceptionally long reasoning chains (i.e., 64k tokens) with dense tool interactions (e.g.. 64 calls), scale that overwhelms conventional batch- synchronous training architectures. AgentMath alleviates this scalability challenge through suite of technical innovations, including request-level asynchronous rollout scheduling, agentic partial rollouts, and prefix-aware weighted load balancing. These techniques enable efficient RI. training on ultra-long sequences with massive tool usage, boosting training throughput by 4-5x and paving the way for developing more sophisticated and scalable mathematical reasoning agents.
A.2 PROBLEM FORMULATION AND INTERACTION PROTOCOL.
A.2.1 PROBLEM FORMULATION
Tool-augmented mathematical reasoning is formalized Markov Decision Process (MDP), wherein the LLM-based policy agent iteratively interacts with a sandboxed execution environment. Given problem statement P, the policy me generates trajectories comprising interleaved reasoning segments and executable code blocks, while the environment & deterministically executes submitted code and returns corresponding outputs.
The objective to construct an optimal trajectory r* = {(21, o1),..., (zr, or)), where (2,, oe) pair at timestep t. The state transition dynamics characterized by:
denotes the
To(. 8t);
8t 11)
0 [E(a),
if a EC
2 if
7 = T U ((),
where 8, represents the current state comprising the problem and interaction history, and T denote the code and thought action spaces respectively, and E(c,) returns the execution result of code block er. The interaction terminates upon generation of a terminal token or exhaustion of the computa-
tional budget.
A.2.2 STRUCTURED INTERACTION PROTOCOL
The implementation employs a structured markup protocol to delineate reasoning and tool invoca- tion boundaries. Natural language reasoning is encapsulated within <think> <..<//hink>> tags, ex- ecutable code is delimited by <code>> //ooe>> tags, and execution feedback is injected through <interpreter>> <..//iierpreee>>>>>>>>.......>>. tags.
The ceneration-executian cycle operates through bidirectional information exchange: upon comple- tion of <code>> segment, generation is suspended while the extracted code undergoes execution in the sandboxed environment. The resulting output, whether successful execution, enror message, or timeout notification, is subsequently incorporated into the context as an <'inerrpeter>>> segment. This feedback mechanism enables adaptive strategy refinement, wherein the model conditions its subsequent generation on execution outcomes to perform error correction, strategy adjustment, or continued reasoning. Such fine-grained interaction traces provide rich supervision signals amenable to reinforcement learning optimization.
A.2.3 SUPERVISED -UNE-TUNING WITH SELECTIVE FEEDBACK MASKING
During supervised fine-tume toninuung on DSFT, the model must learn to generate reasoning and code while avoiding memorization of deterministic interpreter outputs. Consequently, tool outputs masked during loss computation. For a training sample = (2), .1,..., 27, 0r). where 2, represents model- generated segments and o denotes external feedback, the standard autoregressive loss is expressed
24

---

Published as a conference paper at ICLR 2026
Algorithm Agentic Reinforcement Learning with Partial Rollouts
1: Initialize: Unfinished pool U 0; Experience buffer B 0: Global limits Legoball T globbal'
Segment limits Lxeg. Teeg-
2: for each training iteration k 1,2,...da do
3: Lisks_to_rroeess + Sample(P UU)
4: new uew_ssgmmetts f- Rollout -o-pocess,,, Luog, Tsog)
Asynchronous generation
5: finished trajectories t- 0. next ueetininfd pool + for each trajectory in new new.segments do
6:
7:
if r ends with EOS or length(T) 2 Lglobul or tools(T) 2 gilobal then Add T to finished trajectories else
8:
9:
10:
Add T to next.unfinished..po
11:
U + next_unfinished.po
Add finished trajectors to UpdatcPolicy(B)
T
LSFT = - E
log (2s P,Tet).
A masking function I(() is introduced to identify tokens originating from <interpreter>> segments, yielding the modified loss:
T ΣΣ (1-1((2) I(21,:)) log To(2L | 2t,k))
LsFT-msked (0)=
where žt,k denotes the k-th token segment 2,, and T(2+,)) = and only 21, resides within <interpreter>> tags. This selective masking ensures that gradient updates originate exclusively from model-generated reasoning and code, thereby shaping intrinsic reasoning capabilities and decision- making processes while treating external feedback non-trainable contextual information.
A.3 GROUP RELATIVE POLICY OPTIMIZATION
We employs Group Relative Policy Optimization (GRPO) as the core optimization algorithm. GRPO eliminates the requirement for value function approximation, thereby reducing computational com- plexity through group-wise trajectory sampling and intra-group reward normalization. The opti- mization objective is formulated as:
I Σ
min (0),(0) A., clip(r.,(0), 1 1 1, 1))
(B) EE
PND,
G
(T.)
where (0) denotes the importance sampling ratio. The advantage estimate Å is computed
through within-group normalization:
4 R(T.) PR
OR
with PR and dR representing the group mean and standard deviation, respectively, and å serving as a numerical stability constant. Following recent advances DAPO, the KI. divergence penalty is omitted to facilitate exploration, while the Clip-Higher strategy is adopted to enhance learning of high-entropy, low-probayility tokens critical for complex reasoning tasks.
A.4
AGENTIC PARTIAL ROLLOUT ALGORITHM
A.5
EXPERIMENTAL DETAILS
This section describes the training data construction, model training, and evaluation settings.
25

---

Published as a conference paper at ICLR 2026
A.5.1 SUPERVISED FINE-TUNING (SFT) DATA CONSTRUCTION
The supervised fine-tuning (SFT) data construction pipeline consists of three phases.
Stage 1: Foundational Data Curation and Filtering. We aggregate a raw corpus from mul- tiple public math reasoning datasets (i.e., AM-Thinking, OpenThoughts, and AceReason). After problem-level deduplication, we apply N-gram (N = 4) and MinHash LSH algorithms to eliminate overlaps with all evaluation sets, including AIME24, AIME25 and HMMT25 with 0.6 similarity threshold. To further prevent data leakage, we compute semantic similarities between training data and evaluation sets using gte-large moden et al., 2024a) filtering out the top-5 most similar samples. We then annotate each problem with difficulty scores from 0 to 10 using Qwen3-30B, retaining data with scores above 5, which yields 392k samples. Finally, using DeepSeek-R R1-0528 to generate solutions and removing instances with incorrect answers, obtain 346k high-quality data.
Stage 2: Tool-Augmented Data Synthesis. We firstly decompose the problem-solving process into discrete reasoning segments and perform tool-augmented synthesis for each segment using the Prompt presented in Appendix [A.622] via DeepSeek-V3-0324, V3-0324. During synthesis, we filter out samples with synthesis format error, tool execution failures, or incorrect final answer, yielding 302k high-fidelity cool-augmented solution trajectories.
Stage 3: Self-Correction Data Generation. To incorporate self-correction mechanisms, sam- ple 30k instances from trajectories with unsuccessful code execution and leverage the Self-correction Prompt presented in Appendix A.6. to guide DeepSeek V3-0324 in generating correction pro- cesses, producing 14k valid celf-correction trajectories.
Through this comprehensive pipeline, construct a 316k tool-augmented synthetic training set with an average of 8.3 tool calls and an average sequence length of 16.9K tokens per sample.
A.5.2 REINFORCEMENT LEARNING (RL) DATA CONSTRUCTION
For RL, we collect problems from multiple public high-quality RL datasets (i.e., DeepScaler, Skywork-OR1, Retool, POLARIS). We apply the same deduplication strategy as for SFT data, en- suring no overlap with evaluation sets through N-gram (N=4), MinHash LSH, and semantic similar- ity computation with 0.6 similarity threshold. To identify challenging problems, we use AgentMath- 8B-SFT to perform 8 inference attempts on all data, filtering out problems that are solved correctly 8 times, culminating in a final set of 42k high-diffiully RL training data. This focuses training hard instances that push the model's strategic capabilities and maximize potential gains from RI.
A.5.3 TRAINING SETTINGS
Base Models. Our experiments utilize four base models from the Qwen3 series: Qwen3-1.7BB Base and Qwen3-8B- -Bsee are pre-trained models without post-training; Qwen3-30B-A3B ---B----Insttt 2507 (Non-Thinken MoinThin mode, 30B total parameters, 3B activated) and Qwen3-235B-A22 A---B--AAB----nttue 2507 (Non-Thinking mode, 235B total parameters, 22B activated) are instruction-tuned Mixture-of- Experts (MoE) models without long thain-of-thought training.
SFT Training. We employ the Llama-Factory framework, training for 6 epochs with learning rates of 6 10-5 (1.7B, 8B, A3B models) and 2 10-5 (A22B model), using cosine decay with 10% warmup, batch size of 512, and maximum sequence length of 32k tokens.
RL Training. Our RL training is built on the verl 0.5.0.dev0 framework (hhng et al. 2024), ini- tializing from the best SFT checkpoint and using VLLM Kwon et al. 2023) as the inference engine, with a 128-node Sandbox cluster for large-scale code execution. We use constant learning rate of x 10 6, a batch size of 64, and a temperature of 1.0, performing 8 rollouts per problem. Training progresses through three stages, dynamically adjusted to maintain length truncation and tool-call excess rates below 10%: maximum response length increases from 48k to 72k to 96k tokens, with corresponding tool invocation limits of 48, 72, and 96 calls, and partial rollout counts of 2, and 4,
26

---

Published as a conference paper at ICLR 2026
ensuring each segment rollout remains within 24k tokens and 24 tool invocations. Due to computa- tional constraints,
235B-A22B trained solely via supervised fine-tuning (SFT).
A.5.4 EVALUATION SETTINGS
Benchmarks. We primarily evaluate on AIME24, AIME25, and HMMT25. These challenging U.S. high school math competitions feature problems in algebra, number theory, combinatorics, and geometry, providing a robust test of advanced mathematical modeling, multi-step logical reasoning, and strategic problem-solving.
Evaluation Metries. To ensure robust evaluation, we perform 32 independent inference runs per test sample, using avg @32 as the pass@1 metric.
Inference Parameters. We use consistent configuration: maximum sequence length 96K tokens, maximum tool calls 96, code interpreter output limit 1024 tokens, temperature 0.6, and top-p = 0.95.
Answer Extraction and Validation. We extract final answers from \boxed ) markers in model responses and employ Math- Verify library for exact comparison with ground truth answer, determin- ing correctness only when verification returns True.
A.5.5 DETAIL RESULTS
27

---

Published as conference paper at ICLR 2026
Table 4: Performance comparison (avg @2 accuracy) of AgentMath against state-of-the-art models on AIME24, AIME25, and HMMT25 benchmarks. Evaluation follows DeepSeek-RI framework (temperature=0.6, (op-p=0.95). AgentMath models (highlighted in blue) achieve superior results across all scales, with the 30B variant competitive against 67 models.
Model
Tool Use AIME24
HMMT22
Proprictary models
monn---ot-minss//ulll OpenAA 2025
98.7
99.5
Grok-4-w/oolss// XAI 212
9%.8
OpenAL-o3-whhe monnl-oo--whonntteee www.oot 2015
95.2
98.4
OpenAI-o4 OpenAt
Gemini :ean:::: 2023
92.0
$8.0
$2.5
OpenAt-o3 Opency 2036)
916
88.9
77.5
Seed- Bocd-L..-thmmknngg
2025
90.3
OpenAl-o3 Opsiit1
53.0
Chade-Opu--1.0 (-00.. Mints mide
2025
83.0
72.0
58.3
Grok-3-Bet 3-Beaa m11 M773
83.9
77.3
Kimi-k1.5 Team 23160
77.5
Frontier Models
ToRI-11.9((..ttal 20225
(war2.5-Math-11-. (-5th-1.5B-Bas
26.7
DeepSeck- -wwe DespSeekcAl er 2025
Owen2.5-Math- Math-1158. 1.5B-BBsee
28.8
21.8
15.3
DemSSaleR-1. 1.B.R---1.PPPiiiiie (Luo
DeepSeck RR--istill-Qeerr 1,58
X
40.0
30.0
CoRF-1.5H ( (l. 20250
DeepScek--R R1-Disril--uuee Diirill-ueen-1.3
Nemoutron-Keseer Kas.oii---ninnn--0 Owen-1
2025c
DeepSack-RI-Diie Swenlll-Qeee-ll
21.7
Qwen3-1.78 Thinking Tsam 2025c
Owen3 77B--Bse
52.0
35.3
23.3
OpenThinker2--1 1.5B G.5t er 2025
Owen2 5- LB Instract
52.0
41.7
27.3
OpenReasoning-
Necionoo
0025
Qwen2 5- 5--Inssrcctt
55.5
45.6
31.5
AgentMath-
Owen3-1.BB
Fronter Models (7B 8B)
Qwen2.5-7B Mosli--nstruut-111 TIR Yong et 202
Qwen2. Mathh- 7B-Base
20.0
26.7
Curus-2-PRMME Cui 2015
Owen 2.5 Math: Base
26.7
SimpkRL-2ere
Qwen Math-ahh- 7B-
ToRL-7B 0l 10730
Qwen2 Qwen2.5-Math- M-athb-BBB 7B-Bao-- BBBo
43.3
30.0
ZonoTTR-775 M01--7555--- Man
2025
Qwen-2 5-78-BaB--
46.7
30.11
22.5
SimpleTIR. 711 XHl
2125
Qwen2 S-B--os
50.5
$0.9
29.7
ALM-7B(Li
Qwen2
/Sur-MalFQwen--i Zining
Qwen2 Math-7B-- Base
53.3
-I--See-----iirilll Dipel uisppee Al 2005
Qwen2.5 Math-7B-BBse
35.0
39.7
OpenKI-Distill- (Fac 2022
Qwen2.5 S. Math 7B. 7B.-BBBBBB
57.7
39.7
25.7
Light- (e-a-RR1-B--pa
er 2025
DeepSoek- Ri-iskil----------- Disstill-Quen--
Distill-
59.1
44.3
27.6
CiR-Qwen--NT---DDDD. NDD
Qwen3-SB
AReal-boha-71/ Pu b025)
DeepSeek: R1 RI-Distill-Quee Saaulll-Qeen-ie
61.9
48.3
29.4
Slywork-ORL- 71 su253
DeepSeck RI Distilled Qwen-BB
70.2
$4.6
35.7
POLARIS-7B 1L---7B-11
Bee--B--Peeiae Prefiee
DeepSeck- RI Distill Qwen- 7B
72.6
AucReasoe-Neannii -7BBl:
DeepSeck-Ri Seeill-Qeen-Te
OpenMaill-enmee N1 Morner oi
Qwen2.5 5-Math-BB
74.8
61.2
Qwen3-8Bt Thinking Tentu 2013
Thinking
Qwen3-BB- Rose
76.0
67.3
44.7
MiMa-7B(Xisomii Dizg
MiMo-7B- BBMse
70.2
35.7
OpenReiseming-Ne
Qwan2.5-77 5-78-mmsktt
DeepSeck-R L-----------22-0000 R1-0628-Qwen3-00 ai Dss
Owen3-8B-Bee
AgentMath-B8
Qwes3 8B-BnBe Buse
89.8
84.7
71.3
Fronterr Models (308 32B)
Sky-TI-32B. Team
20255
Qwen2 5 Instract
43.3
Open Reasoner Qwerste
0wen2.5.32B
DAPO-Qwen-220 Owen- Yu 2025
Owen2.5-322 Bass
50.0
32.1
s1-32B Muespichar me
Owen2.5.32B. Instrrtt
56.7
50.0
37.0
ZeroTTk
0wen-2.5.30 32B Base
33.3
20.0
START-32B 2010
QwQ-32B
Qwen2 (rrm2.5-32BB-aam Insttrcct
66.7
59.8
ReToull322 onii 20255
Qwen2.5-328 5-528-Insttect
67.0
49.3
/Sur2-Agen Suur2-AAensowwen.... aoroana.sss1 Srann dl 2025
Qwen2.5 5-225---s-ttn-
69.4
$7.3
ReTool-R1-320 1oo--RR--32--isiiii
DeepSeck-R1 Di--isilll-Qeees
DeepSeck-R Ri-isal----- (hiiiill--we-----
A. el al
Qwen2 5-32-BBss
72.9
33.0
Qwen3 508-5-5 308-A-8--Inse 1Bt.A111--ssmmt225000
Temm 217A71
(NNil minking)
Qwes3-30B: A18--A3B-Bos
72.9
61.3
43.0
Saan-R--3BBWenn et 2025
Qwen2.5 328-sssttt
76.6
64.6
CokT-32B
A0250
DeepSeek Ql-Distill- Saalll--en--222
76.7
67.1
Thny81133 Team
DeepSack-R1 Diw.keek-RI-issill-.e Quaill---ee-222
QwQ-32B(Team) 20.6
79.5
55.3
48.0
Qwen3-50B-xx1--1 10.-000--30--mppp Eniniii Team 2005c
Qwen3-300 30B A.-.AAB-BBus
80.4
70.9
$1.0
Qwen3-32B Thinking
20136
Qwen3 32B Base
81.4
72.9
STILL-3-TOOL-320 10BT0O0LL////CeEAA
DeopScok- RI-Distill-Quee Qwlll--wen322B
Skywori-OR1-117 Nireel eu2al
DeepSeek-IR Distill-- 1---sill-Quee-7222
Qwen-328
82.2
73.3
AM AMM-Thinkin--1 vi] v113211 Dr om 2014
Qwen 2.5-32B Base
85.3
74.4
AM-DoopSac--RL RR---- Di52:-Damills3
2025
Qwen 1--32BBBase
87.1
Qwen3-30B -0-3--0---38--TMM A38--Tiinkngg25500
BMen3-30B-A3B--
OpenReneening-Neee Nemtop--eenttoo3333 Anmid
Owwnn2.5332B-nmee
73.8
AgentMath- 308-
Qwen3 S0es3-308-A3BBllae A3B-IInsmmet25000
90.6
86.4
73.8
Prontier Models (>>32B)
Qwen3-2ASB-A2200 wit.--222---ntterr------ 2507% Mon-Thinkinng
Qwmn3-23BBAA28-
79.2
70.3
55.4
DeepSeek-R1 R1-67 DeegSeee
2005
DeegSock-V3.B
79.8
Qwon3-2358-A220 228-1 minkenz toam K0AA
Qwwn8--338BA422..
2222-Ros
85.7
81.5
62.5
DeepSteek-Rt R -66 B-0528 De CoSteeAeee of
2024
DeepSeck. V V
91.4
87.5
77.0
Seed-Ost Ost-36B--BBB-- Instrace Telmn
Seed-O5S. 36B-BBBBB
91.7
84.7
235B Thininkh
Dwan3-215B-A225
AgenMaath- 235B-A22B-SBF A2TB-SFT
Qwen3-2352 2am2-233B-A22B- 120---A222----tttt--111 1557
93.1
90.8
$1.7
28

---

Published as a conference paper at ICLR 2026
A.6 PROMPT
A.6.1
DATA SYNTHESIS FOR CODE SELF-CORRECTION PROMPT
Prompt 1: Data Synthesis For Code Self-correction Prompt
The following is Your Response based on User Instruction. But there was a code interpreter exccution error during the process. Please do the following: Please:
1. Based on the interpreter's 's failed execution output, identify the exact code segment that caused the error and explain the reason for the failure.
2. Immediately after interpreter's failed output, transition sentence such as: "Oops, the code above appears to be throwing an emor. need to fix this to ensure it runs successfully."
3. Correct the erroneous code to it runs successfully.
4. Continue the process from where you left off in your response, completing the remaining steps as planned.
5. Wrap the final output coutput><</utputte
User Instruction: (Input) Your Response:
<revised prapcingggrooos>>ooooup <hhnvised
A.6.2 TUGMENTEE DATA SYNTHESIS PROMPT
29

---

Published as a conference paper at ICLR 2026
Prompt 2: Tool-Augmented Data Synthesis Prompt
You are professional assistant with expertise in mathematics and Python programming. and multiple gold medalist in mathematics olympiads and programming competitions. You have the capability write code and the results will enhance the reasoning process. Task Objective
code interpreter for calculation. The code will be executed sandhox environment, and
Enhance the provided mathematical problem-solving process by replacing complex manual calculations with Python code and their execution results, while preserving the original reasoning logic and structure. Instructions
Identify Computational Steps for Code Replacement
Identify steps that would benefit from code execution, including:
Complex symbolic algebra: polynomial expansion, factorization, solving equations
Advanced calculus: differentiation, integration, evaluating definite integrals
Probability and combinatories: complex counting, probability distributions
Linear algebra: matrix operations, inversion, eigenvalue decomposition
Numerical computations: approximations, large number caleulations, geometric calculations
Any error-prone computationally intensive calculations
Important: Simple calculations (basic arithmetic like x should remain as text. Only use code for complex computations that require at least 5 lines of implementation.
2. Code Implementation Requirements Fach code snippet must:
Be complete and executable, including all necessary imports
Usc appropriate libraries (sympy, numpy, scipy, etc.)
Include clear variable definitions and comments
Explicitly use print() for all outputs
Demonstrate the computation process, not just final results
Contain least 5 lines of code
3. Integration Guidelines
Preserve the original reasoning flow; Keep all logical steps, explanations, and even failed attempts unchanged
Seamless integration: Code and results should naturally fit within the surrounding text
Context preservation: Maintain semantic coherence and logical consistency
No unnecessary changes: Do not modify, delete, or polish unrelated content
Formatting Requirements
Wrap each code snippet as follows:
<code>><
\n 'python' \n Mnnk/code
Follow immediately with execution results:
5. Input Format User Question: (question)
6. Output Format
Provide the enhanced thinking process wrapped in:
<revi seriiedd_tee thinkhinkiieee
7. Key Principles
1. Code Necessity: Use code only for complex calculations that warrant automation.
2. Minimal Changes: Modify only the computational steps, preserving all other text.
3. Complete Scripts: Each code block must be self-contained exccutable.
4. Accuracy: Execution results must match exactly what the code produces.
30

---

Published as a conference paper at ICLR 2026
A.6.3 CONSISTENCY JUDGMENT PROMPT
Prompt 3: Consistency Judgment Prompt
Role and Objective
are meticulous scientific analyst. Your mission to determine if and Text are substantively equivalent in their core results.
Focus strictly numerical values, mathematical expressions, and final conclusions. Ignore all differ- ences in wording, formatting, and sentence structure. Core Principles
1. Content Over Form: Prioritize the core message and data. Completely ignore stylistic choices like wording, formatting (bolding, spacing), and sentence construction.
2. Standardize Before Comparing: Before comparison, you must normalize all values to common standard. Convert percentages to decimals (50% - 5), unify units (1m + c00mm), and resolve scientific notation.
Compare Overlapping Information Only: one text contains information absent in the other, ignore the missing parts. Base your comparison solely on the data and claims present in both texts. Asymmetry is not basis for inequivalence.
Evaluation Criteria
1. Numerical Equivalence
Rule: Two numbers, A B, are equivalent their absolute difference than or equal to 1. after normalization.
Formula: |A
B| S 1.0
Examples:
Equivalent: [1.453125 1.390625| 0.0625 (5 1.0)
Equivalent: - 1..106744-(3325157744) = 0.5 (5 1.0)
Inequivalent: |12.91829- 11.78172 1.136 1.0)
Clarifications:
Differences in significant figures decimal places acceptable as long as the absolute difference rule met.
Numerical signs must match (c.g., and -5 are not equivalent).
2. Mathematical Expression Equivalence
Rule: Assess if expressions are mathematically equivalent, not just if they are written iden- tically. Examples of Equivalence:
CommutativityAAsoooaativ a b is equivalent to b
Alternative Forms: x/2 equivalent to 0, 5x
Factoring/Expansion:: x² is equivalent to (z 1) + 1)
Variable Renaming: f(x) x² equivalent to (
Important Caveat: transformation alters the domain in way that impacts the conclu-
sion (e.g.. introduces division by zero), the expressions are not equivalent.
3. Conclusion Equivalence
Rule:
The final answers or main conclusions must align.
Numerical Conclusions: Must meet the numerical equivalence standard defined above. Categorical Conclusions: Must be identical (c.g.. "Positive" vs. "Positive"; "Category A" vs. "Category A").
Strict Output Format
1. Reasons: Provide concise, objective, bulleted list explaining your rationale.
2. Verdict: State the final decision: True (for equivalent) or False (for inequivalent).
3. Boxed: Wrap the final boolean value in \)boxed(). for example (Treeed/Truue
(False.
Texts to Evaluate:
[Text A] [Txt_A)
(Text B] TTxx B)
31

---

Published as a conference paper at ICLR 2026
Teat Rusocid
Puol
l
New
(a) Training response length
(b) AIME24 response length
(c) A1ME25 response length
Figure 5: The evolution of sequence lengths for AgentMath and Text-Based model during RL train- ing and on the AIME24 and AIME25, Both models started from their best SFT checkpoints trained on 20k data.
Table 5: Performance of different backbone models in SFT and RL stage on AIME24/25
Models
AIME24
AIME25
Swen3-1.7B-SFF--N.
44.5%
34.8%
Qwen3-1.7B-RL
59.6%
48.1%
-W-n3-8B-SFFT-30
78.4%
72.2%
Qwen3-8B-RL
89.8%
84.7%
Somn3-30-A3B--93
83.9%
80.5%
Owen3-30B-A3B-.
90.6%
86.4%
A.7 DETAILED ANALYSIS
A.7.1 TOOL-AUGMENTED SYNTHETIC DATA VS. TEXT-BASED DATA
To assess the effectiveness of AgentMath, our proposed tool- aoo-eeueented agent framework for com- plex mathematical reasoning. we conduct experiments addressing two key questions: (1) What per- formance advantages do tool-augmented synthetic data provide over text-only data in SFT phase? (2) Can tool augmentation enhance both model performance and training efficiency in RL phase? All experiments employ Qwen3-8B-Ban as the backbone model with a maximum sequence length of 64k and limit of 64 tool invocations in RL.
Supervised Fine- Moni-Tunigg Performance. As shown Table when trained on an identical 20k SFT data, the vol-augmented model achieved accuracies of 60.5% on AIME24 and 53.3% AIME25, surpassing the plain-text baseline (57.1% 1% and 49.2%) by margins of 3.4% and 4.1%, respectively. This result confirms the effectiveness of our data synthesis method, which transforms computation- intensive reasoning steps into executable code.
Agent RL Efficiency. The benefits of tool augmentation further amplified during RL. As de- tailed in Figure and Table 2 the tool-augmented model required only approximately 400 training steps improve from 60.5% to 76.2% AIME24 and from 53.3% to 67.5% AIME25. In contrast, the text-based model needed around 1,600 steps to reach 68.7% and 57.5%. This repre- sents 4.0x improvement in training efficiency. Notably, the tool-augmented model matched the final performance of the text-based model within just 100-200 steps, underscoring the advantage of dynamically interleaving natural language reasoning with code execution for accelerated policy optimization.
Improved Inference Efficiency and Scalability, As indicated in Figure [S] the tool-augmented model also demonstrated superior inference efficiency. During RL training and inference, its se- quence length ranged from 24k to 29k tokens, compared 28k-34k for the text-based model, with reduction of roughly 4k tokens (~14%). Furthermore, the growth in sequence length was sig- nificantly slower for the tool-augmented model as training progressed. These efficiency gains stem from precise tool-based computations replacing verbose and error-prone manual calculation steps.
In conclusion, AgentMath, by seamlessly integrating natural language reasoning with precise com- putational tools, demonstrates substantial improvements across all critical metries (accuracy, train- ing efficiency, and inference cost). These findings validate the effectiveness of both our tool- augmented data synthesis method and the agent-based RL framework.
32

---

Published as a conference paper at ICLR 2026
Table Efficiency evaluation of AgentMath RL training framework
Method
Time per step (s)
Speedup
Static Batch Synchronous Rollout
Request-Levee Asynchronous Rollout
2100-2500 1100-1300 750-900
1.5-1.8x
Agentic Partial Rollout
3.0-3.3x
Prefix-Aware Weighted Load Balancing
4.0-5.0x
Table 7: Impact of the number of partial rollout segments (N) on training efficiency and model performance.
Partial Rollout (N)
Time (100 steps)
AIME24
AIME25
Partial Rollout
62h
70.5%
60.5%
Partial Rollout N 2
28h
70.1%
60.7%
Partial Rollout N 4
22h
70.8%
60.7%
Partial Rollout N 6
22h
69.8%
60.1%
Partial Rollout N 8
23h
69.5%
60.5%
A.7.2 MULTI STAGE RL TRAINING
Following SFT, the model frequently produced responses exceeding 32k tokens on complex mathe- matical problems, with particularly challenging instances surpassing the 64k tokens. To balance training efficiency with model capacity, we developed an adaptive, multi-stage reinforcement learning strategy. This method progressively unlocks the model's potential by dynamically panding the sequence length and tool-call budget. truncation rate exceeding 10% for either re- sponse length or tool usage triggers automatic budget increase: context length increases from 48k to 72k (step 120) and finally to 96k (step 280), while the tool-call limit expands from 48 to 72 (step 140) and then 96 (step 320) as shown in Figure nnd
31
As illustrated in Figure 3 the training progression reveals significant trends: As training progresses, generated trajectory lengths increase from 24k 30k (Figure tool invocation frequency rises from 27 to 31 calls per problem (Figure Be). and code utilization improved markedly from 70% to 95% (Figure 3dp. These metrics indicate the model's growing proficiency in complex, multi- step reasoning and sophisticated tool use. Correspondingly. accuracy on the AIME24 benchmark rose from 78,4% to 89.8% (+11.4%), and on AIME25 from 72.2% to 84.7% (+12.5%), as shown in Figure 3a Accuracy consistently improves following each capacity expansion. Crucially, the model exhibited emergent capabilities in self-correcting its generated code (Figure These results confirm the efficacy of our multi-stage reinforcement learning strategy. which strikes optimal balance between computational efficiency and model capability. Additionally, Table [5 details the performance of AgentMath with different backbones on AIME24 and AIME25. It shows that approach brings significant enhancement in both SFT and RL stages, demonstrating the robustness and effectiveness of the data synthesis method and the multi-stage RL training strategy.
Key Findings. The experiments establish three critical insights: (1.) Expanded capacity is cru- cial: Increasing the sequence length and tool-call budget is essential for facilitating deeper reasoning chains. (2.) Effective reward shaping: The sustained growth in tool usage confirms that our compos- ite reward function successfully guides the model's tool-call decisions. (3.) Framework robustness: Stable training under the extreme configuration of 96k tokens and 96 tool calls underscores the robustness of both the AgentMath framework and its asynchronous training infrastructure.
A.7.3 SYNTHETIC DATA REFINEMENT AND SCALING LAW
Table 3 presents a comprehensive evaluation of AgentMath's data synthesis pipeline. The initial, unrefined synthetic data yielded suboptimal results (AIME24: 35.3%; AIME25: 25.7%), primar- ily due to formatting inconsistencies and non-executable code. By progressively applying multi- dimensional quality refinement, including format consistency, code executability verification, and environment feedback alignment, model performance improved substantially, achieving accuracies of 58.6% on AIME24 and 50.8% AIME25. The subsequent integration of self-correction
33

---

Published as a conference paper at ICLR 2026
78
AINED 1
Partial Rolboat No1
Partill Rodioa Ne1
Partial Rollout N-2
Partial Rullout N-2
Partial Rollout N-6
Portial Rollout
50
Partial Rolloat N-8
Purtial Rollout N-8
Trin
Triin
AIME24 Acc
(b) AIME25 Acc
Figure 6: Exploring the performance impact of agent partial rollout segment count on AIME24/25, adopt the Qwen3-8B-2w-SFT model the RL initial point.
AIME24
80
75.4
AIME25
73.8
70
Testing Accuracy (%)
60.5
06.4
60
61.8
40
30
20
21.1
2k
20k
40k
80k
160k
300k
Training Data Scale
Figure 7: Exploring the scaling laws of tool-enhanced synthetic data, with performance evaluated on AIME24 and AIME25, using the Qwen3-8B. 1--33-8--BBss model as the backbone.
34

---

Published as a conference paper at ICLR 2026
pabilities, combined with supervised fine-tuning using selective feedback masking based code execution results, yielded final performance of 60.5% on AIME24 and 53.3% on AIME25, These results underscore the critical contribution of each refinement operation.
Building upon this validated data synthesis pipline, we further explored the impact of scaling the tool-augmented dataset, shown in Figure 7 Scaling the dataset from 2k 300k led to a perfor- mance increase from 27.2% to 78.4% on AIME24 and from 21.1% to 72.2% on AIME25, demon- strating the effective scalability of our approach. By combining rigorous quality control with ef- fective scaling, AgentMath effectively alleviates the data scarcity in tool-augmented mathematical reasoning, laying a robust foundation for developing segh-performnce reasoning agents.
A.7.4 EFFICIENCY OF AGENTMATH RL TRAINING FRAMEWORK
To alleviate the computational bottlenecks in agent reinforcement learning caused by ultra-long quences and frequent tool use, we evaluated the efficiency of our AgentMath training framework. As shown in Table a conventional static, batch-synchronous rollout approach required 3600- 4000s per training step. By introducing request-level asynchronous rollout scheduling, we cut this latency to 2100-2500 s (a 1.5-1.8x speedup), mitigating head-of-line blocking from tool invoca- tions. Incorporating agentic partial rollouts further reduced latency to 1100-1300s (a 3.0-3.3x speedup). Finally, adding prefix-aware weighted load balancing brought the per-step latency down to just 750-900 achieving total 4.0-5.0x speedup and demonstrating AgentMath's 's advantages for long-sequence, tool-interactive tasks.
We also investigated how the number of partial rollout segments (N) affects training efficiency. As shown in Table 7 and Figure [6] training 100 steps took 62 hours with = 1, but this was reduced to 28 hours with N 2 and 22 hours with N 4. However, the benefits plateaued for N > 6 due to the scheduling overhead from excessive segmentation. Critically, these optimizations did not harm performance; the model maintained consistent accuracy of approximately 70% AIME24 and 60% on AIME25 across all segmentation strategies. These results confirm that AgentMath effectively resolves the efficiency challenges of long-sequence agent RI., offering scalable solution for scenarios that require extended sequences and intensive tool use.
A.8 CASE STUDY
The following example illustrates the dynamic interaction between text reasoning and tool use in AgentMath's problem-solving process. Notably, the model also exhibits an emergent capability for code self-correction. Code blocks highlighted red indicate an execution error.
A.8.1
AGENTMATH CASE 1
A.8.2
AGENTMATH CASE 2
A.9
LLM USAGE STATEMENT:
L.M is employed solely for grammar checking and expression polishing enhance the readability of the text.
35

---

Published as a conference paper at ICLR 2026
(AIME24-22) Question: Let SABCS be a triangle inscribed in cirele SioomegaS. Let the langents to SWomegaS at SBS and SCS intersect at point SDS, and let AlD intersect SioomeaaS at SPS. If SAB-SS, SBC-9S, and SAC=10S, SAP$ can be written as the form Sifrac [m3 [n]S, where SmS and SnS are relatively prime integers. Find Sm n8. Ground Truth Answer: 113
AgentMath Response: I need to solve this geometry problem. We have triangle ABC inscribed in a circle o. Tangents to o at points B and C intersect at point D. Line AD intersects the cirele again at point P. Given the sides AB 5, BC 9, = 10, need to find AP as a fraction m/n in simplest terms and then find m n.An'nFirst, So equations:\nlnl) 252x + y= 0'min2) 252x - 33 2 add them together: 2268 / 504. Let's use
(252x 33s y) + (252x - 33s y) + 2268 504x 2268 code to simplify this fraction.In in <code> \n pyt"htooonoofrom fractions import Fraction'n'n# We need to solve for from the equation: 504x 2268 The initial fraction for x is 2268/504.nn x fraction Fraction 504)in in print(f" simplified value for x is: {x fraction)" "))m "Nnn </code> in in <interpreter> \n The simplified value for x is: 9/2 \m/ </interpreter>n in So x = 9/2 = 4.5. Now from equation 1: 252x + 33 s y =00, so 33 $ y -252 y- / (33 s). Leti's calculate this with code.'n in <code> In "python in from fractions import Fraction in \n# We substitute x = 9/2 into the equation y -252x (33*s)ln# where = sqrt((44).nft We calculate the coefficient of 1/s. An coefficient = Fraction(-252) FFactionn9,, / 33 'n in The coefficient for 1/s coefficient for 1/s is: 37811Nbbne/ineeprrde--lllbbb y = -378 (11 sqrt(14)). Rationalize: multiply numerator and denominator by sqrt(14): [-378 sqrt(14)) / (11 * -3) sqrt(14) / 154. Let''s use code to simplify this fraction.ln'n <code> \n "'python \n from fractions import Fraction in \n # We need to simplify the fraction -378/154. 'n # The full expression for y is -378 sqrt(14) / 154, 'n simplified fraction = Fraction(-378, 154) \n In Then simplified fraction is: (simplified ffrctioo)"" In \n </code> \n In In <interpreter> In The simplified fraction is: -27/11hainn/hnterprreer>- ...Now, we can compute the squares and the Numerator partsinvall = 264**21nval2 = 225**2mn'nf Denominator partsinval3 22**2\nval4 = 157 = 107 Calculate the full numerator and denominatorimmmmrrtoor vall valZundeommintor
-252
is: "hwp://a"""oooo/oooeeeeonnnieeer
mathinind
va13 val4 * val5salaprrin(((2266222 = (vall"")prrin(((222>> = (val2"))aprin(((2222 = (MMB)))Japrin(("NNuerrtt) (264^2 225*2) = (numerator)"" "aarrnia("ci(((""noooiito
(22^2 * 157 107) |denominator)")uu hwwe///oeeeaiiniieeeeeeeeeeeeeee
=
69696222522 50625'n2222 484mnummrattr (264^2 * 225*2)
352836000ineenmmnnttr (22^2 * 157 * 107) 8130776minniinteppeetr.... 100 and
13 coprime, yes.In'nSo m/n 100/13, m=n=1133.nnmuut let me confirm/n'nIn the problem, sides are 5,9,10, which is almost isosceles, but not, and AP=100/13=77.62, which plausible.ln'nNww, to box the answer.inlnSo 1-===1100-1331113mmmmul let me make sure-lninSince we have AP, and it's rational, and matches. maatches.nnnAAnothee An'nhr way to confirm: using the fonlula.nnn recall that in some geometries, the length can found using formulas, but I think it's correct,IninSo the answer is 113.
Figure 8: AgentMath Case study
36

---

Published as a conference paper at ICLR 2026
(AIME24-29) Question: Let Stoomeaai eq 1$ be a 13th root of unity. Find the remainder when [[iprrd_((-0)<<111_ (1-0)=((12(2222ommme) Kooeeoeee
is divided by 1000.
Ground Truth Answer: 321
AathtMath Response: The problem states: Let # 1 be 13th root of unity. I need to find the remainder when the product from k=0 to 12 of (2 2k + oo (2k)) is divided by 1000. Since it''s product over all 13th roots of unity except 1, but k goes from 0 to 12, which includes all roots. o a primitive 13th root of unity, I assume, since it''s specified #1, but k=O gives @ 00011, so it includes k-0. in In Let me denote the expression inside as f(k) = 2 2k + o (2k). product numerically to check. in in <code> in "'python in import emath In import math
Now, let me compute the
Lett omega be the primitive 5th root of unity, omega exp(2*pi*i 5) in omega emath.exp(2 emaahhpp 5) 'n An # Calculate the five terms g(omega> in g_0 0 omega + 2 In g_2 omega*#4.
(0)*eg***)**22 -2 (omega**0) *00 + 2 'n g_ omega**2-2 "meea**2 + 2 'n g_3 - ammega*6-- - 2 omega**3 + + 'n g_4 omega****** **8 2 * omega**4 + 'n An # Calculate the product of the five terms in ootal_product = g_ 0
g_ =_ 1133*__4W In \n # Print the individual terms and the final product 'n # Using round to get cleaner numbers as the imaginary parts might be very small in print(f'gg() = (round(g 0, 5)}") in print(f"g((mmegy = fround 1, 5)3") 'n print(""g(omega"
{round(g 2, 5) 'nn
{round(g 3, 5)}") in prime("ggommega44 = {round(g_44 4, 5)}") An print(f"Total product Loannddtttal_prduddd 5)}") In 'n </code> 'nnn <interpreter> In Traceback (most recent call last): in File httmp//mmwiiiz//////n/ooooortt"" line 19, in <module> In print(f"'g()) (round(g 0, 5)3") 'n εаа.0/ооооа in TypeError: type complex doesn'tt define round method in \n </interpreter> In in Oops, the code above appears to be throwing an error. I need fix this ensure it runs successfully.< The error occurs because the 'roundO' function cannot handle complex numbers directly. I will modify the code to handle complex numbers correctly. 'n in <code> 'n "'python in import cmath 'n import math In In # Let omega be the primitive 5th root of unity, omega = exp(2*pi*i / 5) in omega cmath.exp(2 emath.pi 1j / 5) 'n 'nn # Calculate the five terms g(omega"k) in g_0 (omega**0) #0)#* -2 (omega**0) + 'ing 1 omega**2. 2 * omega 2 'n g__ 2 omega**4 2 * omega ##2 + 2 'n g_3 omega**6 2 03ega**3 + 2 \n 4 = omega** **8
omega* #4 + In \n # Calculate the product of the five terms 'n total _acal__rouuctttt 0_ * g
* 2 g 3 * g_4 4 In 'n Print the individual terms and the final product in print(f"g(())
{g_ 5f {g_ 0.imag:+.5f))"" 'n print(f"
Irell:el) {g_ Limag:+.5f)j"" In
mrin(("g((omega()) = {g 2.real:.5f) ig 2.ima":+.5fl"") 'n pnine("g(oomeea"))
(g_3.real:.5f) (ll_33.ima::+5511) 'n print("g((omega"4)== {g 5.r {g 4.mma::+,5f))"))
In print(f"Total product = (total product.real:.5ff {total prw.uct..mag:+t-5333 in \n
1.00000+0.00000 'n g(omega) = 0,57295 1.31433j In g(omega²2) 3.92705 2. 12663 \n g(omega'3) = 3.92705+2.12663) In g(omega"4) 0.57295-1.31433, in Total product- 41 10000000.00000 In in </interpreter>> Anin The numerical calculation shows the total product is 41. The small imaginary part is due to floating-point inaccuracies... here f-g(x), so f2-1, f1=-2, f0-2 'innn gxr(x), so g1-664, g0=1 Inin Thus P-8321. in'n Now, 8321 mod 1000 is 321. Anin So the remainder is 321.
</code> in <interpreter>ing(]
Figure 9: AgentMath Case study For code self-correction
37

---

Published as a conference paper at ICLR 2026
A.10 MOTIVATION AND CORE CONTRIBUTIONS
While agentic RL training -nncluding coding-assisted assisted math reasoning has been explored in prior works such as ReTool and ToRL, the field currently faces two critical challenges that remain inade- quately addressed.
Challenge 1: Lack of efficient tool-augmented data synthesis methods and scaling-law stud- ies for long-chain reasoning. Existing approaches face substantial limitations when synthesizing tool-augmented data in long-chain reasoning scenarios. For instance, CoRT relies on manual notation, making it costly and difficult to scale. While ReTool employs LLMM-based synthesis, produces only 2k samples and provides insufficient details about the synthesis process. Moreover, existing research lacks systematic investigation of scaling laws for agent-synthesized data: in code- assisted mathematical reasoning, what is the relationship between data scale and performance, and can superior performance be achieved through data scaling alone? These fundamental questions have rarely been systematically studied in the open source community.
Challenge 2: Agent RL training encounters severe efficiency bottlenecks in ultra-long se- quence and large-scale tool invocation scenarios. Even when models perform well code- assisted mathematical reasoning after supervised fine-tuning, it remains an open question whether performance can be continuously improved through large-scale Agent RL to achieve adaptive tool invocation. Under conditions involving ultra-long reasoning chains and large-scale tool invocations (e.g., 96k token length, 96 tool calls), training efficiency becomes critical bottleneck. Frequent environment interactions and ultra-long trajectory generation cause traditional batch-synchronous rollout training on Qwen3-8B to require 3,600-4,000 seconds per training step, with rollout gen- eration accounting for 70%-80% of total training time. This inefficiency severely constrains re- searchers' exploration of Agent RL training in long-horizon, large-scale tool-calling scenarios. We note that training efficiency is crucial for RL training: the ability to train for thousands of steps and break through performance plateaus directly impacts final results. For instance, DeepSeek-RI- Zero requires 4,000-8,000 training steps to observe the model's "Aha moment" phenomenon and continuous performance improvements.
Furthermore, while leading proprietary models (e.g., GPT-5, Gemini 3.0, Claude 4.5) demonstrate exceptional performance on agent tasks such as code-assisted mathematical reasoning. the lack of technical details and training methods hinders the open-source community's 's ability to reproduce and advance this research. By providing a clear, transparent, and complete pipeline for data synthesis and large-scale Agent RL training that achieves competitive performance with proprietary models (e.g., Gemini 2.5-Pro, OpenAT-o3), our work aims to offer the open-source community a reproducible technical roadmap, thereby advancing the development of the Agent LLM field.
To address these challenges, we highlight four core technical contributions.
A.10.1 EFFICIENT TOOL-AUGMENTED DATA SYNTHESIS METHOD
Segmented synthesis with focused tool invocation. We observe that previous methods (e.g.. Re- Tool) suffer from tool overuse issues, including invoking tools for simple calculations and perform- ing redundant verification of already-verified correct results, leading to an increase in ineffective tokens. To address this, AgentMath focuses on applying tool invocations to genuinely necessary complex computational scenarios (e-g., solving complex equations, performing large-number arith- metic, and handling advanced linear algebra and calculus operations). We propose a segmented synthesis strategy that partitions DeepSeek-RR -R'ss long chain-of- of-thouught process into fixed-length segments (e.g., 3K tokens) and transforms them into tool-augmented chains-of-thougg incorporat- ing automated multi-dimennional quality refinement to ensure high data quality.
Under identical training settings- uuigg the same 2k synthetic data problems from ReTooll's offi- cial open-source release, the same DeepSeck-RI teacher model responses, the same base model (Qwen2.5-32D1 5-2-...5522B-nstruccllll and the same number of RL training steps (400 Ateps)-AgemtMthh's 's synthesis method significantly outperforms ReTool's original approach. As shown in Table g SaautMatth-2B-SFT-2 achieves 44.1% on AIME24 and 37.3% AIME25, outperforming ReTool- (ooool-32B-SFTE - (40.9% and 34.5%). After RL training, AgentMath-322B- reaches 74.8% on AIME24 and 56.6% on AIME25, similarly outperforming ReTooll-3B--R (67.0% and 49.3%).
38

---

Published as a conference paper at ICLR 2026
These results validate the effectiveness of our data synthesis method and the superior quality of the synthesized data.
Table 8: Performance comparison between AgentMath and Re Tool on AIME24 and AIME25 under identical training configurations, demonstrating the superiority of AgentMath -ol-auumented data synthesis method in both SFT and RL stages.
Model
AIME24 Acc
AIME25 Acc
Re ReTool-33BBSSFT-22
40.9%
34.5%
AgentMath-32B-SFF--K
44.1%
37.3%
ReTool-32B-RL
67.0%
49.3%
AgentMath-32B-RI
74.8%
56.6%
Multi-dimensional quality refinement pipeline. We further emphasize the multi-dimennsinal quality refinement pipeline in data synthesis process, which includes correcting format incon- sistencies, excluding samples with failed code execution, performing environmental feedback align- ment, filtering low-complexity code (c.g.. <5 lines), and injecting self-correction capabilities. As shown in Table 9 through this pipeline, accuracy on AIME24 improves from 35.3% with initially unrefined synthetic data to 60.5%, and on AIME25 from 25.7% to 53.3%, substantially demonstrat- ing the critical role and effectiveness of each refinement module in optimizing data quality. We are committed to maintaining transparency and clarity in our synthesis and refinement procedures, whereas some recent works provide limited details on this crucial aspect, thereby posing challenges for research reproducibility.
Table 9: Progressive improvement in model accuracy AIME24 and AIME25 through the multi- dimensional quality refinement pipeline. Each refinement step is applied cumulatively, demonstrat- ing substantial accuracy gains from 35.3% to 60.5% AIME24 and from 25.7% to 53.3% AIME25.
Refinement Steps
AIME24
AIME25
Initial Unrefined CI-Synthetic Data (20k)
35.3%
25.7%
+ Format consistency correction
47.4%
40.1%
Code executability verification
52.8%
44.8%
Environmental feedback aligmment
56.3%
48.3%
Tool-usage rationality assessment
57.2%
48.9%
Self-correction capability injection
58.6%
50.8%
SFT with selective feedback masking
60.5%
53.3%
1.1 SCALING LAWS FOR TOOL- AUGLENAUGE SYNTHETIC DATA
We systematically validate the scaling laws associated with augmeneedd synthetic data. Through AgentMath's segmented data synthesis strategy and multi-dimensional quality refinement, we pro- gressively scale the dataset from 2k to 300k samples. Fine-tuning the Qwen3-8B-Base model, as shown in Table 10 observe consistent performance improvements: accuracy AIME24 in- creases from 27.2% to 78.4%, and on AIME25 from 21.1% to 72.2%. Performance continues to improve with data scale, ultimately achieving excellent results. These findings robustly demonstrate that scaling laws remain effective for tool-augmented synthetic data. In contrast, some recent studies (e.g.. g.. ReTool utilize only limited data (e.g., 2k samples) and do not sufficiently explore the potential benefits of data scaling.
A.10.3 EFFICIENT AGENT RL TRAINING FOR ULTRA-LONG SEQUENCES AND LARGE-SCALE TOOL CALLING
In Agent RL, the combination of long-co song-context generation and frequent external tool interactions creates heterogeneous computational workloads, posing significant challenges to training efficiency. To address this, design targeted optimization strategies comprising three core improvements:
39

---

Published as a conference paper at ICLR 2026
Table 10: Data scaling law of AgentMath-SB models as function of SFT training data vol-
ume, demonstrating consistent improvements from 2k to 300k samples on AIME24, AIME25, and Google-IMO-Answeenr
SFT Training Data Volume
AIMF24
ATME25
Moagle-MOOAnsseeriie
Avg Score
AgentMath-8B--5
27.2
21.1
5.3
17.9
SeoniMat--8B-ST--0
60.5
53.3
12.8
42.2
AgemtMath-8B
SaanMaat-8B-SFT-4000
66.8
57.9
17.6
47.4
AgentMath-882 -aB--att----SF--SS---
69.2
61.8
21.4
50.8
AgentMath-88 Saath-8B-SFF--6600
73.8
66.4
25.6
55.3
DgenMatt-8B-SSF--S.
78.4
72.2
28.8
59.8
Request-Level Asynchronous Rollout Scheduling. We replace the conventional static batch synchronous processing with a coroutine-driven request-level asynchronous sched- uler. Each trajectory rollout treated as an independent long-running request, where inference engine (server) and the agent (client) are fully decoupled through asynchronous communication. This effectively mitigates the efficiency bottleneck caused by synchronous waiting.
2. Agentic Partial Rollout. To alleviate the long-tail latency issues arising from ultra-long sequences and large-scale tool calls, we propose an agentic partial rollout mechanism, in- cluding Length Partial Rollout and Tool Partial Rollout. This mechanism decomposes each trajectory into audget-constrained segments, where each segment is limited to fixed maximum sequence length (e.g., 32k tokens) and fixed maximum number of tool calls (e.g., 32). Incomplete sequences or tool calls are carried over to subsequent batches for continued execution, while completed trajectories immediately participate training, thereby significantly improving training efficiency.
3. Prefix-Aware Weighted Load Balancing. Since Partial Rollout introduces ultra-long quences, leading to a substantial increase in KV cache memory consumption and prefill computational overhead, we design prefix-aware weighted load balancing strategy. This strategy dynamically assigns weights based on the prefix length of requests and intelligently routes them the least-loaded inference engine instances, effectively alleviating memory and computational pressure.
We systematically evaluate the efficiency improvements of the AgentMath training framework. As shown in Table []] the traditional static batch synchronous rollout method requires 3,600-4,000 seconds per training step. With the introduction of request-level asynchronous rollout scheduling, latency reduced 2,100-2,500 seconds (a 1.5-1.8x speedup). Further incorporating the Agentic Partial Rollout mechanism reduces latency to 1,100-1 1,300 seconds (a 3.0-3.3x speedup). Finally, with the incorporation prefix-ave weighted load balancing. the per-step latency drops to 750- 900 seconds, achieving an overall training speedup of 4.0-5.0xx This framework enables efficient Agent RL training in extreme scenarios (c.g., 96k sequence length, 96 tool calls), for which training with such ultra-long sequences and large-scale tool calls has few precedents in the open-source community.
Table 11: Progressive training efficiency improvements in the AgentMath framework through cu- mulative optimization strategies, demonstrating up to 5.0x speedup over baseline static batch syn- chronous rollout.
Method
Time per step (s) Speedup
Static Batch Synchronous Rollout
3,600-4,000
Request-Level Asynchronous Rollout
2,100-2,500
1.5-1.8x
Agentic Partial Rollout
1,100- 750-900
3.0-3.3x
Prefix Aware Weighted Load Balancing
4.0-5.0x
40

---

Published as a conference paper at ICLR 2026
A.10.4 STATE-OF-THE-AN ART PERFORMANCE VIA LARGE-SCALE AGENT RL
AgentMath undergoes ttree-stage curriculum learning process, with sequence lengths progres- sively increasing from 48k 72k to 96k tokens and tool invocations scaling from 48 to 72 to 96. Across diverse model backbones spanning 1.5B to 32B parameters, we successfully conduct sta- ble Agent RI training ultra-long sequences with large-scale tool invocations, achieving sistent performance improvements. As detailed Table [2] we achieve state-of-the-arr perfor- mance on challenging mathematical competition benchmarks including AIME24, AIME25, and HMMT25, significantly outperforming leading open-source models of comparable size. Specifi- cally, AgentMath -0B-------3BBBB achieves accuracies of 90.6%, 86.4%, and 73.8% on AIME24, AIME25, and HMMT25 respectively, surpassing OpenAl-oô-man and Claude-Opus-4.0m -ww.o..p-suuiiieieeeeeee Tew.hhiiiiiig while re- maining competitive with OpenAI-o3, Gemini-2.5-Prr and DeepSeek-RI-oi -erpSeek-R1-671B-. 11-671B-0522. These re- sults validate the effectiveness and scalability of our data synthesis method and Agent RL training strategy, paving the way for building more sophisticated and scalable mathematical reasoning agents.
Table 12: Performance comparison of AgentMath against proprietary and frontier open- source mod- els on AIME24, AIME25, and HMMT25. AgentMath models are highlighted in blue.
Models
AIME24
AIME25
HMMT25
Proprietary Models
Open-llo4-mini-wttools
98.7
99.5
Gemini-2.5-Pro
92.0
88.0
82.5
OpenAl-o3
91.6
88.9
77.5
OpenAI-o3-minn
87.3
86.3
53.0
83.0
72.0
58.3
Frontier Models (1B-2B)
DeepSeek R1 R1-Distill-Qwen
28.8
21.8
15.3
Qwen3-1.78 Thinking
52.0
35.3
23.3
Boikeasoninng-Nemotooo--- AgentMath-1.7B
55.5
45.6
31.5
59.6
48.1
40.2
Frontier Models (7B-8B)
DeepSeek Braeek-RR--Be RI-tistlll--l-------- RI-Dissill-Qwen-7B
55.0
39.7
Qwen3 8B Thinking
76.0
67.3
44.7
Open Nemoiroonn-
84.7
78.2
63.5
DeepSeek-RP PeepSeek-R1105225 --528-QQen3-8B
86.0
76.3
61.5
AgentMath-8B
89.8
84.7
71.3
Frontier Models (30B-32B)
ReTool-32B
67.0
49.3
DeepSeek R1-Distill- Dissill-Qwen-32B
72.9
59.0
33.0
1rem3-30B-A333 A
87.7
85.0
71.4
OpenReasoning-Neeooroon AgentMath-300B-A3B
89.2
84.0
73.8
90.6
86.4
73.8
Frontier Models
(>32B)
DeepSeck-R1-671B-n0
91.4
87.5
77.0
Qwen3-235B-A22) ThinB-TTiiiiiinn-555 2507
94.2
92.3
83.9
AgentMath-23551 225.-aah--2255--22------2-
93.4
90.8
81.7
Summary of contributions.
In summary, our core contributions are as follows:
We propose an efficient tool-augmented data synthesis method that focuses on applying tool invocation to genuinely demanding complex computational scenarios (e.g., solving complex equations, large-number arithmetic, advanced linear algebra, and calculus). By combining segmented synthesis strategy with multi-dimension quality refinement
41

---

Published as a conference paper at ICLR 2026
mechanism, AgentMath significantly outperforms ReTool in both SFT and RL stages AIME24 and AIME25 under identical training configurations.
2. We systematically validate the scaling laws associated with rool-augmented synthetic data, scaling the dataset from 2K to 300K samples. Building on the Qwen3-8B-Base model, accuracy AIME24 improves from 27.2% to 78.4%, and AIME25 from 21.1% to 72.2%, demonstrating consistent performance gains with increased data scale.
3. To alleviate the training efficiency bottleneck in Agent RL under ultra-long sequence and large-scale tool invocation scenarios, we introduce request-level asynchronous roll- out scheduling, propose Agentic Partial Rollout, and design prefix-aware weighted load balancing. Through these technical innovations, our framework achieves a 4-5x speedup in training efficiency, reducing the time per training step from 3,600-4,000 seconds to 750- 900 seconds, thereby enabling efficient Agent RL training in extreme scenarios (e.g., 96K sequence length with 96 tool invocations). To the best of our knowledge, such exploration of large-scale Agent RL training with extremely long sequences and massive tool invoca- tions is rarely conducted in the apen-source community.
4. Through three-stage curriculum Agent RL training, AgentMath achieves state-of-the-an performance on challenging mathematical competition benchmarks including AIME24, AIME25, and HMMT25 across model backbones ranging from 1.5B 32B parame- ters, significantly outperforming leading open-source models of comparable size. No- tably, AgentMath-3005 -3BB surpasses OpenAl-o3-mine while remaining competitive with Gemini-22 2.5-Pro and DeepSeck-R1-677 R81--22B-- We are committed to open open-sourcing our code, data synthesis pipeline, and model training workflow to advance the L.MM reasoning community.
A.11 RELIABILITY OF EVALUATION BENCHMARKS
The math benchmarks used in our main evaluation (AIME24, AIME25, HMMT25) each contain only about 30 questions, and prior work has raised concerns that these datasets may have leaked into open-source model training corpora. To systematically validate the reliability of our results and mit- igate potential evaluation benchmark leakage concerns, conducted supplementary experiments and analyses from three complementary perspectives.
A.11.1 IMO-ANSWERBENCHI A LATEST DIFFI-IIFYYCLLYY BENCIIMRRK
Considering that datasets such AIME24, AIME25, and HMMT25 may have partially appeared in the pre-training corpora of open-source models, we evaluate AgentMath on IMO-AnswerBench (Luong al.] 2025), recent mathematics olympiad benchmark released by Google DeepMind in November 2025. This benchmark comprises 400 carefully curated problems spanning four algebra, combinatories, geometry, and number theory-with with 100 problems per category, all sourced from national, regional, and international olympiad competitions.
domains-algebra,
To mitigate data memorization and leakage, BeO-n systematically rewrites original competition problems through several strategies: renaming points or lines in geometry problems, rephrasing problem statements, modifying numerical values and/or introducing distractors, and re- formulating problems using entirely different yet semantically equivalent expressions. This design substantially reduces the probability of verbatim matches in pre-training corpora and minimizes the potential for memorization-Be based advantages, thereby enhancing evaluation robustness and faimess.
The performance of current frontier models on this 235B at 53.8%, and DeepSeek-RL. at 60.8%-indicctes that the benchmark possesses sufficient discriminative power and challenge. Moreover, the benchmark's release date (November 2025) notably later than the public release of Qwen3-series models (May 2025), further minimizing the likelihood of contamination in their training corpora.
-Deenarkk--eeSSek------ V3 at 37.0%, Qwen3-
Following the experimental setup specified in the IMO-AnswerBennch paper, report results aver- aged over 8 independent runs. As detailed Table [3] AgentMath achieves strong performance at comparable parameter scales across all four categories:
At the 1.5B-1.7B scale: AgentMath-Qwen3-L1 achieves 20.3%, outperforming
OpenReessoning-Nemotoon (17.8%).
42

---

Published as conference paper at ICLR 2026
At the 7B-8B scale: SeenttMath--ee2.5--12 achieves 35.1% and BgentMath-Qwen3--e achieves 37.9%, both substantially outperforming OpenReasoning-Neemen Nemotron-78 (33.0%).
AgentMath- Baiiltthh----A--- 30B-A3B achieves 51.2%, surpassing Awen3-30B-A3B. Thinking-2507 (41.4%), Claude Sonnet 4, DeepSeek-V3, and Kimi-K2-Instruct, while approach- ing the performance of Qwen3-235B.
At the 30B-32B scale:
At scales larger than 32B: AgentMath- AgentMath-235B-A222 235B-A222-BBP achieves 55.4%, surpassing Qwen3- 235B and approaching DeepSeek-Ra
These results demonstrate that AgentMath maintains strong performance and robustness across dif- ferent parameter scales on a challenging evaluation set that is demonstrably free from data leakage concerns.
Table 13: Performance of AgentMath on Mooggl--MM-AAneeBBBBee a recent mathematics olympiad benchmark released by Google DeepMind in November 2025. Our model (highlighted in blue) is compared against other leading models, with accuracy (avg@8)) as the evaluation metric.
Model
Algebra
Combinatorics
Geometry
Number Theory
Avg Score
Proprietary Models
Gemini Deep Think (TMO Gold)
85.0%
69.0%
88.0%
78.0%
80.0%
Grok
75.5%
80.
80.9%
73.1%
Gemini 2.5 Deep Think
78.0%
49.0%
83.0%
77.0%
71 1%.8%
o4-mini (high reasoning) GPT-5
71.3%
46.6%
78.4%
75.3%
67.9%
03
62.8%
43.0%
70.6%
68.0%
61.1%
Clande Sonnet
20.6%
17.8%
26.0%
27.6%
Claude Opus 4
19.4%
20.0%
23.3%
26.6%
22.3%
Models (1R-2R)
DeepSeek-RI BroSsekk-II-istille -i-Distil-Qwenn--.5
5,5%
9,9%
14.5%
8.7%
9.7%
Qwen3-1 Thinking
11.4%
12.5%
OpenReasoning Nemotron-L.S
16.3%
14.0%
22.6%
18.4%
17.8%
AgontMath- AgentMat--wenn--..
16.2%
20.3%
Frontier Models (7B- KB)
DeepSeek-R RI I-Distill-Lamaa lama3. 1-88
10.2%
17.4%
21.6%
17.0%
16.6%
DeepSeek RI Distill Qwen
12.8%
20.6%
22.8%
18.0%
18.5%
Owen3-8B-TThinkam
17.0%
26.8%
38.0%
25.8%
26.9%
DeepSeek RI 0.0528 Distill. Qwen3-B8 8B
21.2%
31.7%
43.3%
27.3%
30.9%.
OpenBeenoning-Nemooooo-- Nemotron-7B
23.6%
35.2%
43.2%
24.9%
33.0%%
AgcniMath- CeenMMtt--Qwe22555 Swwe
24.9%
37.4%
45.6%
32.5%
35.1%
AgeniMath- Qwen3-8B
28.3%
39.8%
49.5%
33.8%
37.9%
Fronter Models (30R-12R)
DeepScok-RI-Dee Diit-----Dssill-ea----22
17.2%
25.1%
33.0%
22.5%
24.4%
27.8%
29.9%
36.2%
27.4%
30.3%
AM-Thinking-v v 11-22B
27.9%
36.2%
1%
29.2%
34,7%
44.3%
$3.2%
33.3%
41.4%
AgentMath. 30B-A3B
43.4%
40.0%
69.6%
$1.9%
$1.2%
Frontier Models (g328)
DeepSeek V3
39.0%
26.0%
35.0%%
48.0%
37.0%%
Kimi- K2-Instruct
45.6%
Qwen 235B
57.6%
37.5%
57.6%
62.3%
53.8%
AgeniMathh -32B-A22--FFT
43.2%
DoepScok RI
65.0%
40.0%
73.0%
65.0%
60.8%
A.11.2 VALIDATION USING BASE MODELS RELEASED PRIOR TO BENCHMARK PUBLICATION
To further mitigate concerns regarding potential evaluation data leakage into the pre-training corpus, employ open-source base models released before the publication of the evaluation benchmarks for our SFT and RI training. The relevant timeline is as follows:
Qwen2.5-7B-Bar release: September 2024
Llama3. 1e.---8B-Bass release: July 2024
AIME25 publication: February 2025
HMMT25 publication: February 2025
43

---

Published as conference paper at ICLR 2026
GGoglee-MO-AnsserBBeee publication: November 2025
All benchmarks were published after model pre-training was completed, thereby precluding any possibility of data contamination. As shown in Tables [4] AgentMath trained on these earlier base models achieves superior performance across multiple benchmarks:
Building on and HMMT25, respectively, significantly outperforming Laamal1-Nemotron-Nano-BBB (48.0% and 26.7%)
Base,, AgentMath-Llama-55 achieves 66.2% and 53.1% AIME25
Building on Owen2.5-7B-Baal AgemtMath-Qwen2.55-1 attains 79.8% and 65.9% on AIME25 and HMMT25, respectively, surpassing Openkeessoning-Neoorroo (78.2% and 63.5%).
On the more challenging Googlee-MOO-AsserrBBan AgentMath-LLamm-S8l achieves 25.2%, (16.6%), while AgentMath-Qwen2.555
outperforming DeppSek-RI--Distil-Laaa... (35.1%) 1%) also exceeds OpenRRessoingg-eeoorrn-- (33.0%).
By leveraging base models released in 2024 (Qwen2.5-7BB-Bas and Llama3. 1-8B-Base), Agent- Math demonstrates consistent and leading performance across multiple mathematical reasoning benchmarks released in 2025 (AIME25, HMMT25, and IMO-AnswerBench), These results not only eliminate data contamination risks but also establish the robustness and reliability of our ap- proach.
Table 14: Performance of AgentMath on AIME25 and HMMT25 based on Llama3, 1-8B-Base and Qwen2.5-7B Base. Our model (highlighted in blue) is compared against other leading models, with accuracy (avg@ @3) as the evaluation metric.
Models
Base Model
Tool Use
AIME25
HMMT25
Based on Llama3.1-8 M-.--BBBBe/nsttuee
DeepSeck-RR R1 Distill-Llama-
Llama-3.1
28.7
13.8
Llama3. I-Nemooroon-Num -aa-n---n----
48.0
26.7
SrontMath-Llama--a
Llama-3.1-8B-Bal
66.2
53.1
Based Qwen2.5-7B Base/lustruct
ZeroTIR-7B
Qwcn-2.5-7B-Base
30.0
22.5
SimpleTIR-7B
Qwen2.5-7B-Baa BBBBasc
30.9
29.7
DeepSeek-R11
S1--iitill-Qwen-..
Qwen2.5-Math Mat.5--Mat--BBBBBs
39.7
16.3
OpenRI-Distille
Owen2.5-Math-7l-Be
39.7
25.7
Light-R1-7B-DS
DeepScek-RI-Diie Disttill-QQen--7
44.3
27.6
AReal-hoba-78
DeepSeek-R DDstill-Qwent
48.3
29.4
Skywork-OR1-7B
DeepSeek-R1-n Baaibeek--R-Disilllle S11Distilled-Qwert
54.6
35.7
AucReusonnNemmotron-LLLee
DeepSeck-R1-DDe R---DistlllQwen---
64.8
42,9
OpenReasoning NenccsominngNNmmmmmmm-111 Nemotron-71
Qwen2.5-7B. 1een2.557BBInstruet
78.2
63.5
AgentMauh-Qowen225577
Qwen2.5-7B-Bas
79.8
65.9
Table 15: Performance AgentMath on Google-IMO Google-MO--usweeBeerie AnswerBench based on Llama3. 1-8B-Base and Swen2.5-7B-Basee Base. Our model (highlighted in blue) is compared against other leading models, with accuracy (avg@8) @)) as the evaluation metric.
Model (Gongle IMO-Annwereae AnswerBBeee)))
Base Model
Algebra
Combinatorics
Geometry
Number Theory
Avg Score
1-88 M8--aae
DeepSeek-Ri: Ri-Distill Di-siiiil--laaaae Lluma3. AgeniMath- Jama-SB SB
Llama3,1 1-0ma BB-Buse
10.2%
17.4%
21.6% 33,7%
17.0% 23.7%
16.6% 25.2%
Lama3. I 8B
18.8%
24.6%
Owen2 Rast/hasructt
OpenReasoning-None Nemotron-7B AgentMath SennMMth--Qeen.555 Qwen1.5.BB
Qwen2. Dwen2.5-7B.. 5-7B- "1B-nsstrctt
23.6%
35.2%
43.2% 45.6%
29.9%
33.0%
Qwen2.5:7B 5:7B-Base Base
24.9%
37.4%
32.5%
35.1%
A.11.3 SYSTEMATIC COMPARISON OF SET AND RL CONTRIBUTIONS
To rigorously evaluate the independent contributions of SFT and RL in AgentMath while mitigating potential data contamination, we conduct systematic comparisons exclusively benchmarks finmed to be released after the pre- vreitraining cutoff date of their corresponding backbone models, as shown in Tables 16 and 17
44

---

Published as a conference paper at ICLR 2026
For Llama3.1 1-8--Base and Qwen2.5-7B-Bal report results AIME25, HMMT25, and Google-IMO- IMO-AnswerBenchh
For Qwen3-1.7B-Baae Swen3-8B-Bass and Beeen---0B-AABB-nstme------ on Moogle-IMO-AnswerBBBee only.
report results
The results reveal consistent improvements across all base models. Based on Llama3. 1-ama..1-BB-B... 1-8B-Base, AgentMath-Llama- Laa---lmmmaa------ 8B-SFT achieves an average score of 39.2% across the three benchmarks, while AgentMatth-Llama Baalllmaa----RR 8B-RL reaches 48.2%. Based on Qwen2.5-7BB BB-Base, AgentMath- AgentMath-QQen2..5775 SFT attains 49.6%, with AgentMatth-Qween25-775. achhiii 60.3%.
On the Qwen3 series, we observe similar trends on 1.7B-Base AggntMMath-Qwen3-11775 Qwen3-1.7B-SFF scores 11.7% while the RL variant scores 20.3%; based on Qwen3-8B-Ban AgentMath-QQeen-SS--STT scores 28.8% while the RL variant scores 37.9%; based on variant scores 51.2%.
based on Qwen3-
AgentMath-30B-A3BB
scores 43.5% while the RL
These results demonstrate that tool-augmented SFT substantially enhances mathematical reasoning capabilities across all base models, while RL consistently delivers an additional 7%-10% perfor- mance gain over SFT.
Table 16: SFT and RL performance of AgentMath AIME25, HMMT25, and Google-IMO- AnswerBench based on
and Qwen2.5-7B-Bas.
Model
Base Model
AIME25
HMM125
Google MM0-AAmMMMMA Score
Rosed on Hama3. 1-8B- BR-e
AgentMath-- 8B-tnn-BB-FFTT-
Llama3. 1-8B-Base
53.5%
45.8%
18.3%
39.2%
AgentMath Llama 8B RL.
AgentMath- Llama- SFT
66.2%
53.1%
25.2%
48.2%
Based Qween.55-7BBBe
AgentMath- Seaetl--wenn2-5-1 BB-TT
Qwen2.5-7BB 5-7B-Base
66.4%
55.1%
27.3%
49.6%
AgentMath S-antMaat--Qen255--100 Q.en2.5-7B-RL
AgcotMath-QQwen25-7B5.
79.8%
65.9%
35.1%
60.3%
Table 17: SFT and RL performance of AgentMath on Google-MOOAAnsweeBeech based on Qwen3- series base models (1.7B, 8B, 30B).
Model
Base Model
Google-IMOO-Anserreennd
Based on Qwen3-1.78-Baae
1aanMMathh-Owe3311725 SeenMMuth-QOwem3-117--..
Qwen3-1. Doven--17B... 1.----..BB-Bccc
11.7%
AgentMath- Owwn3-1.7B-SSF
20.3%
Based on Qwen3-8B-Base
BheniMath-Qwen3-811
Qwen3-8B (aaa--B-Basee
28.8%
AgentMath-QwenZ--BB Qwen3-8B-SFT
37.9%
Based on Qwen3- 30B-A38 A.B.-AA.-Instttuee 4.38--nstructt2507
Dwenn--30B-A3B-nnssmttt Qwen3-30B- 30---30-B-BBBBaa
30.3%
Qwen3-30B-A3B -nst---t-tttt
43.5%
51.2%
A.1 SUMMARY
We mitigate potential benchmark contamination through both temporal controls-utilzzae a lat- est high-difficulty benchmark (Google-IMOS -M--AnseerBennnhh November 2025) together with earlier pre-trained base models (Llama3. 1----B8BBeaee and Dwen2.5-7B-Baat 2024)-and model diversity- validating across architectures spanning 1.7B to 30B parameters. The consistent and substantial per- formance improvements across all configurations validate the effectiveness of our fool-augmented data synthesis method and large-scale reinforcement learning on ultra-long sequences with massive tool calls, while confirming the reliability and robustness of our approach.
45

---

Published as a conference paper at ICLR 2026
A.12
BAANING-TESTING OVERLAP ANALYSIS
A.12.1 DATA DECONTAMINATION
To rigorously prevent data contamination between our training corpora (346K SFT and 42K RL samples) and the evaluation benchmarks (AIME24, AIME25, HMMT25), we employed a three- stage progressive deduplication pipeline.
Stage I: Problem-Level Exact Deduplication. We first remove all training samples whose prob- lem statements are exact string matches with those in the test sets.
4-gram MinHash LSH Similarity-Bast -Baee Deduplication. To further eliminate prob- lems with highly similar surface forms, employ combination of 4-gram analysis and MinHash Locality-Senne Sensitive Hashing (LSH):
Stage 2:
4-gram construction: For each problem, we construct n-gram set (n=4) by first tok- enizing the text (word-level tokenization for English; Jieba tokenizer for Chinese) and then extracting all consecutive 4-token sequences.
Similarity measurement: For any test problem A and training problem B, we compute the Jaccard similarity of their 4-gram sets:
B
(AA
A U B
where J(A, B) E [0, 1J, with higher values indicating greater n-gram-level overlap.
Filtering criterion: We set the Jaccard similarity threshold at 0.6. Any training sample with J > 0.6 relative to any test problem is removed.
Computational optimization: To efficiently handle large-scale deduplication, we em- ploy MinHash LSH, which maps MinHash signatures into hash buckets that similar texts are clustered together, transforming the problem from global pairwise comparisons into localized bucket-wise searches and substantially reducing computational complex- ity. This procedure is implemented using NLTK and the MinHashLSH module from the datasket library.
Stage 3: Semantic Embedding- Based Deduplication. To account for problems that exhibit sub- stantial lexical differences yet remain semantically equivalent, we perform an additional semantic deduplication step:
We generate embeddings for all problems in both training and test sets using the gte-large model.
We compute cosine similarity between each test problem and all training problems using the sentence transforrrssss library.
For each test problem, we rank all training problems by descending similarity and remove the top5 most semantically similar training samples.
Through this comprehensive three-stage pipeline, removed approximately 8.3K samples from the training set, ensuring that AgentMath's SFT and RL training data contain no overlapping or highly similar samples with respect to any evaluation benchmark.
A.1 EFFECT OF TRAINING DATA VOLUME
We further investigate how training data volume affects downstream performance through systematic scaling experiments for both the SFT and RL stages.
SFT Data Scaling, Using Qwen3-8B-Ban as the backbone, we vary the SFT training data from 2K to 300K samples. As shown in Table [8] the model exhibits significant and consistent perfor- mance gains across all benchmarks: AIME24 accuracy increases from 27.2% 78.4%, AIME25 from 21.1% to 72.2%, and the overall average improves from 17.9% to 59.8% (+41.9 pp). These results demonstrate that our tool-augmented data synthesis approach exhibits strong scalability.
46

---

Published as a conference paper at ICLR 2026
Table 18: Effect of SFT training data volume on benchmark performance.
SFT Data Volume
AIME24
AIME25
IMO-AnswerBenchh
Avg
SFT-2k
27.2
21.1
5.3
17.9
SFT-20k
60.5
53.3
12.8
42.2
SFT-40k
66.8
57.9
17.6
47.4
SFT-80k
69.2
61.8
21.4
50.8
SFT-160k
73.8
66.4
25.6
55.3
SFT-300k
78.4
72.2
28.8
59.8
RL Data Scaling. Starting from the best SFT checkpoint (300K), vary the RL training data from 10K to 42K samples while keeping the number of training steps (200) and all hyperparameters fixed. As shown in Table [9 the model demonstrates consistent improvements: AIME24 accuracy increases from 80.8% to 85.3%, AIME25 from 75.5% to 80.4%, and the overall average improves from 62.4% to 66.8%.
Table 19: Effect of RL training data volume on benchmark performance. The first row shows the SFT-only baseline.
RL Data Volume
AIME24
AIME25
IMO-AnswerBench
Avg
SFT-300k only
78.4
72.2
28.8
59.8
RL-10k
80.8
75.5
30.9
62.4
RL-20k
82.7
77.8
32.4
64.3
RL-30k
84.2
79.0
33.7
65.6
RL-42k
85.3
80.4
34.7
66.8
Combining both sets of scaling experiments, we observe that AgentMath exhibits sustained and consistent performance improvements as the tool-augmented training data size increases across both the SFT and RI. stages, further validating the strong scalability of our approach.
A.13 DATA SYNTHESIS METHODOLOGY AND ANALYSIS
The data synthesis process in AgentMath involves several key design choices, including computa- tional component segmentation and code complexity filtering. In this section, we provide detailed analysis and empirical justification for these design decisions, demonstrate scalability do- mains, and analyze the computational costs of our pipeline.
A.13.1 COMPUTATIONAL COMPONENT SEGMENTATION
Our segmentation strategy divides each long chain-of-thought response generated by DeepScek-RI into fixed-length chunks (e.g., 3k tokens), yielding N segments (S1, S2, S3, Sx). We then use DeepSeek- V3-0344 independently perform tool-augmented rewriting each segment. In 346k-example synthetic SFT dataset, responses have an average length of 18.3k tokens; thus, each response on average split into 6 segments.
Motivation. In early experiments, we attempted to rewrite the full DeepSeek-RI response without segmentation. This led to several issues:
DeepSeek-R1 responses are excessively long.
The instruction-following ability of DeepSeck- V3-0324 degrades markedly under ultra- long inputs, and the model becomes more prone to hallucinations.
Consequently, the rewritten data exhibits low frequency of tool usage, high code execu- tion failure rate, and extensive abbreviation or omission intermediate natural language reasoning steps (which refer to the "textual reasoning omission rate").
47

---

Published as a conference paper at ICLR 2026
To quantitatively assess the effect of segment length data quality and downstream performance, we conducted controlled experiments on 20k synthetic examples and performed SFT on Qwen3-8B. Base. As shown in Table 20, the results demonstrate clear trends:
No segmentation: The average number of tool calls is 1.6; the code execution failure rate is 31.8%; the textual reasoning omission rate is 78.5%; and AIME24/AIME25 accuracies are 40.3%/35.66%.
Segmentation with 9k tokens:
segmentation, but segments remain relatively long, so tool usage and evaluation performance are still limited.
This setting improves
Segmentation with 3k tokens: The average number of tool calls increases to 7.8; the code execution failure rate drops to 6.2%; the textual reasoning omission rate decreases to 2.3%; and AIME24/AIME25 accuracies reach 60.5%
Segmentation with 2k tokens: Although the number of tool calls further increases, down- stream performance does not improve significantly and appears saturate.
Based on this analysis, we adopt 3k-token segment length as the default configuration, as it strikes a favorable balance between data-quality metrics and downstream task performance.
Table 20: Effect of computational component segmentation on data quality and model performance using 20k AgentMath synthetic data.
Configuration
Calls
Codk Exccution
Reasoning Omission Rate
AIME24 Acc
AIME25
No Segmentation
1.5
31.8%
78.5%
40.3%
35.6%
Sogment- -k-ok-n
2.1
28.4%:
67.9%
44.8%
38.7%
Sogment-66
Segment-4ee -eet-----ooken
5.7
12.5%
8.8%
3%
$1.6%
Segment- Pk-token
8.3
6.2%
2.3%
60.5%
53.3%
Segment- 2k-token
10.1
5.9%
2.1%
59.45
51.9%
A.13.2 CODE COMPLEXITY FILTERING
We conducted statistical analysis of code-length distributions over 20k synthetic instances. The results showed that samples with fewer than 5 lines of code constituted 5% of the dataset, with exactly lines representing 7%, 6 lines 13%, lines 24%, lines 29%, 9 lines 13%, and 10 or more lines comprising 10%.
To investigate the relationship between code complexity and line count, we performed stratified uni- form sampling. drawing 40 samples from each line-count category (280 samples total). Three au- thors independently annotated each sample's complexity as "Easy," "Medium," " or "Hard" based on whether involved computationally demanding operations (e.g.. nurger arithmetie, complex equation solving, advanced linear algebra, combinatorial computations, or calculus). The annotation
statisties presented in Table 21
Our analysis shows that among samples with < lines of code, over 65% were labeled "Easy" while fewer than 10% were labeled "Hard." To prevent the model from learning to overuse tool invocations during training, we conservatively filtered out all samples containing 5 or fewer lines of code. In future work, we plan employ LLMs such Qwen3-30B for automated complexity assessment, replacing manual annotation to improve both objectivity and scalability of our filtering pipeline.
A. 1.13 SCALABILITY TO NEW DOMAINS
To demonstrate the generalizability of our approach, we extended AgentMath's automated data syn- thesis and cleaning pipeline to mon-mathematical domains, including physics, chemistry, and biol- ogy, and evaluated the resulting model on the GPQA Diamond benchmark. GPQA Diamond sesses large language models' ability to solve graduate-level problems across biology, physics, and chemistry highly challenging benchmark where even PhD-level domain experts achieve only 65% accuracy.
We randomly sampled 40K examples from the open-source Science 220K dataset released by AM- Thinking and applied AgentMath's 's data synthesis method. Specifically, we segmented DeepSeek- R1's 's's reasoning response into 3K-token chunks and transformed them into tool-augmented reasoning
48

---

Published as a conference paper at ICLR 2026
Table 21: Statistics of code complexity distribution by lines of code.
Lines of Code
Easy (%)
Medium (%)
Hard (%)
5 lines
87%
9%
4%
5 lines
66%
27%
7%
6 lines
43%
38%
19%
= 7 lines
24%
49%
27%
8 lines
13%
56%
31%
9 lines
7%
51%
42%
2 10 lines
4%
43%
53%
chains by replacing complex, error-promt computational steps with executable code. We then applied automated cleaning pipeline, which includes: (1) filtering examples with S 5 lines of code, (2) correcting formatting errors, (3) removing samples with failed code exccution, and (4) performing environmental feedback alignment ensure consistency.
Using Sive3-8B-BBase as the foundation model, we conducted SFT and evaluated performance on GPQA across independent runs, reporting the average accuracy (Avg@8). As shown in Table 22, AgentMath- SaattMatt--B-SF----A 8B-SFT-40k achieves 58.9% accuracy on GPQA, substantially outperform- ing -errsSee-RR--sstiil-----en augmented synthesis approach and automated cleaning pipeline generalize effectively beyond math- ematics scientific domains, highlighting the scalability and broad applicability our method.
(49.1%) These results demonstrate that AgentMath's tool-
Table 22: GPQA accuracy comparison of different models at the 7B-8B scale.
Model
Base Model
GPQA Ace
OpenThinker-77
Swen2.5-7B-Instruu
42.4%
AReal -beal--bbbaR11-7B
DeepSeek-RR - 1-Distill-Qwen-71
47.6%
DeepSeek-R1-De Ri-istilll -wenll-B-ee-BB
Qwen2.5-Mathh
49.1%
Light-R 1-7B
DeepSeek-RI-Disill-OUuee
I--istill-Qwen--
49.4%
AgentMath-8B-SFT--A
Qwen3-8B-Base
58.9%
A. 13.4 COMPUTATIONAL COST ANALYSIS
We build upon the open-source AM-Thinking dataset, leveraging their existing responses generated V3 for data synthesis,
by DeepSeek-R1 R10528 to obtain 346K data samples. We employ DeepSeek-VS V3 completing this process on 128 GPUs (each with 96GB memory) in approximately 62 hours. We then utilize Qwen3-30B as a judge model to verify consistency between actual code execution results and model-simulated outputs, requiring approximately 3 hours.
Subsequently, perform SFT on the 316K synthetic samples using OWen3-8B-Bass the base model, which takes approximately 44 hours. In total, the complete pipeline encompassing data synthesis, cleaning, verification, and SFT training-requires approximately 62 44 109 hours.
cost-effective alternative address computational overhead. As illustrated in Figure we apply SFT to only 20K samples starting from Qwen3-8B-Bass followed by 400 steps of large-scale reinforeement learning. This streamlined approach enables our Agent- Math model to achieve leading performance: 76.2% on AIME24 and 67.5% on AIME25, surpassing both CrenMath-Neeotter 7B (74.8% and 61.2%, respectively) and Qwen3-8B- ---nnkiininggggggg (76.0% and 67.3%, respectively). This entire workflow, including data synthesis, cleaning, SFT, and RL training, requires only hours total, providing a substantially more efficient solution.
Notably, also explore
A.14 TOOL-AUGMENTED DATA SYNTHESIS PIPELINE
This section details our augl-nugmented data synthesis methodology, including the segmentation strategy, code integration criteria, and teacher model selection.
49

---

Published as a conference paper at ICLR 2026
A.14.1 RESPONSE SEGMENTATION STRATEGY
We utilized long chain-of-thowe -fthoughtt responses generated by DeepSeek-RI from the open-source AM- Thinking dataset. Each response is segmented into fixed-length 3k-token chunks, yielding N seg- ments (S1. S2, S3.... ......... S,). Across the 316k SFT synthetic data, the responses have an average length of 16.9k tokens, corresponding to roughly six segments per response.
Subsequently, for each segment, we applied the Tool-Augmented Data Synthesis Prompt detailed in Appendix A.6.2 We employed DeepSeek-V3-nn V3-0324 to independently transform each segment by replacing complex and error-prone computational processes with executable code.
To accelerate the synthesis process, we initially instructed DeepSeck-V3-0524 execution results based on the textual reasoning context. During post-processing, we replaced these simulated results with actual code execution outputs and employed Qwen3-30B as a judge model to assess their consistency. Samples exhibiting inconsistencies were filtered out to ensure data quality.
V3-0324 to simulate code
A.14.2 CODE INTEGRATION CRITERIA
key design consideration is determining when code usage is beneficial within the reasoning chain. Our Too--Augeented Data Synthesis Prompt (Appendix A.6.2 guides DeepSeek-Var V3-0324 to first identify complex and error-prone computational processes, and then replace these manual calcula- tions with code. Specifically, it covers the following types of computational tasks:
Complex Symbolic Algebra: polynomial expansion, factorization, equation solving
Advanced Calculus: differentiation, integration, definite integral computation
Probability and Combinatorics: complex counting problems, probability distribution cal- culations
Linear Algebra:
matrix operations, matrix inversion, eigenvalue decomposition
Numerical Computation: approximation calculations, large number arithmetic, geometric calculations
Other Error-Prone Computaation-Ittesiiy Problems
To control code complexity and ensure the reliability of the synthesized data, we further filter out low-complexity samples with S 5 lines of code. Furthermore, we randomly sampled 100 instances from the synthesized data and manually annotated whether DeepSeek- V3 effectively replaced com- plex textual computational processes with code. Results show that code replacement is benefi- cial and reasonable 84% of the samples, thereby confirming the reliability of our proposed tool- augmented data synthesis approach.
A. 14.3 EFFECTIVENESS OF CODE-AUGMENTED REASONING
We demonstrate the effectiveness of code-augmented methods in replacing manual complex com- putations from multiple perspectives. As shown Table 23 we systematically compare code-based and pure text-based reasoning along two dimensions: inference efficiency (measured by average token count) and performance metrics.
Token Analysis on Synthetic Data: The SET training dataset comprises 316k instances in total. Pure text-based reasoning averages 18.3k tokens per sample, while tool augmen- tation reduces this to 16.9k tokens. This demonstrates that substituting code for lengthy computational processes effectively reduces overall token consumption.
SFT Stage: We comprehensively compare the efficiency and performance of code aug- mentation against pure text-based reasoning. Specifically, when trained on identical 20k SFT data, AgentMath-SSFT-20k achieves accuracies of 60.5% and 53.3% on AIME24 and AIME25, respectively, significantly outperforming Text-based-SFT-2200 (57.1 % and 49. (9) Additionally, SeentMtth-SFF2200 reduces the average token count by approxi- mately 1.3k-3k compared to -aa-sdd-FF--2k...
RL Stage: tool-augmented RL training. using only 440 steps, AgemMaathRRL-44 steps improves accuracy from 60.5% to 76.2% AIME24 and from 53.3% 67.5%
50

---

Published as a conference paper at ICLR 2026
AIME25. In contrast, pure text-based RL training requires 1600 steps (Text-based-Re RL- 1600=steps) to achieve improvements from 57.1% to 68.7% on AIME24 and from 49.2% to 57.5% on AIME25. These results demonstrate that the code-augmented approach achieves approximately 4x higher training efficiency while reducing the overall token count by 2.5k-5k compared to Sext-based-RL-11000---
In summary, introducing code augmentation in both SFT and RL stages yields substantial improve- ments in performance and efficiency over pure text-based reasoning, thereby validating that using code is more effective for complex computational segments of the reasoning chain.
Table 23:
Accuracy and average inference token usage on AIME24 and AIME25 for pure text-based models and code-augmented AgentMath both SFT and RL stages.
Model
Base Model
AIME24 Ace
AIME24 Avg Tokens
AIME25 Ace
AIME25 Avg Tokens
Text SFT:2k
B.ven3-8Fi 19-Base
57.1%
29k
49.2%
28k
AgentMath-SFIT -FF-22kk
Qwen3-8B-Band
60.5%
26k
53.3%
26.7k
Seat-bssed-LL. -t-pps0ssssss
Text best-based--o SHFT-2UK
32.3K
31.2k
AgontMath-RL -40-Stpss
AgentMatb-SFF
76.2%
27.2k
67.5%
28.6k
A.14.4 TEACHER MODEL SELECTION
Different models may prefer distinct code-usage patterns for the same question, including variations in code injection positions, line counts, and code implementation details. To identify the optimal teacher model for our data synthesis pipeline, we conducted a comparative study between two can- didate models: DeepSeek-V V3-0324 and (wen3--351B-Instruett.
Specifically, we synthesized training data using each model on the 20k training set and performed supervised fine-tuning on Qwen3-8B-Batt
As shown in Table 24 data synthesized by DeepSeek- V3-0324 averaged 6.5 tool calls per sample, compared to 5.3 tool calls per sample for Qwen3- 235B-Instrruc-25577 On the AIME24 and AIME25 benchmarks, the model trained on DeepSeek- V3-0324-synthesized data (AgentMath- MagentMMth-20k-SSFE 53.3%, respectively, marginally outperforming the model trained on Qwen3-2355 235---32tttttt Instruct-2507- synthesized data (59.3% and 51.7%). Notably, the former also demonstrated a higher average tool call frequency.
V33) achieved accuracies of 60.5% and
Based on these comprehensive performance results, we selected SeepSeek-V3-0324 as the teacher model for our data synthesis pipeline.
A.15 FAIRNESS OF COMPARISON WITH TEACHER MODEL DISTILLATION
The dataset synthesis in AgentMath employs larger teacher models (DeepSeek RI and V3, Qwen- 30B), which may imply knowledge distillation from more powerful teacher models. To address potential concerns about the fairness of comparing such distilled models with those trained without teacher guidance, we conduct controlled experiments to isolate the contributions of our proposed method.
Controlled Comparison with Retool. To ensure a rigorous and fair comparison, we strictly fol- lowed the experimental setup of Retool, controlling for all key factors: the same 2k synthetic dataset, identical teacher model responses from DeepSeek-RI. 1, the same base model (Qwen2.5- Instruuct,,,,, and 400 training steps during the RL phase. The sole difference lies in the data synthesis strategy: AgentMath leverages our proposed lool-augmented data synthesis method (Ap- pendix A.6.2), whereas Retool employs its original synthesis prompt. Both approaches undergo identical two-stage training comprising SFT and RI., As shown in Table 25 the experimental results demonstrate that:
AgentMath-3228 32B-SFT-2k achieves 44.1% AIME24 and 37.3% on AIME25, surpassing Retool-32B- 3-F--B--B-- (40.9% and 34.5%, respectively), thereby demonstrating the effective- ness of our tool-augmented data synthesis approach;
51

---

AIME25 Acc Published as a conference paper at ICLR 2026
Table 24: Comparison Qwen3-8B- models fine-tuned the 20k AgentMath SFT dataset synthesized by DeepSeck-V3-ne V3-0324 and (onn3--33--Instrutt-222 AIME25 accuracies (%) and the average number of tool calls per problem on each benchmark.
We report AIME24 and
Model
AIME24
AIME24 Tool Calls Avg
AIME25
AIME25 Tool Calls Avg
AgemMath- Satth-20-SSS--eSSSeeee 3FT-DeepSeek-V3
AgemiMath 20k SPT Qwen3 235B Instract-
59.3%
4.4
$1.7%
5.0
AgentMath-32BB attains 74.8% on AIME24 and 56.6% on AIME25, consistently out- performing Retool-32B-RR (67.0% and 49.3%, respectively), further validating the superi- ority of our reinforcement learning training strategy.
These results confirm that AgentMath exhibits consistent and substantial performance gains across both training stages, validating the efficacy and robustness of our proposed method independent of teacher model choice.
Table 25: Comparison of AgentMath and Retool performance under the same experimental setup in both the SFT and RL stages.
Model
AIME24 Acc
Retool-32B-SSFT--
40.9%
34.5%
AgentMath-32B-99FT-2K
44.1%
37.3%
Retool-32B-RL.
67.0%
49.3%
AgentMath-3255
74.8%
56.6%
Impact Teacher Model Distillation on Competing Methods. To investigate whether the per- formance gap would shrink if competing models were also allowed to distill from comparable teacher models, we adopted the open-source dataset from AM-Thinking, using the same 316k prompts and responses generated by DeepSeck-RI. We conducted supervised fine-tuning (SFT) on the Qwen3-8B-Ban model for 6 epochs with learning rate of 6e-5, To ensure experimental rigor, all configurations were kept identical except for the core difference in reasoning chain types: AM- Thinking uses pure textual reasoning chains, while AgentMath employs cool-augmented reasoning chains. We conducted 8 independent training runs and report pass@) results from the checkpoint with the best validation performance.
As shown in Table 26 AgentMath-Se -B--h- achieves 78.4% accuracy on AIME24 and 72.2% aceu- racy on AIME25, outperforming AM--hinking--BB- 8B-SFT (74.9% and 67.4%, respectively), with an overall average performance gap of 4.1%. These results confirm that while the performance gap shrinks when competing models also distill from comparable teacher models, AgentMath consis- tently maintains its advantage due the inherent benefits of augl-aug reasoning chains.
Table 26:
Comparison of pass @1 accuracy AIME24 and AIME25 for AM-Thinking- 8B-SFT and AgentMath--e 8B-SFBT under the same training settings.
Model
AIME24 Ace
AIME25 Acc
Avg Score
74.9%
67.4% 72.2%
71.2% 75.3%
AgentMath--DD-SFT
78.4%
16 REWARD DESIGN IN RL
Our reward function consists of two components: an answer correctness reward Ras: and tool usage efficiency reward Rmot The correctness reward Race provides binary (0,1) feedback based mathematical equivalence (verified using the math_verify library):
1, if iq.eeiivale a),
Race
0, otherwise,
52

---

Published as a conference paper at ICLR 2026
where à denotes the predicted answer and the ground truth answer. Conditioned answer rectness, the tool usage reward Rico incentivizes efficient utilization of computational resources:
Raoal
min Rauxx a + 3 Nudde) if Neude > 0,
where is the base tool usage reward, 8 is the scaling factor for the number of invocations, and Rmax is the reward upper bound. The total reward function is defined as:
Rioal = Race + 1(Bas = 1) Rioo--
In our work, we set 0.1 and B 0.01, and constrain the maximum tool reward Raax 1, which encourages tool usage while preventing over-reliance. This design is motivated by the following considerations: applying Redd only to correct answers encourages the model to use tools appropriately to arrive at accurate solutions, avoiding rewarding erroneous tool invocations; = 0.1 differentiates between cases with and without tool usage, while the smaller 0.01 accommodates high-frequeee tool calls during RL training (some exceeding 64 invocations), preventing reward inflation.
To validate the effectiveness of the reward design, we conducted controlled experiments using the SFT model trained on 20k synthetic data (AgentMath- -aaatttMt---F-20k)) SFT-20k), with a fixed 100 training steps, as shown in Table 27 Our experiments demonstrate:
Effectiveness of tool rewards: Introducing tool rewards (all answers) outperforms the tool-reward baseline in both average accuracy (61.4% vs. 60.2%) and average tool calls (6.2 5.7), indicating that tool rewards effectively incentivize the model leverage com- putational tools and improve performance.
Advantage of tool rewards (correct answers only): Furthermore, compared to tool re- wards (all answers), applying tool rewards exclusively correct answers (a 0.1, 0.01) achieves the best performance in both average accuracy (62.6% vs. 61.4%) and aver- age tool calls (6.3 6.2).
Optimal hyperparameter configuration: In grid search over (0.1,0.2) and B {0.01, 0.02]. & = 0.1, 3 = 0.01 achieves the best accuracy-effieinng trade-off (62.6% average accuracy, 6.3 average tool calls).
Therefore, based on performance and tool call metrics, ultimately adopt the correctness- conditioned tool reward design with parameters = 0.1 and B = 0.01 for RL training.
Table 27: Ablation study results on reinforcement learning reward design.
Reward Configuration
AIME24
AIME24 Tool Calls
AIME25 Accuracy
AIMEIS Tool Calks
Arg. Accuracy
Asp
Accuracy
Tool Calls
Math SFT-20k (Baseline)
60.5%
5.1
53.3%
3.7
56.9%
5.4
RL. with No sool rewaed
65.25
5.5
55.15%
5.9
60.2%
5.7
RL. with Tool reward answens, 0.1.
66.4%
3/
61.4%
6.2
RL Tool reward (correct only). 3
67.4%
57.7%
6.3
Toal
only).
$6.89%
61
reward (cornct answer only), o 0.01
56.5%
53


Reviews:
[{"review_id": "SVbIYfSuW1", "reviewer": "Reviewer_iSnP", "summary": "This work introduces AgentMath, a framework that enhances the mathematical reasoning ability of large language models by combining their natural‐language chain‐of‐thought with precise external tool use (e.g., code execution). The authors propose (1) an automated method to convert natural language reasoning traces into structured tool-augmented interaction trajectories (for supervised fine-tuning), (2) a multi-turn agentic reinforcement‐learning paradigm that interleaves language reasoning and real‐time code/tool invocation to learn optimal strategies for tool use and error correction, and (3) a highly efficient training infrastructure (with asynchronous rollout scheduling, partial rollouts, and prefix‐aware load balancing) enabling meaningful scaling on ultra-long sequences with many tool calls. The result is a model better able to tackle complex mathematical problems beyond pure language reasoning—off-loading heavy computation to tools while retaining robust step‐by‐step reasoning in language.", "questions": "My major concern, as mentioned in weakness, is on the technical innovation of this paper, which I would love to hear more from the authors.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper did a good work in presentation. All the components in the pipeline are illustrated in good details and intuitions. This provides a good recipe for open-source math prover training.\n\n- A very comprehensive comparison between the model trained from this work and other open/closed-source models are provided, making the results very convincing.", "weaknesses": "- The paper's contribution is mostly on the development of the entire pipeline from my perspective. As many technical innovations claimed in the paper are either standard or mostly from engineering aspects, as agentic RL training is not a fresh concept nowadays (including coding-assisted math reasoning). I value a lot of the paper's efforts on developing such a comprehensive pipeline (SFT data collection and large-scale RL system building), which I understand is very challenging and helpful for the open-source community; however, the lack of technical contribution is the major factor preventing me providing a higher score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AoZJOVIbxk", "reviewer": "Reviewer_xGwU", "summary": "The paper proposes AgentMath, a tool-augmented agentic framework for competition-level math reasoning. It tackles three pain points: (i) scarcity and noisiness of tool-use data; (ii) lack of agentic RL that interleaves generation with code execution; (iii) infrastructure bottlenecks for ultra-long contexts and many tool invocations. Concretely, it introduces an automated tool-augmented trajectory synthesis pipeline (code injection → executability verification → environment-feedback alignment → self-correction examples), an agentic RL scheme (GRPO-style optimization with interleaved code execution and selective masking of environment outputs; a reward combining correctness with tool-usage efficiency), and a scalable training system (request-level async scheduling, agentic partial rollout, prefix-aware weighted load balancing). On AIME24/25 and HMMT25, the approach reports strong results; e.g., AgentMath-30B-A3B = 90.6/86.4/73.8%, competitive with proprietary frontier models and surpassing open counterparts of comparable size. The system ablations claim 4–5× end-to-end RL throughput gains from the engineering stack", "questions": "See my comments on Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The refinement pipeline (format consistency, executability checks, feedback alignment, self-correction) shows sizable SFT gains and scaling trends before RL.\n2. The async scheduler + partial rollout + prefix-aware balancing reportedly lift RL throughput 4–5×, and the paper provides a breakdown and sensitivity to segment count.\n3. The 30B model achieves 90.6/86.4/73.8% on AIME24/25/HMMT25", "weaknesses": "1. Novelty over prior tool-augmented reasoning may feel incremental.\nThe community already has multiple tool-augmented RL frameworks (e.g., ReTool-style RL teaching strategic tool calls) and mature RL infra with asynchronous rollout and truncation/partial-trajectory techniques (e.g. AREAL, ROLL). Much of AgentMath’s lift appears to stem from more elaborate SFT data synthesis and a careful infra implementation rather than a fundamentally new RL principle. From a novelty lens, the async scheduler and partial rollout echo patterns well-established in modern RL systems; similarly, the use of truncated/segmented rollouts conceptually aligns with prior truncated policy optimization ideas (e.g. Truncated Proximal Policy Optimization). The work’s value is therefore primarily systems consolidation + scale rather than conceptual RL novelty.\n2. I could not find a code repository or an explicit release statement in the current manuscript (PDF). Given that a large portion of the contribution is engineering (decoupled async architecture, agentic partial rollout, prefix-aware balancing, data-synthesis tooling), lack of code release significantly limits reproducibility and adoption. Please clarify whether code, data pipelines, and training/eval scripts will be released (e.g., upon acceptance) and provide a URL if available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HtDumXJnpv", "reviewer": "Reviewer_5L8W", "summary": "The paper presents AgentMath, an agentic framework integrating code interpreters with LLMs to enhance math problem solving and mathematical reasoning to be more accurate and computationally efficient through tool-augmented learning. It contains two major contributions. The first is the introduction of a three-step data synthesis pipeline for generating tool-augmented training trajectories. The second is a reinforcement learning (RL) techniques that combine natural language reasoning with symbolic code execution. To accelerate RL training, the authors design an efficient infrastructure leveraging asynchronous rollout scheduling and adaptive load balancing. The paper primarily uses math reasoning benchmarks (AIME24, AIME25, and HMMT) for experiments, and reports commendable improvements on those datasets over baseline models/approaches.", "questions": "1. Given that AIME24, AIME25, and HMMT each contain fewer than 30 questions and have been reported as leaked in prior work, how do the authors ensure that these benchmarks were not inadvertently seen during pretraining or fine-tuning? Have the authors considered using larger or newly constructed evaluation sets to validate the robustness of the results?\n1. With 346k training questions for SFT and 42k for RL but fewer than 100 test questions, how do the authors ensure that the training data do not overlap or contain structurally similar questions to the test set?\n\n1. What exact n-gram filtering algorithm and similarity threshold were used to remove overlapping data?\n\n1. Have the authors conducted ablation studies to quantify how training data volume affects final test accuracy? Would performance remain stable if a smaller, filtered subset of training data were used?\n\n1. Many steps in the data synthesis pipeline involve human heuristics such as computational component segmentation and code complexity filtering. How much manual adjustment or iteration do these steps require?\n\n1. Can the synthesis and cleaning pipeline be generalized or automated for other benchmarks or non-math domains?\n\n1. What is the total computational cost (e.g., GPU hours or LLM API calls) associated with synthesizing and verifying the datasets?\n\n1. How are textual reasoning steps segmented into computational components when deciding where to inject code? What criteria or metrics determine whether using code is more effective for a given part of the reasoning chain?\n\n1. Since different models can prefer different code usage patterns for the same question, how robust is the segmentation procedure across models?\n\n1. The paper uses large teacher models (DeepSeek R1, DeepSeek V3, and Qwen-30B) for data synthesis. How do the authors ensure fairness when comparing with models trained without such teacher guidance? Would the performance gap shrink if competing models were also allowed to distill from comparable teacher models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper conducts an important study on integrating the Code Interpreter into LLMs to improve mathematical reasoning via symbolic computation.\n\n* The data synthesis pipeline and training strategies (including the code execution sandbox and adaptive load balancing) are technically sound and thoughtfully designed to improve training efficiency.\n\n* The proposed RL with Code Interpreter integration is shown to be effective through extensive experimental results.\n\n* The experiments are extensive and systematically conducted.\n\n* The paper is well written and easy to follow.", "weaknesses": "* Concern about benchmark: The math benchmarks (AIME24, AIME25, HMMT) each contain only about 30 questions. Prior work has shown that these datasets may have leaked into open-source model training corpora. Thus, it is unclear whether the reported results are entirely reliable. The authors are encouraged to provide more evidence on the independent contribution of the proposed approach.\n\n* Training-testing overlap: The authors report 346k training questions for SFT and 42k for RL, while the total number of testing questions is under 100. It is questionable whether the training data already contains components or similar structures to the test questions. Although an n-gram filtering algorithm is mentioned, the specific implementation details are missing. Have the authors performed ablation studies to evaluate the effect of training data volume on test performance?\n\n* Unclear heuristic data synthesis: The data synthesis process involves multiple human heuristics (e.g., computational component segmentation, code complexity filtering), and several components appear to require iterative manual adjustment. These steps may limit scalability to new benchmarks or domains. Moreover, the data synthesis and cleaning costs seem high given the repeated use of LLMs.\n\n* The paper does not clarify how textual answers are segmented into computational parts or how the model determines when code usage is beneficial. Different models may prefer distinct code-usage patterns even for the same question.\n\n* The dataset synthesis employs larger teacher models (DeepSeek R1 and V3, Qwen-30B). This may imply knowledge distillation from more powerful teacher models. Comparing such distilled models with those trained without teacher guidance may be unfair.\n\n* The authors should consider moving key details, such as dataset sizes for SFT and RL, to the main paper because they are critical for comprehension.\n\n* Many designs in the algorithms, such as the four quality refinement modules in Step 2 of the data set synthesis, and reward design in RL, need further justification or ablation studies. Otherwise the overall method seems to contain too many ad-hoc details that may limit the generalizability of the proposed approach.\n\n* Typo: In line 1600, the figure reference is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)