SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning
Authors: Zhenghai Xue
Longtao Zheng
Qian Liu
Yingru Li
Xiaosen Zheng
Zejun MA
Bo An
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Tool Integrated Reasoning
Reinforcement Learning
Code Generation
LLM Reasoning
Primary area: reinforcement learning

Abstract: Large Language Models (LLMs) can enhance their reasoning by interacting with external tools, a paradigm known as Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn settings using Reinforcement Learning (RL) often exhibits training instability and degraded performance. We attribute the instability to harmful negative samples resulting from distributional drift and compounding errors induced by using external tool outputs during multi-turn rollout. To address this issue, we introduce SimpleTIR, a simple method that stabilizes multi-turn TIR training via filtering out trajectories with "void turns", i.e., turns that yield neither a code block nor a final answer. Specifically, we remove those trajectories from the policy update to block harmful gradients, while retaining them in advantage estimation to keep the estimate unbiased. Extensive experiments show that SimpleTIR effectively mitigates gradient norm explosion and stabilizes multi-turn RL training from base models. It achieves state-of-the-art performance on challenging math reasoning benchmarks, including an AIME24 score of 50.5 starting from the Qwen2.5-7B base model. SimpleTIR also promotes more diverse reasoning behaviors such as self-correction and cross-validation, outperforming prior methods trained from stronger instruction-tuned models.

Published as a conference paper at ICLR 2026
SIMPLETIR: END-TO-END REINFORCEMENT LEARN- ING FOR BULTI-TURN TOOL-INTEGRATED REASONING
Zhenghai Xuel*
Longtao Zheng!
Qian Li
Yingru Li?
Xiaosen Zheng²
Zejun Ma² Bo An
Nanyang Technological University, Singapore
TikkTok, Singapore
ABSTRACT
Large Language Models (LLMs) can significantly improve their reasoning capa- bilities by interacting with external tools, a paradigm known as Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios using Rein- forcement Learning (RL) is often hindered by training instability and performance collapse. We identify that such instability primarily caused by a distributional drift from external tool feedback, leading to the generation of extremely low- probability tokens. This issue compounds over successive turns, causing erroncous responses and catastrophic gradient norm explosions. Through careful inspections, we find phenomenon named "void turns" as preferable sign of pathological generations, where neither code block nor final answer is yielded. We introduce SimpleTIR, a plug-and-play algorithm that identifies and filters out trajectories containing void turns when performing the policy update. SimpleTIR effectively blocks the harmful, high-magnitude gradients, thus stabilizing the learning dy- namics. Extensive experiments show that SimpleTIR achieves state-of-the-arr tf-tth-art performance on challenging math reasoning benchmarks, notably elevating the AIME24 score from text-only baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model. Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR encourages the model to discover diverse and sophisticated reasoning patterns, such as self-correction and cross-validation.
SimpleTIR
Naive Multi-tumn Training
Baseline (DAPO)
50
45-
Averoge AIME24 Accuracy
40
30
Gradient Norm
50
25-
20
15
400
1200
400
1200
1600
Training Steps Reinforcement Learning
Training Steps Reinforcement Learning
Figure 1: Starting from Qwen2.5-7B base model, The training dynamics of SimpleTIR highly stable, and clearly outperforms the baseline method without TIR (DAPO). The gradient norm remains bell-behaved with almost no spikes. contrast, Naive Multi-turn Training not only suffers from unstable dynamics catastrophic gradient norm explosions, but also fails to match the performance of the baseline without TIR.
'The first two authors contribute equally.
1 'Correspondence: Qian Liu (liuqian.ser seanesssaaeee (aaaammmml.mmmmmm

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Training Large Languagee Models (L.LMs) from pretrained models using only outcome-basee re- wards, paradigm known Zero Reinforcement Learning (RL), represents a fundamental step towards autonomous agents and has garnered significant attention since the success of DeepSeek- R1 (DeepSeek-AI Team 2025). By freeing models from carefully selected Supervised Fine- tene-ttning (SFT) data, Zero RL holds the potential to unlock more general problem-solving capabilities and fos- ter the emergence of diverse, novel reasoning patterns. A critical frontier for realizing this paradigm's 's potential is enabling LLMs to perform multi-turn Tool-Integrated Reasoning (TIR), where models iteratively reason, generate code, execute it, and use the output for informed reasoning in subsequent turns. TIR directly addresses inherent LLM limitations such as poor computational accuracy and knowledge cutoffs by allowing them to leverage external tools like Python interpreter or search engine. Despite its clear potential, training LLMs for multi-turn TIR with Zero RL remains highly challenging due frequent instability and gradient explosion issues (Liu et al. 2025a Mai et al.)
2025 Baronio et al.) 2025 Moonshot AI, 2025; Li et al. (222c)). One common solution is to "cold start" the model with distilled TIR trajectories to enhance stability (Feng et al. 2025). However, this SFT-based approach fundamentally violates the core philosophy of Zero RL, constraining the model to cuuman-annotall patterns and thus preventing the discovery of novel reasoning strategies.
In this paper, our key insight lies in identifying the root cause of this instability: the emergence and accumulation of extremely low-p probabibbiit tokens stemming from a distributional drift. When external tool feedback is used as model input in multi-turn TIR, such input may deviate from the model's pretrained data distribution. Although the tool feedback itself is masked when computing the policy loss (Jin al. 2025 Mai et al., 2025), the model's subsequent generations inherit this distributional shift, leading to increased stochasticity. Consequently, the model is more likely to sample low cow-oobbyyllity tokens. This issue is compounded in the multi-turn loop, as these low- probability tokens are fed back as input, exacerbating the distributional shift in subsequent turns. We make theoretical analysis of the gradient norm on softmax logits and find two dominating terms with negative correlations to token probabilities. This explains the gradient explosion issues observed in prior work.
Building on this analysis, we propose SimpleTIR, an effective trajectory filtering algorithm that stabilizes multi-turn TIR training. We observe that the accumulation of low-probability tokens and high generation stochasticity frequently results in what we define as void turn: an LLM response that contains neither a complete code block nor a final answer. It serves as a powerful heuristic indicator for identifying problematic trajectories. The core strategy of SimpleTIR is to filter out trajectories containing void turns. By excluding these trajectories from the policy loss computation, SimpleTIR blocks the harmful, high-magnitude gradients associated with the problematic low- probability sequences, directly addressing the gradient explosion issue. This filtering approach is general and plug-and-play, making a practical solution for stabilizing Zero RL training different scenarios without significant engineering overhead.
To demonstrate the effectiveness of SimpleTIR, we conduct comprehensive experiments on chal- lenging mathematical reasoning tasks. SimpleTIR establishes new state-of-the- on a suite of challenging mathematical reasoning benchmarks, improving the AIME24 score from 22.1 50.5 on Qwen2.5-7B and outperforming prior methods trained specialized data. Our ablation studies confirm that filtering trajectories with void turns is the crucial component for stabilizing training, overcoming the instability that plagues naive multi-turn approaches and enabling significant perfor- mance gains. We also highlight key advantage of the Zero RL approach. In contrast to methods that rely "cold-start" SFT phase, SimpleTIR encourages the model to discover novel and diverse reasoning patterns such as cross validatin,,,,onn, progressive reasoning, and self-correction, facilitating future data-based researches.
2 PRELIMINARIES
End-to-end training of multi-turn TIR agents with RL is challenging due to its compositional structure. We therefore model the process Hierarchical Markov Decision Process (Hauskrecht et al. [1998). which separates cicisiionmmaiing into two levels: a high-level policy governing the sequence of conversational turns and low-level policy for generating tokens within each turn.
2

---

Published as a conference paper at ICLR 2026
2.1 HIERARCHICAL MDP FORMULATION FOR MULTI-TURN TIR
A full interaction trajectory is a sequence o - (g, lo, lo....,Ik-.. -1, IK-I.,,. where 4 is the model's generated response turn and f& is the subsequent tool feedback. The high-level MDP, Mr AH, TH, RH, (H), operates at the turn level to govern the overall strategy. It includes the state Sk (q, lo. fo .o......... 1,.11,111111 fa-1) representing the complete conversation history before the current turn, the action A& indicating an option, or high-level sub-policy, the transition Th: Sa+1 = the final trajectory based on its overall success. The low-level MDP, _M2- = (St. AL, Tr, RL, ? operates at the token level to execute the chosen high-level action (Li et al. 2024), including the state s; = S o (aI,.... dt-1) indicating the sequence of tokens generated so far within the current turn k, the action d, E Ar. which is single token selected from the model's 's vocabulary, the transition TE: $1+1 = S o ae. and the reward Ry = 0. The low-level policy receives intrinsic reward, as its only goal is complete the high-level action.
Sk ft), and the terminal reward assigned
In this framework, o denotes concatenation, and we set the discount factor y = "H = YL 1. We train a single, unified policy Ta (se) to implicitly solve this two-level problem.
2.2 JOINT POLICY OPTIMIZATION AND FEEDRACK MASKING
The unified policy is trained using Group Relative Policy Optimization (GRPO) DeepSeck-Al Al Team 2025). GRPO circumvents the need for learned value function (Li al. 2024) by calculating the ad vanaage based on the relative performance within group of G trajectories sampled from the same prompt. The advantage for trajectory is A; = mcan (fr; 1 }=1) where is the terminal reward from the high-level MDP.
A critical adaptation is required for the TIR setting. The policy is only responsible for generating response tokens (4&), not the mnvironment-prvidded feedback tokens (fic). To ensure correct credit assignment, we employ feedback token masking (Jin et al. 22025; Mai al. (2025). The loss is accumulated only over the timesteps corresponding to the agent's actions, effectively excluding feedback tokens from the gradient computation.
This leads to our final training objective, JII((()
lad mix LCUP((,i,
1 T
JTIR (0) E
go-gos {or) 1 ae (a (1a)
Mi,t
where mi, is a binary mask that is 1 if the token at step belongs to any response 4g and 0 otherwise. The term LeuP is the standard clipped surrogate objective from PPO (Schulman et al.) 2017 Wang
=)Ai) with the importance et al. 2019): LCLIP(O,i,)) i,t) min (O), cli -, e,
ro(0s,)los,s) sampling ratio (0) Fezu (ot,elon.c))
METHODOLOGY
In this section, first identify a certain abnormal model behavior specific to multi-turn TIR: the emergence of low-probability tokens. We further interconnect such behavior with the phenomenon of unstable multi-turn TIR training. It is demonstrated how these tokens drive gradient explosions and create a credit assignment dilemma during Zero-RL training. To alleviate the influence of low- probability tokens, we propose an effective trajectory filtering approach, leading to the SimpleTIR algorithm that stabilizes multi-turn training and ensures that successful early turns are not penalized for later collapse.
3.1 THE EMERGENCE OF PRW-PRRAYIILITY TOKENS IN MULTI-TURN TIR
The instability of multi-turn TIR training is known challenge Wang et al. 2025 Mai et al. 2025 yet its root cause remains largely underexplored. Instability can plausibly arise from two stages: the choice of using tools (before tool calling) and the follow-up analysis of tool feedback (after tool calling). We contrast multi-turn TIR with a minimal, single-turn TIR setting where the model produces exactly response containing reasoning and an optional code block. This single-turn
3

---

Published as a conference paper at ICLR 2026
Noive Hulti-turn Training
Nalve Single-turn Training
1750
Resporee Length
1500
Code Use Frecencyy
NHHE2 Score
1250
1000
500
Steps
Steps
Steps
Steps
Figure 2: Training statistics comparing naive single-ture turn and multi-turn TIR. Single-turn training proceeds
smoothly and achieves higher performance, while multi-turn training is unstable.
Token Probability (Log (Log Scale)
-2
-10
Turn 1 with Tool Call
Turn with Tool Call Turn 3 without Tool Call
Turn 4 with Collapsed Response
Figure Visualization of token probabilitics in multi-turn TIR trajectory. The y-axis is log-scaled. Distribu- tional drift from tool feedback in carly turns leads to collapse in token probabilities in later turns. The detailed responses are in Tab. of Appendix A.1
setup involves no analysis tool feedback, thereby isolating the pre-tool-call reasoning phase. As shown in Fig. 2 the single-turn baseline trains smoothly and achieves reasonable performance. whereas the multi-turn equivalent suffers from performance collapse and recurring gradient spikes.
This comparative analysis indicates that the key factor contributing instability lies in the tool feedback loop. In multi-turn TIR, the tool feedback f* from turn & is concatenated into the prompt for turn k + Because this feedback originates from an externl interpreter, can deviate significantly from the LLM's pretrained data distribution in Zero RL. Conditioned on such out-of oftto--disttibution (OOD) input, the model's subsequent generations can drift away from pretrained patterns, becoming highly stochastic and assigning anomalously low probabilities to selected tokens.
We verify this claim with a case study in Fig. We train the multi-turn TIR with the original GRPO algorithm for 20 steps and select trajectory with erroneous response According to the figure, the tool feedback after Turn and Turn 2 contains tokens with extremely low probabilities, confirming their OOD nature. While masking the loss on these feedback tokens is known practice (Jin et al.) 2025] Mai et al. 2025), is insufficient. The distributional drift induced by these tokens contaminates subsequent model generations. As seen in Fig. 3, while response token probabilities in Turn are relatively high, low-probability segments emerge the model's own text in Turns and 3. This compounding drift culminates in collapsed, nonsensical response with extremely low token probabilities in Turn 4.
3.2 How LOW-PROBABINE PROBABILITY TOKENS COMPROMISE ZERO-RE TRAINING
After identifying the emergence of low-probabilit tokens, we analyze their two primary detrimental effects on Zero-RL training: gradient explosion and misaligned credit assignment.
Gradient Explosion. A primary failure mode in multi-tur TIR is the explosion of gradient norms (Fig. 2). Intuitively, as the token probability Told appears in the denominator the PPO ratio term,
Detailed responses in this be found in Tab. in Appendix A.1
4

---

Published as a conference paper at ICLR 2026
End-to-End Multi-Turn Agent Training
Agent-Env Interoction
Trajectory Filtering
Reinforcement Learning
Code
Rossoring incomplete
Pecsoning
Rocsoning
Codo
Redsonin
Agent
Result
Code 2
Hasulls
Muttiple
Vold Turn Stip Censrcctio
with Volid Tragectoru
Policy Upcate
Figure 4: An overview trajectories that contain void lurn LLM response that fails to produce either complete code block or final answer.
of SIMPLETIR. During the policy update, SIMPLETIR identifies and filters out entire
low-probability tokens can directly lead to the high policy gradients. To formalize this, we analyze the policy gradient with respect to the pre-softmax logits z following Li (025)). This gradient component of the total gradient with respect to model parameters.
Proposition 1. Consider token at timestep t of trajectory o,. The L2 norm of the policy gradient with respect to the logits 2₁ is:
E, mi Pt,) Ii. [Ail fi- 2P(c) + Σ P()
Mi.t
V JTRRll2
(2)
JEA
where ma, is the feedback mask, (0) the importance ratio, |Ail I is the absolute advantage, P is the policy's probability distribution To Te( (-loi,ct), and Ji.. is gating function active when the PPO update is not clipped.
Proposition 1 reveals that the gradient norm is highly sensitive to two factors that are exacerbated by low-probability tokens:
Unclipped Importance Ratio: The ratio (t,(()) = se(osalo is primary source of gradient new (o1
spikes. For negatively-rewarded trajectory (Á; O), this ratio is unbounded from above. If token e was generated with a very low probability by the old policy, Te., (cl-) is minute. Even a small update to (c)-) can cause (0) explode, leading to the gradient spikes observed in training.
term, / 1 2P(c) + E, P(j),
Sustained High Gradient Norm: The rroaabllity-dependee
can sustain large gradients. When the policy assigns a low probability to the sampled token 1 2c(c) nears maximum of 1. If the policy otherwise confident (i.e., the distribution is sharp), the collision probability Σj P(j)* remains large, preventing the gradient norm from diminishing and thereby contributing to unstable training Li 2025).
Misaligned Credit Assignment. Beyond gradient instability, low-probability tokens introduce severe credit assignment problem. As seen in Fig. 3 these tokens are more prevalent in later turns. With a sparse, terminal reward, a trajectory that fails in its final turns receives a single negative reward for the entire sequence. This signal does not distinguish between correct, high-pro grgghhrbbbbility reasoning in early turns and the faulty, oow-prbbbbility tokens that caused the eventual failure. This dynamic unfairly penalizes valid multi-turn behavior, causing the policy to collapse toward safer, single-turn generations.
3.3 SIMPLETIR: STABILIZING TRAINING BY FILTERING VOID TURNS
Given that low- wow-probabilityy tokens the root cause, simple heuristics like masking hegh-perplexity trajectories or clipping the importance ratio may seem appealing. While effective in some con- texts (Zheng al. 2025a| Zhang et al. 2025). show in Fig. 5 (bottom) that these methods fail to resolve instability in multi-tum TIR, as their thresholds are difficult to tune and they do not adequately address the credit assignment problem.
more robust filtering criterion needed. We observe that the collapsed Turn 4 in Fig. 3 follows a turn that produced neither tool call nor final answer. Intuitively, such turn makes no progress in
5

---

Published as a conference paper at ICLR 2026
the reasoning process and should not appear in an effective TIR trajectory. We define these turns as void turns, with incomplete code blocks being common causes. Concrete examples of trajectories containing void turns can be found in Appendix [ [..]] Void turns are often symptoms of distributional drift, where high generation stochasticity leads to a premature end-of-sequence token. Because void turns are rare successful trajectories but indicative of pathological ones, they serve powerful heuristic for identifying problematic trajectories.
This insight leads to the SimpleTIR algorithm, illustrated in Fig. The procedure is straightforwar: for each sampled trajectory, we inspect its turns. If any turn contains neither a complete code block nor final answer. removing it from the batch before the GRPO update. This single step simultaneously prevents the large gradients from lowpprobability tokens from backpropagating and corrects misaligned credit assignment by ensuring successful early turns are not penalized for a later collapse. SIMPIEETRR filtering approach is agnostic to the specific RL, algorithm used and is orthogonal other recent improvements in RL for LLM reasoning (Liu et al. 2025a Zheng et al.) 2025a Chen et al. 2025 Li
labeled void turn. We then mask the policy loss for the entire trajectory.
ct al. 2025c 2026b a).
3.4
IMPLEMENTATION DETAILS
To further enhance training stability and efficiency, we adopt several key practices. First, to avoid dut-o--distributtin special tokens when using base models, we do not use chat templates. Instead, we prepend tool outputs with a simple prefix, "Code Execution Result:", Second, to provide a shortcut for simple tasks and improve sample efficiency, 'final answer' function, allowing the model to terminate and answer within a single turn if possible. Finally, to prevent the model from hallucinating tool outputs, we strictly halt LLM generation after complete code block is produced and always append the true, external tool feedback before the next turn begins.
prepend every LLM- LLM-generated code block with
EXPERIMENTS
4.1 SETUP
Training We prepare training code with the VeRL (Sheng al. 2024 and Search-R1 (Jin et al. 2025) framework. We use Sandbox Fusion as an asynchronous code interpreter. The training datasets are Math3-5 from SimpleRL (Zng al. 2025) and Deepscaler (Luo al. 2025). SimpleTIR follows the Zero RL setting and uses the unaligned Qwen series as the base models, including Qwen2.5-7B, Qwen2.5-32B, and Qwen3-4B-BBB Base. During training, the rollout batch size is set 512, and the mini update size is set to 128. The maximum response length is initially set to 16K, with maximum of five turns of code execution. When the average response length plateaus, we increase the maximum response length to 24K and the largest number of turns to 10. Other training hyperparameters are in Appendix B.2.
Evaluation Our evaluation is conducted Math500 (Hendrycks et al. 2021), AIME24, AIME25,
AMC23, and Hmmt Feb 25, using a temperature of and reporting average @ 32 scores to reduce variance, following the standard practice in evaluating LLMs reasoning abilities (Yu al. 2025). For comparison, we consider three categories of baselines. The first is non-TIR Zero RL, where we use SimpleRL-Zoo (Zeng et al. (2025) and DAPO (Yu et al.) (2025) as representative baselines. The score of 22.1 in Fig. [ exactly obtained with DAPO on the Qwen2.5-78 model. The performance gap between these methods and SimpleTIRR highlights the advantage of incorporating TIR mathematical reasoning. The second category is TIR RL from cold-start or specialized models. ARPO (Dong et al. 2025) finetunes Effective CIR (Bai et al. 2025) apply RL to the Qwen2.5 2-nn2.55Ma 5-Mthh series. The final category is Zero RL with TIR, where, to the best of our knowledge, Zero- ZIR Mai et al., 2025) is the only method that strictly follows the Zero RL paradigm by training TIR models directly from base models. To the best of our knowledge, there is no published work that finetunes the Qwen3-4B- 4B--Base model with RL. So for SimpleTIR-4B we only prepare with the base model.
ToRL (Li et al., 2025b), VerlTool (Jiang et al. 2025), and

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison various math benchmarks. Check and marks in the "TIR" column refers whether the method involves TIR during training and evaluation. The "From" column indicates the type of base models we train the model from. We fill the scores with if they are not provided in respective reports.
Model
TIR
From
AIME24 AIME25 MATH500 Olympiad
AMC23
Hmmt25
Models based on Qwen2.5-7B
Qwen2.5-7B
X
Base
3.2
1.1
51.9
15.4
21.7
0.0
Owen2.5-7B-TIU
Base
1.7
0.6
18.0
6.2
10.8
1.9
SimpleRL-Zoo-77
X
Base
15.6
78.2
40.4
62.5
ToRL-7B
Math- Inst
40.2
27.9
82.2
49.9
75.0
Effective TIR-7B
Math
42.3
29.2
86.4
74.2
VT-7B
Math
36.7
82.8
51.6
75.0
ARPO-7B
Inst
30.0
30.0
78.8
ZeroTIR-7B
Base
39.6
25.0
80.2
22.5
SimpleTIR-7B
Base
50.5
30.9
88.4
54.8
79.1
29.7
Models based on Qwen2.5-32H
Qwen2.5-32B
X
Base
4.2
1.6
43.1
17.8
28.0
0.2
Qwen2.5-32B-TIn
Basc
7.1
5.0
37.0
16.9
20.0
5.2
DAPO
X
Basc
50.0
ZeroTIR-32 32B
/
Base
48
27
87.8
20.0
SimpleTIR-32
Base
59.9
49.2
92.9
63.7
91.6
34.6
Models based Qwen3-4B
Qwen3-4B-BB
Base
4.9
2.9
46.6
21.7
0.5
Qwen3-4B-TIR
/ Base
4.9
3.1
24.9
12.7
15.9
2.9
SimpleTIR-4B
Base
48.1
40.2
90.0
57.4
83.1
28.2
4.2 TRAINING RESULTS
As shown in Tab. I SimpleTIR achieves significant performance improvements over the base models and outperforms all Zero RL baselines, both with and without TIR. Notably, Simple'TIR also surpasses baselines that start from specialized, math-instructed models, such as ToRL and Effective TIR. This result highlights the power and potential of the Zero RL paradigm: by ensuring training stability and avoiding the premature penalization of complex multi-turn behaviors, models can develop sophisticated reasoning abilities directly from outcome-biic based rewards, without relying on uaman-annotated SFT data. Furthermore, the consistent performance gains observed across various base models. including Qwen2.5-7B, Qwen2.5-32B, and Qwen3-4B-Baste confirm the general applicability and scalability of the SimpleTTR framework.
4.3 TRAINING CURVES, METRICS, AND ABLATION STUDIES
We show the training curve of SimpleTIR from Qwen2.5-7B with 5, and 10 turns of generation in Fig. (Top). In all these settings, SimpleTIR exhibits constant and smooth increases of the average response length and performance scores. The average number of turns first arises quickly then remains constant for multi-turn SimpleTIR. We also observe that the response length and the Math500 score scales with more turns, while the AIME24 score does not benefit clearly. This indicates that different tasks require distinct reasoning patterns. Some may be solvable with few steps of reasoning, but others will take a number of external feedback before reaching the correct answer.
We also conduct ablation studies to demonstrate the effectiveness of trajectory filtering in SimpleTTR. first investigate two alternative filtering criteria: high importance ratio and low token probabilities, as specified in the first paragraph of Sec. 3.3 As shown in Fig. 5 (Bottom), these two filtering approach canmot resolve the issue of gradient explosion, exhibiting unstable curves training scores. stable curve of gradient norm, thanks to the mild token probability
SImpleTIR features distributions. This demonstrates the effectiveness of void turn filtering in stabilizing multi-tumn TIR training. We then consider ablation method where L.MM generation is terminated void turns
7

---

Published as a conference paper at ICLR 2026
Average Respormee Langth
WMM
Number or Turns
5.0
4000
Moth500 Score
2.5
0.6
0.3
AIMEZ4
2000
1.5
1000
0.2
1.0
0.0
400
800
1200
400
500
1200
400
800
1200
400
800
1200
Steps
Steps
Steps
Steps
Mox Turms11
Max Turnpe5
Mox Turns:
0.55
Min Token Token Probasiing
0.50
Grai Norm
-15
("waaeee WW
Mox Roto
-20
0.40
225
0.55
-30
0.25
80
160
240
60
160
240
0
80
160
240
0
80
160
240
Steps
Stepa
Staps
Stapa
High Filtering
Filtering
SimpleTIR
Figure 5: Top: Training curves for SimpleTIR with different maximum number of turns. SimpleTIR with maximum 10 turns resumed at 200 steps from SimpleTTR with maximum turns, SimpleTTR clearly benefits from scaling interaction turns from to 5. Bottom: Trajectory filtering with high importance ratios or low probability tokens cannot resolve the challenge of training instability, while SimpleTIR suffers less from low probability tokens and gradient explosion.
The training curves for ablation studies in the first 320 steps.
Table 2: Results ablation studies. Considering the unstable training ablated methods, we report the highest scores within 1000 gradient steps. "Naive Multi-Turn' directly applies RLVR in multi-tum TIR. "Low Prob" and "High Ratio" filtering refers to masking the policy loss tokens with lowest probabilities or highest importance ratio.
Naive Multi-Tune -urn
Low Prob Filtering
High Ratio Filtering
Stop Generation w/o Filtering
SimpleTIR-7B
AIME24
50.5
20.8
23.3
26.3
26.1 1
Math500
88.4
73.1
72.8
75.0
77.3
but resulting trajectories are not filtered when computing policy loss. According to the validation results Tab. [ this method is also inferior SimpleTIR. This can attributed to misaligned credit assignment since trajectories containing void turns can hardly obtain positive outcome. Simple'TIR handles such issue by masking the loss of whole responses containing void turns.
4.4 EMERGENCE OF DIVERSE REASONING BEHAVIORS
Thanks to the framework of Zero RL training, SimpleTIR automatically reinforces useful reasoning patterns obtained in the pretraining phase, rather than sticking to predefined patterns in the SFT dataset. In Appendix A.2 we show SimpleTIR responses with diverse multi-turn reasoning behaviors. They are mostly combinations of the three main reasoning patterns illustrated in Figure 6 namely Cross Validation, Progressive Reasoning, and Error Correction.
We also use Claude-3.7-Some generated by ReTool and CmmpleTIR322BB The responses are filtered so that they all lead to the correct final answer. Both models demonstrate strong tendency to conduct multiple rounds of cross verification. Meanwhile, SimpleTTIR CimpleTIR-322 32B exhibits more instances of progressive reasoning and error correction. This illustrates the advantage of Zero RI, preserving more diversity in reasoning patterns.
to identify and count the frequency of reasoning patterns in responses
8

---

Published as a conference paper at ICLR 2026
Pattern 1: Cross Valldattonn
Pattern 2: Progressive Reasoning
Pottern 3: Error Correction Loop
Code solve a0
Code 2 solve b0
Code holve_step10
Code
buggy func
Retry If Error
</>
</>
<>
Result 1
/p
Code
Initial Result
Result
Result
solve_step20
</>
Result 2
Val dalidated Results
Code solve step30
Final Results
<>
Final Results
Figure 6: Demonstration of three reasoning patterns observed in responses generated by SimpleTIR.
Table 3: Comparison of reasoning pattern frequencies in ReTool and SiR- responses. The summation of frequencies may exceed 100% there may be more than reasoning patterns in response.
Progressive Reasoning (%)
Cross
Error
Verification (%)
Correction (%)
ReTool
18.9
82.4
25.8
SimpleTiRR-35
46.5
86.0
38.0
5
RELATED WORK
5.1 ZERO RL FOR LLM REASONING
DeepSeek-R1 (DeepSeek-A) AI Team 2025) first demonstrated that starting from an unaligned base
model, large-scale RI. training with outcome rewards unlock emergent chain-of-thoure reasoning abilities. This paradigm was subsequently termed Zero RI. SimpleRI. (Zeng et al. 2025) provides reproducible framework for running Zero RL on various open-source base models. Open-Reasonee
Zero (Hu et al., 2025) shows that vanilla PPO with GAE (A = 1, = 1) and without KL regularization
sufficient scale up Zero RL training. DAPO (Yu al. 2025) introduces several training
techniques that make Zero RL stable and efficient, such as increasing the clipping ratio in PPO and GRPO, and filtering tasks with
5.2 RL FOR TOOL-INTEGRATED REASONING
Several recent works focus on applying RL improve the tool-use capabilities of LLMs. Search- R1 (Jin et al.) 2025) and R1 R1-Search (Song et al., 2025) focus question-answerin tasks by integrating search tools. For mathematical reasoning tasks, Python interpreters serve as useful tools for numerical calculation and enumeration. ReToool (Feng et al. 2025) employs a cold-start SFT phase prior RI, while ToRL. (Li et al. 2025b) and Effective CIR (Bai al. 2025) explore training recipes using math-specialized base models. These pipelines often rely domain data, instruction tuning, or other supervision that introduces bias and complexity. In contrast, Zero RL is more general notoriously unstable in multi-turn settings. Our work directly addresses this stability gap in Zero RL by filtering trajectories with void turns. ZeroTIR (Mai et al., (025) is similarly framed within the Zero RL setting, proposing stabilizing techniques that strictly orthogonal to approach. Furthermore, recent theoretical work (Lin & Xu 2025) explains why TIR is more effective than text-only reasoning: SimpleTIR provides strong empirical evidence supporting this claim.
5.3 STABILIZING RL TRAINING
Training instability remains significant challenge when applying RL LLMs, often manifesting as entropy collapse and gradient norm explosions. Recent investigations reveal that these sudden training collapses frequently exacerbated by fundamental training-inference mismatch (Liu

---

Published as a conference paper at ICLR 2026
et al. 2025a). To delay distributional narrowing, entropy-based methods explicitly maintain policy entropy or encourage re- generattion at pivotal tokens (Cui et al., 2025 Li et al. 2025a Liu et al.) 2025bj. Another line of work controls the importance sampling (IS) ratio reduce gradient variance and prevent brittle updates. These approaches achieve stability by reweighting or constraining likelihood ratios-suu -ucih as shifting from token-level IS sequence- Lequence-level objectives and clipping (Liu et al., [025a] [Chnn et al. 2025; Zhao et al. 2025 Zheng et al.) 2025b; Li et al. 2025c) or by pruning the extreme tail of the vocabulary (Li al. 2026b). To strictly enforce trust regions in long-
horizon tasks, sequence-level masking methods entirely exclude trajectories that violate token-level divergence thresholds (Li et al., 2025c). To further mitigate exploding gradient variance, optimal baseline theory has been extended to the token level, dynamically weighting updates inversely to their accumulated stochasticity (Li et al. 2026a). Alternatively, data and trajectory filtering schemes, such as
filter approaches, stabilize training by discarding uninformative or harmful samples (Shrivastava et al., 2025). Regarding the learning signal itself, negative-only gradient updates have been shown to improve stability and generalization without sacrificing exploration, focusing updates on low- low-probale wwwprbbbabiiiyy or high-entropy branching tokens (Zhu et al. (025). SimpleTTRR departs from the above methods by targeting root cause specific to TIR: the distribution shift induced by external tool outputs, which is compounded by multi-turn error accumulation. It also remains wholly orthogonal existing entropy regularization, IS ratio control, and regative-gradien schemes.
CONCLUSION
In this work, we introduce SimpleTIR, RL framework designed to stabilize and enhance multi-turn TIR under the Zero RL setting. By addressing the key challenge of harmful negative samples via filtering out trajectories with void turns, our method achieves stable training dynamics and improves reasoning performance across a variety of mathematical benchmarks. Beyond state-of-the-ed results starting from the Qwen2.5-7B model, Simple'TIR also encourages the emergence diverse reasoning patterns. These results highlight the potential of end-to-end multi-turn TIR RL, without relying on cold-start human data, as pathway to scalable and reliable multi-turn reasoning in future LLM agent development.
Limitations and Future Work While effective, our method has several limitations. First, we use void turns indicator of owwprobabillity tokens in multi-turn TIR. However, this indicator may not be directly applicable to tasks beyond multi-turn TIR. Second, we currently restrict the maximum number of turns to 10 for mathematical reasoning, though more interactions may be required for complex multi-turn agent tasks. Third, our training relies on highly parallel sandbox for code execution. Therefore, the development of faster and more reliable sandbox an important direction for future work. Finally, achieving fully asynchronous rollout and reward calculation remains an open challenge. These limitations raise additional concerns around rollout efficiency, memory management, and credit assignment, which we leave for future exploration.
This work is supported by Tencent and Tencent-NTU Joint Research Labora- tory (CENTURY), Nanyang Technological University, Singapore. This research is also supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier (RG18/24).
Acknowledgements
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Fei Bai, Yinggian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Towards Effective Interlnte Reasoning, arXiv preprint arXiv:2505. 24480, 2025.
Carlo Baronio, Pietro Marsella, Ben Pan, and Silas Alberti. Multi-Turn RL Training for CUDA Kernel Generation. https://cogniiion.//boog/// witbblog/cvviin//2222 2025.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang. Cheng Zhu, al. MiniMax-MI: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv preprint arXiv:2506. 13585, 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv preprint arXiv:2505.22617, 2025.
DeepSeek-AI Team. DeepSeek-RI: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501. 12948, 2025.
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization. arXiv preprint 25XX::2507.11849 2025.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. ReTool:: Reinforcement Learning for Strategic Tool Use in LLMs. arXiv preprint arXiv:2504.1 /536, 2025.
Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaclbling, Thomas L. Dean, and Craig Boutilier. Hicrarchical solution of markov decision processes using macro-actions. UAI, 1998.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring Mathematical Problem Solving With the MATH Dataset. In NeurIPS Datasets and Benchmarks, 2021.
Jingeheng Hu, Yinmin Zhang, Qi Han. Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Reen-RRasoner-Zer An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model. arXiv preprint
2025.
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du et al. VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use. arXiv preprint arXiv:2509.01055: 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv preprint Xrx3: 09516, 2025.
Qingbin Li, Rongkun Xue, Jie Wang, Ming Zhou, Zhi Li, Xiaofeng Ji, Yongqi Wang, Miao Liu, Zheming Yang, Minghui Qiu, et al. CURE: Sestical-Toke-GGuiidee Re-concatenation for Entropy- collapse Prevention. arXiv preprint arXiv:2508. 1016, 2025a.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. ToRL: Scaling Tool-Integrated RL. arXiv preprint arXiv:2503.23383, 2025b.
Yingru Li. Logit Dynamics in Softmax Policy Gradient Methods. arXiv preprint arXiv:2506.12912. 2025.
Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, and Baoxiang Wang. Trust Region Masking for Long-Horizon L.M Reinforcement Learning. arXiv preprint arXiv:2512.23075, 2025c.
Yingru Li, Jiawei Xu, Ziniu Li, Jiacai Liu, Yuxuan Tong, Wei Liu, Longtao Zheng, Zhenghai Xue, Yaxiang Zhang, Tianle Cai, Ge Zhang, Qian Liu, and Baoxiang Wang. The Optimal Token Baseline: Variance Reduction for Long-Horizon L.M-RRI. arXiv preprint arXiv:2602.00 07078, 2026a.
11

---

Published as a conference paper at ICLR 2026
Yingru Li, Jiawei Xu, Jiacai Liu, Yuxuan Tong, Ziniu Li, Tianle Cai, Ge Zhang, Qian Liu, and Baoxiang Wang. Dynamic vocabulary pruning: Stable llm-rl by taming the tail, 2026b. URL
https://arxiv httr://bb//212.23000
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi QuanQ Luo. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. In ICMI, 2024.
Heng Lin and Zhongwen Xu. arXiv:2508. 19201, 2025.
Understanding Tool-Integrated Reasoning. arXiv preprini
Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When Speed Kills Stability: Demystifying RI. Collapse from the Training-Inference Mismatch, 2025a. Notion Blog.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv preprint arXiv:2505.244: 24864, 2025b.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. DeepScaleR: Surpassing O1-Preview with a 1 1.5B Model by Scaling RL, 2025. Notion Blog.
Xinji Mai. Haotian Xu, Xing W. Weinong Wang, Yingying Zhang, and Wenqiang Zhang. Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving, arXiv preprint arXiv:2505.07771 2025.
Moonshot AI. Kimi-Researcher: End-to-End RL Training for Emerging Agentic Capabilities.
bttps:://oonssaoee githt /iio//iiii---. Reeeeeeeeeeaerreeeeeeeee June 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint 170.:v::777...330 2017.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework. arXiv preprint 2rXiv:2409.1992 2024.
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning. arXiv preprint vrxiv:2508.09726 2025.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-Searcher: Incentivizing the Search Capability LLMs via Reinforcement Learning. arXiv preprint arXiv: 05552, 2025.
Qing Wang, Yingru Li, Jiechao Xiong, and Tong Zhang. Divergence-augmennte policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. RAGEN: Understanding Self-evolution in LLM Agents via Multi-turn Reinforcement Learning, arXiv preprint arXiv:2504.20073, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv preprint arXiv:2503. 14476, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL- Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild. arXiv preprint ar 2503 18892, 2025.
Yifan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, Haojie Ding, Jiankang Chen, Fan Yang, Zhang Zhang, Tingting Gao, and Liang Wang. Re-Rewrrd: Training Multimodal Reward Model Through Stable Reinforcement Learning. arXiv preprint 2xXv:25055.02835 2025.
12

---

Published as a conference paper at ICLR 2026
Yuzhong Zhao, Yue Liu, Jumpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shao- han Huang, Lei Cui, Qixiang Ye, et al. Geometric-Mean Policy Optimization. arXiv preprint arXiv:2507.20673, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group Sequence Policy Optimization. arXiv preprint arXiv:250..8877, 2025a.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang. Yuqiong Liu, Rui Men, An Yang, et al. Group Sequence Policy Optimization. arXiv preprint wrXv:22507.1807/ 2025b.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning. arXiv preprint arX1v:2506.01347, 2025.
13

---

Published as a conference paper at ICLR 2026
A EXAMPLE RESPONSES
A.1 INCOMPLETE RESPONSE
We present representative failure cases that contain void turns, ie., turns that produce neither complete, executable code block nor boxed final answer. These examples serve diagnostic role: they illustrate how OOD tool feedback and compounding errors precipitate collapsed generations and gradient spikes during Zero RL. Tab. H shows the LLM response corresponding to the token probability statistics in Fig. Tab. shows another typical trajectory in which a void turn disrupts subsequent decoding and leads corrupted outputs, motivating our trajectory filtering rule.
A.2 RESPONSE WITH EMERGENT REASONING BEHAVIORS
We provide qualitative rollouts that demonstrate the diverse multi-turn behaviors SimpleTIR elicits without instruction-level biases. Tab. 7 illustrates progressive reasoning and code improvement. Tab. 6 demonstrates another behavior of self reflection. Taken together with the quantitative pattern analysis in the main text, these cases substantiate our claim that Zero RL with TIR encourages richer strategies than cold-start SFT.
B EXPERIMENTS
B. PROMPT FOR MULTI-TURN TIR GENERATION
We include the exact prompt template used to generate multi-turn TIR trajectories in Tab. 8b The design emphasizes: (1) selective use of Python wrapped triple backticks as complete scripts (with imports); (2) explicit printing of intermediate quantities so that execution feedback can guide later turns; and (3) a standardized answer channel (final [Einal_answer (...) or \boxed(....) 1) that cleanly terminates trajectories when a solution is reached. These choices stabilize interaction with the interpreter, reduce format variance, and make easy to detect valid tool calls versus void turns.
B.2 HYPERPARAMETERS
We show the training hyperparameters of SimpleTIR in Tab. 8a Below explain the rationale behind the hyperparameters. We cap the initial max response length at 16,384 tokens to accommodate complete code blocks and verbose execution traces without premature truncation. Initial max interaction turns = 5 bounds episode length and compute while still allowing the model to plan, execute, and verify within a single trajectory. The interaction turns scaled to 10 on training step 800. We set rollout and evaluation temperature = 1.0 to preserve diversity in candidate solutions. Each update uses a sampling batch size of 1,280 with group number 16 rollouts per prompt.
We use PPO-style clipping with ratio = 0.2 / 0.28 (low/high) to constrain policy updates. The slightly looser upper bound avoids over-penalizing udvantageous moves identified by execution feedback. PPO epochs 4 provides sufficient reuse of samples without overfitting to batch noise. The train batch size 512 balances gradient estimate quality and memory use. A small actor learning rate le-6 and gradient clipping = 1 (global-nore) -nrmm) prevent instability from long- long-context backprop and occasional high-magnitude advantages produced by sparse rewards. We treat each dialogue as an undiscounted episodic task with 7 1.0. We also set entropy coefficient = 0 and KL coefficient = 0. Exploration instead induced by temperature and multi-rollout sampling.
B.3 PROMPT FOR EXTRACTING REASONING PATTERNS
To analyze behaviors systematically, in Tab. we provide the prompt used to label reasoning behaviors in trajectories.. It first enforces minimal bookkeeping (e.g., number code blocks, whether any execution equals the final answer), then queries four patterns: decomposed sub-solves (with non-final execution), induction, self- verificaaia,, and format improves label consistency and allows fair frequency comparisons across methods.
via additional code. The constrained output
14

---

Published as a conference paper at ICLR 2026
Token Probabiiltt
Traning Score
AIMIZ4 Score
Staps
Stepa
Stepa
Staps
High Filtering
Prob Filtering
SimpleTIR
Figure 7: The full training curve ablation studies. "High Ratio Filtering" "Low Filtering" approaches are early stopped due to NaN in policy gradients.
Average AIMEZ4 Acturocy (%)
1
Void
Steps
Steps
Steps
Steps
SimpleTTu
Min Taker probabilig
2.-
AIMEZ4 Score
0.-
0.-
Steps
Steps
Stepa
Filter Suffix
SimpleTIR
Figure 8: Top: The leftmost figure shows the ratio of trajectories containing void turns in the training of SimpleTIR. The other figures show the relationship between low token probabilities and gradient norm, demonstrated by trials with distinct trends of gradient algorithm which filters only suffix process but the performance remains suboptimal.
Bottom: Comparative analysis of baseline first void turn (Filter Suffix). It exhibits relatively stable training
15

---

Published as a conference paper at ICLR 2026
B.4 ADDITIONAL RESULTS
We present more detailed experiment results here. Fig. shows the full results of ablation studies. We find that once gradient explosion maintains for a few training steps, the subsequent training process will remain suboptimal. In Fig. (Top), the leftmost figure shows the ratio of trajectories containing void turns the training of SimpleTIR. Initially the ratio is high due the unaligned base model. The number of void turns then drops continuously as training proceeds, maintaining a ratio around 15% among generated trajectories. The other figures in Fig- S (Top) demonstrate two trials with distinct trends of gradient norm. The continuous explosion of gradient norm in naive multi-turn training is related with its decreasing minimal token probabilities, accordance with the theoretical analysis in Sec. 3.2 Fig. 8 (Bottom) shows the training curve of baseline algorithm which filters only the suffix after the first void turn. exhibits a relatively stable training process but the performance remains suboptimal.
16

---

Published as a conference paper at ICLR 2026
Table 4:
The response corresponding to the token probability statistics in Fig. Responses generated in different turns separated with vertical line. Tum I and Turn 2 contain normal TIR trajectories. Turn in Response Part-Il is void where complete code blocks nor boxed response in Turn compounds to subsequent Turn 4, leading to corrupted and repetitive responses with many low-probability tokens.
generated. The OOD
Example #1 from AIME24 Dataset
Question:
Solve for the value of sin given that tan 8 = sin(a a + 3) T3 and both and & are within (0, m).
Labeled Answer:
63
65
Model Output by Qwen2.5-7B with naive multi-turn training (Part-I)
To solve the value sin at given that tan B 3 sin(a + ()) and both a and are within (0, x),
we will follow these steps:
1. Find sin and cos 3:
Since tan / 3. recognize that this corresponds to right triangle where the opposite side is 4 and the adjacent side is 3. The hypotenuse of this triangle is:
h = /2 + v16+ 25 5
Hence, sin 3 = and cos = 2. Use the angle addition formula for sine:
sin(a sin cos 3 sin
Given sin(a + () 13:
5
and substituting cos and sin / we
sin a cos a
13
Simplifying, have:
5
3sin 4cos a
13
5
Multiplying both sides by clear the denominators, we get:
25
3 Ssin a + 4cos
13
Hence,
25
3sin + a
(Equation 1)
13
Some Response Omitted.....
4. Solve the system of equations:
We have two equations:
3 sin + 4 cos 25
3 cos a sin T3
Let's solve these equations using Python.
from sуmpу import symbols, s, sq, solve, sin, cos i Define the symbols
alpha symbols a'alpha')
sin alpha, cos alpha symbols sin alpha cos )lpha')
Define the equations
eql = (3 sin_alpha + 4 cos_alpha, 25 / 13) cq2 Eq cos_alpha 4 sin_alpha, 60 / 13) Solve the system of equations
solution solve (eql, eq2), (sin_alpha) cos cos_alpha)) print (sclution)
Code execution result: cos alpha: 0.861538461538466,,65 sin alphaa -0.507692307692308
17

---

Published as a conference paper at ICLR 2026
Model Output by Qwen2.5-7B with naive multi-ture turn training (Part-II)
It appears that mistakenly received negative values for sin a. This is due to possible misinterpretation or error in handling the constraints or solving the equations. Since a is in the interval (0, s), a should be positive. Let's ensure that we resolve the equations correctly and ensure sin is non-negative. We will re-solve the system of equations for sin and cos a:
sin 4 cos 25
bi
3 cos a 4 4sin a 13
I will use Python to solve these correctly.
from sympy import symbols, Eq, solve
Define the symbols
sin_alpha, cos_alpha symbels
sin_alpha symbolsi'sin_alphe Define the equations
cos_alpha')
eql = Eq(3 sin_alpha + 4 4 cos_alpha, 25 / 13)
cq2 Eq cos_alpha 4 sin_alpha, 60 / 13)
Solve the system of equations
solution = solve (eql, eq2), (sin sin__lpha, cos cos_alpha)) print (sclution)
Code execution result: cos alpha: 0,861538461538466, sin_ssinn___aaa alpha: +0.507692307692308
appears again am getting same result, suggesting a potential error in ensuring solution within the valid range for a in (0, TT. It crucial that sin be non-negative. It might be useful to check the algebra involved, Let's verify resolving these correctly for sin a and cos ensuring sin a positive.
3 sin + 1 cos 25
a sin dd
Som. Response Omitted.....
I will once more use Python solve these equations accurately, paying close attention to ensuring sin a
It appears there might still be persistence in obtaining incorrect signs for sin a. To ensure sin a is correctly positive, let's directly solve the system equations carefully. appears previously I might not have correctly handled solving the system. The cquations to solve are:
3 sin 4 cos 25
bd 13
3 cas a 4 sin
.o.e Response Omitted.....
It now it it now it it it it, it now it it now, it it, it it now it, it, it now, it it, it, it, it it now, it, it, it it, it now, it., now, now. now, now, is now it,, it it,, it now. it now.,
now, it. it., it., now. it. it., it, it. now it... it it., it... it now., it. it.,, it. it now.. it... it now, it.,. it it.. now..... it......... it... now. it... ..... it. now.. it. it..... it.., now.... it.... now. .. it... it. now... it....... it...... it... now.... it.,.... it., now......... previous. now, it it, it... it now.... it. it,,, i........ it... it........ it. now... it... Some Response Omitted.....
it...... it.............................. it.., now.......
To clear the fractions, multiply through by:
273 56 a 340 185
sin + 340 185
329 sin 185 340
329 sin a -555
155
sin a 329
It appears have maintained continuing error. It now it it, it now, it, it it It, it now it previous it it it it, it now it, it, it it. It it, it now, it, it., it it. it now, it it, it, it now. It., it, it, it. E now., it it,., it, it, now., it, it., it.... it., it.,, it it it it., now..... it it now., it, now.... now... im it... now.... it. Before,,, it it. It, it now., it it. It it. it now.
" It,, it.,
18

---

Published as a conference paper at ICLR 2026
Table 5: An example from Qwen2.5-7B on AIME24 dataset. Responses generated in different turns are separated with vertical line. Response Part-ll contains void turn, where no complete code blocks nor boxed answers are generated. Both in and after the void turn, the model generates corrupted responses.
Example from AIME24 Dataset
Question:
Eight circles of radius 34 are sequentially tangent, and two of the circles tangent to AB and BC of triangle ABC, respectively. 2024 circles of radius 1 can be arranged in the same manner. The inradius of triangle ABC can be expressed m, where m and relatively prime positive integers. Find n.
A
Labeled Answer:
197
Model Output by SimpleTIR-7B (Part-I)
To solve this problem, we need to determine the inradius of triangle ABC' given the arrangement of circles. Here is the step-hy-step reasoning:
Understand the Geometry:**
have eight circles of radius 34 that are tangent each other sequentially, with two of them being tangent to the sides AB and BC of triangle ABC. - Additionally, 2024 circles of radius 1 can fit in the same manner. Determine the Distance Between Centers the Larger Circles:
The centers of the eight langent circles of radius 34 will form chain where each circle is tangent to the next, meaning the distance between centers two consecutive circles is 34 34 68. -iice there are 8 circles, the total distance from the center of the first circle the center of the last circle would be 68 x (8 1) = x = 476. However, we need to consider that first circle tangent to AB and the last circle is tangent to BC. This distance of 476 actually represents the length of the tangent chain along the line connecting the points of tangency on AB and BC.
** Relaate Triangle ABC
If we look at the centers of these circles, they form line parallel to BC and AB, and the distance between this line and either AB BC equal the radius of the circles (34 for the larger circles, for the smaller circles).
* "Caleulate the Lengths:**
For the larger circles (radius 34):
The total distance connecting the points tangency on AB and BC is effectively the distance between the parallel line formed by the centers of these circles and the line BC or AB, which is 8 34 272 (since there are 8 circles, and each has a radius of 34, the total vertical distance covered would be the diameter 7 circles if we think about in terms of centers plus radius). However, since they tangent each other sequentially, the distance between the lines AB and BC (parallel to the line connecting the centers) should be 34 + 34 x (8 1) 34 + 34 x 34 x 272.
For the smaller circles (radius 1):
Similarly, if can fit 2024 circles of radius 1, the total vertical distance from the line of tangency on AB the line of tangency on BC would he 2024 x 2024.
* "Inaadius Relationship:**
The inradius of triangle ABC is the radius of the inscribed circle. To find this, we need to understand how the heights and lengths of the triangle are affected by these tangent circles.
Let's denote the length of AB BC = and CAA = b. The height from A to BC' can be considered by looking at how the circles stack up vertically.
For the larger circles:
two cireles are tangent to AB and BC, the length along AB and BC would create a configuration where if you think about the points of tangency, the distance covered by the centers vertically for 8 circles should match up with the triangle height minus twice the radius (since the circles are also on the boundary of the triangle).
19

---

Published as a conference paper at ICLR 2026
Model Output by SimpleTIR-7B (Part-II)
So.. Response Omitted..... Given this, distance between sides should follow proportionally: Distance apart: smaller= 2024x1 Total height of the triangle should include this detailed intermediary:
6x31
Let's re-evaluate and verify it step-by-ster by- via coding:
import sympy as sp
# Define the radius of larger and smaller circles R_large = 34
R_small
num cam__arre__irreee circlesrreeeeeee
num (a___mmal__irree
2024
# Total vertical distance covered by the larger circles' centers from base to the point of tangency
di stance_large_eent How many points can fit in this same height but smaller the vertical 1ine can stack:
num_largeccircles
R_large)
distance_snme ssssssmalll___ saall_centers nun nun_stall circles (2 R small)
Height from tangency the actual top parallel line keeping them
tangent:
height_large = distance_large_coneee
actually includes the tangency
start:
height_smalll = distance_sme sase__mmmlll_eeerrs__
Deriving the total height by proportionality:
R_ratic R_small R_large
num ramir num mim__maai_ccrcles num large circies
large_circies
HR Lenghh = ham__larre_irrreeee
R_large) # covering space.
efffective HR Lengh
This total covers height only parallelly: from base and top distance:
h_total HR_length
inradius = height_large / (hur_ratio 1)
inradius = sp. Rationainaa (h_total (num_ratee + 1))
print (f"calculated inradius: (inradius)")
Code execution result: calculated inradius: 48227322666011122551999933336663
This calculation seems overly complex; let's simplify using simpler, geometric visualization and relationships:
Simplified Geometric Relationship:
Large Circle Tangent Arrangement:
We have circles of radius 34, and these cireles are tangent to and BC. The effective vertical distance taken by these circles, considering their arrangement repeating pattern of centers, is given by:
8 (2-2- 12-2344 34) = 6-68
Small Circle Arrangement:***
For smaller circles of radius such that 2024 circles fit similarly: The vertical distance arrangement for 2024 circles would be:
2024 2 2021 40488
So.....omm Response Omnitee.....
We repeat arranging parallel: Consider:
each pair arrangement, verifying circles top-height Even scan: each arrangement euclidean height method involving polygons: solved for ratio translating through unique noting: Combining parameters: This par: valid:
V(rrac m)] [n]] sum ratio sorts in idlegate ration 1). However, i' sequential seek refining this calculation more consistently: iterating setup considered overall expression precise: Now solve ensuring configurations understandable suggest circle distributions proportionally: If):
M( ( each step repeated configurations: compute Vlargest Thus: We need to find side cognize area further on:
triangle holds hold altitude relation contractive step:
20

---

Published as a conference paper at ICLR 2026
Table 6: An example from SimpleTIR-7B AIME24 dataset. It demonstrates the behavior of cross validation successfully applies TIR correct mistake in text-only reasoning. Responses generated different turns are separated with vertical line.
Example #3 from the DeepScaleR Dataset
Question:
The function f(x) satisfies the equation f(x) = f(x 1) + f(x + 1) for all values of x. If f(1) = 1 and
f(2) 3, what is the value of f20088)?
Labeled Answer:
Model Output by Simple (impleeTRR-BB (Part-l)
To solve this problem, we need to determine the value of the function f(x) at x: 2008 given the recursive relationship:
f()) - f(e 1) + f(x +
We also have the initial conditions:
f(1)
f(2) =
First, let's try to find few more values of the function identify pattern: 1, For 1:
f(1) f(0) + (2)
f(0) +3
f(0) -2
2. For 2:
f(2) f()) 3 1- -
S(3) 1=
3. For 3:
f(8) f(4)
2 3- f(4) f(4)
4. For 4:
f(4) f
-1 2 f(5)
f(5) -
For.r 5:
f(5) (4) +
-3 - 1
f(6) 13
For 6:
f(6) f(5) + 1(7)
-2 = - 3 3(7(7))
f(7) = -2 -2+3=1
For 7:
S(7) [(6) + 188)
1= -=-2+(08)
f(8) - 1
21

---

Published as a conference paper at ICLR 2026
Model Output by SimpleTIR-78 (Part-ll)
We can observe the values:
f(0) - 2, 2, f(1) 1, = 3, - 2, = -1,
= -3, ((6) [()
-2, 1, - 3....
appears that the function values repeat every terms. Let's verify this by calculating few more terms if necessary. If the function is periodic with period of then:
f(z) f(x + 6)
To find f(208)), we can use the periodicity:
2008 mod 6 =
So, f(2008) f2). Since f(2) 3, have:
(2008) 3
Let's write Python code to confirm this pattern and calculate f(2008) accurately.
def find_f (n):
A Let's 8 initialize the first few values based on given condi pondition tions f = [1: 1, 2: 3)
# Calculate values up to (n)
for i in range (3, n + 1):
f[i] = fli 11 - f(i 2]
return [nn
Now we find (2008)
f_20 find f (2008)
print (#"E is: (f "0081")
Code Execution Result: f(2008) is:
It appears there was an error in the initial manual calculation or understanding of the sequence. Let's correct re-implement code step by step more carefully:
Instead of calculating manually for a long time, let's use loop to generate the sequence and check for the periodicity accurately using a Python program to ensure we f(2008) correctly.
We'll start by calculating the first few terms again using loop and then look for any pattern. We'll fill out the values for (xx) for x from to some higher number to see if we identify the correct cycle and compute f(2008). Let's write the code again to find the accurate values and scarch for patterns more precisely.
Initialize the first two values f [1: 1, 2: 31
Let''s calculate the values up to (2008)
for x in range (3, 2009):
[[x] = f(x 1] f[x 2]
Now let's find (2008)
result f12008]
print (result)
Code Execution Result: f(2008) is: -1
So... Response Omitted.....
The calculation clearly and consistently shows that f(2008) = -1. Therefore, the accurate result of f(2008)
1 This indicates the correct solution is indeed -1 The final answer is:
22

---

Published as a conference paper at ICLR 2026
Table 7: Another example from SimpleTIR-7B. It demonstrates the behavior of progressive reasoning and code improvement. Responses generated different turns are separated with a vertical line.
Example #4 from AIME24 Dataset
Question:
Let ABCD be tetrahedron such that AB CD = v41, AC BD = /8, and BC AD = V 89. There exists a point inside the tetrahedron such that the distances from to each of the faces of the tetrahedron are all equal. This distance can be written in the form myf where and positive integers, and p are relatively prime, and n is not divisible by the square of any prime. Find m + n + p. Labeled Answer:
104
Model Output by SimpleTTR TIR-7B (Part-I)
To solve the given problem, need to find the distance from point / (which is the incenter of the tetrahedron ABCD) to each of the faces of the tetrahedron, given that the distances are all equal. This distance is the radius of the inscribed sphere of the tetrahedron.
First, we need to determine the volume V of the tetrahedron and the surface area S, The formula for the radius of the inscribed sphere given by:
3V S
### Step Calculate the Volume the Tetrahedron the Cayley-Menger determinant find the volume of the tetrahedron. The Cayley-Menger determinant for tetrahedron with edge lengths a, b, c, d, e, (where a AB, = AC, c = AD,d D BC, BD, f CD) is given by:
We
1
1
0
à
b²
288V
a
0
dP
e²
6
d3
pa
c²
6²
f²
0
Substituting b v80,
V89, d
89,
80,
f
41:
1
0
11
80
89
288V2
41
80
1
80
89
41
89
80
41
We calculate the determinant of the matrix:
import вуmpу as sp
Define the matrix
matrix sp. M [0, 1, 1, 1, 11, [1, 0, 41, 80, 89], [1, 42, 0, 89.
80], [1, 80, 89, 0, 411, [1, 89, 80, 41, 0]1)
Calculate the determinant det matrix.det
Volume squared
v V_squared det / 288
Volume
V = sp.sqrt (V (_quuured)
print (f"Volume V: (V)")
Code Exccution Result: Volume 160/3
23

---

Published as a conference paper at ICLR 2026
Model Output by SimpleTIR-7B (Part-Il)
### Step Calculate the Surface Area S of the Tetrahedron The surface area is the sum of the areas of the four triangular faces. We can find the area of each triangle using Heron's 's formula.
For triangle A ABC:
-AB VTT AC - V89
The semi perimetrr is:
VII V80
V
Using Heron's formula:
Area V - AB) AC) BC) We need to find the of triangles ACD, and BCD similarly. Let's calculate these step by step using Python.
import sympy as sp
Define the edge lengths
AB sp.sqrt (41)
sp.sqrt (80)
AD sp. sp.sqrt (89)
BC sp. sq.strt (89)
BD sqrt (80)
CD sp.sqrt (41)
Function to compute the area of a triangle using Beron's s formula def triangle_aaee _areeerreaaeee (a, b, c): (a b / sp. sp.sqrr( return area
bj (s c))
Calculate the areas of the triangular faces area ABC triangle_ared (AB, AC, BC)
атеа_Aни triangle_area (AB, AD, HD)
area_ ACD triangle_area (AC, AD, CD)
area_8CD - triangle_area (8C, BD, CD)
Total surface
area_ + ABD + area ACD + area_ BCD
print (f"Area of triangle ABC: (arca_ABC)")
print (f"Area of triangle ABD: (area_ABD}**
print (f"Area of triangle ACD: (area_ACD]")
print (f"Area of triangle BCD: (area_RCD)
print (F"Total surface area 8: (8)")
Code Execution Result: Area of triangle ABC: sqrt(ssqrt(411/22 2 +
+ 2sqrt(5) + sg)r((8))2)))rr(222sr(()))) + sqrt(41)/2 sgrt(88)))sqrr((<qr((883 sqrt(41)/2 2sqr((5))
Area of triangle ABD: sqrt(-sgrt(41)22 2sqrt(5) sqrt(89)/2
...This content has been truncated to stay below 512 characters..
V2 + (0++sprr(0000/)2gr(tttsrr(111111 sqrt(41)/2 2sqr((5)) Total surface area (ssgrr((ssrr((4112++2sssst
sgrt(899/2sgrrt(srr((()))2 2sqrt(5) + sgrr(89))))sppr(22spttt sqrt(41)/2 +grr(89)/2srrrt(ssrr8888 sqrt(41)/2 + 2sqrt(5))
appears the individual areas calculated by Heron's formula are quite complicated and expressed in a form needing simplification find the total surface area S. Let's simplify the expressions to make the calculations clearer and ensure we obtain numerical value for the surface area S. We can use simplify to get more manageable expressions and then compute the numerical value. Let's also calculate 3v step by step
simplify finding the equal distances. We know the volume V = 160
24

---

Published as a conference paper at ICLR 2026
Model Output by SimpleTIR-77 (Part-llI)
We will simplify approach and focus calculating these areas numerically to find S. Afterward, we will compute and ensure the result is in the required form myn Let's simplify our strategy and calculate this accurately in Python.
import sympy sp
Define symbolic variables sqrt_41 sp. sp. (41)
sqrt_80 sp. (80)
sqrt_89 sp.sqrt (89)
FFucttoon to compute area using Beron's s formula def area heron (a, P, c):
s = (a b c) 2
area - sq.t return sp. sp.simplify(aa (area)
(s - a) (s - b) (s c))
Calculate areas
triangles
area area_ABC area heran (sqrt_41, sqrt_80, sqrt_ 89)
area_ABl ABD = area_herow (sqrt_41, sqrt_89, sqrt_80)
area ACD = area area_heron (sqrt_80, sqrt_89, 41rt_41)
arCa_BCD area_heron (Bg_ttt (sgrr__9, sqrt_80, sqrt_41)
Total surface area
S = area_ABC + ABD + area ACD + area BCD
$ Volume
V sp. ap..aatioall (160, 3)
Radius of the inscribed sphere
r
Simplify r
simpliffeed = sp. somplifyy (r)
print ("Radius r:", r_simplified)
Code Execution Result: Radius r: 20'sgrt(21)63
The radius of the inscribed sphere 20v2I
63
This is in the form my/a where m 20, n 21, and p = 63. Here, m and p are relatively prime, n
not divisible by the square of any prime.
### Step Finding m + n + Now, need calculate + pt
+ 20
63
Let's calculate this:
import sympy as sp
Values
20
n 21
63
Sum result m + P
print (result)
Code Execution Result: 104
The value of m + + is 104
This completes the solution. The distance from point / each of the faces of the tetrahedron is 20 and
the final answer 104
25

---

Published as a conference paper at ICLR 2026
Table 8: Key hyperparameters (left) and the prompt for generating multi-turn TIR responses (right).
(a) Key Hyperparameters
(b) The Prompt for Generating Multi-turn TIR Responses
Hyperparameter
Value
Solve the following problem step by step. You have the ability to selectively write executable Python code to enhance your reasoning process. The Python code will be executed by external sandbox, and the output (after "Code execution result: ") returned aid your reasoning and help you arrive at the final answer. The Python code should complete scripts, including necessary imports.
Initial max response length
16384
Rollout Temperature
Rollout Top-p
Evaluation Temperature
Evaluation Top-p
0.95
Initial max interaction turns
5
Train batch size
512
Sampling batch size
1280
Code Format:
Rollouts per prompt (x)
16
Each code snippet is wrapped between "*, You need to use print to output intermediate results.
PPO clip ratio (low high)
0.2
0
0.28
Entropy coefficient
Discount factor y
1.0
Answer Format:
GAE À
1.0
You can use the Final_answes () function in the code to return your final answer. For example, to answer the User Question: What is the result of the 5 3 1294.6787, you can write:
KL coefficient (9)
PPO epochs
Actor learning
1e-6
Gradient Clipping
answer
1294. 678
Final_answer (answer)
You can also use \boxed to return answer. The last part your response should be: \boxed final answer goes here.")
User Question:
Table 9: The prompt that instructs
extract reasoning patterns from the TIR trajectories.
T have reasoning process of L.M.M The L.MM write code and get code execution result. According the following reasoning process, please first answer the following questions:
the code execution result or interpreter output equal to the final answer?
2. How many code blocks are there in the reasoning process? there are several code blocks, are the code execution results all the same?
Format:
1. xxx
xxx
3.
Please then determine whether the following reasoning process contains following four reasoning patterns:
1. Include at least two code blocks, each solving unique sub-questions. **Important: in such case, the code execution result interpreter output should not be equal the final answer**
2. Use induction, from special case to general conclusions
3. Use code or do self-verification
4. Write another code block when the previous code has some bugs
Format:
Reasoning Pattern Yes/No
Reasoning Patter 2: Yes/No Reasoning Pattern Yes/No
Reasoning Pattern 4: Yes/No
Please do not output any other words.
Reasoning process:
26


Reviews:
[{"review_id": "jIKczrbOqK", "reviewer": "Reviewer_Pu3v", "summary": "This paper introduces SimpleTIR, a plug-and-play method to stabilize multi-turn tool integration reinforcement learning. The authors note that tool feedback can induce distribution shifts and lead to low-probability tokens and gradient spikes during multi-turn iterations. They analyse the root cause of this phenomenon and address it by filtering complete trajectories containing void turns, i.e., turns that do not contain a code block or yield an answer. On maths benchmarks, SimpleTIR significantly enhances the performance of Qwen2.5-7B and larger models, yielding smoother training curves compared to standard multi-turn RL. The study also observes emerging patterns such as cross-validation and error-correction mechanisms.", "questions": "What fraction of trajectories are filtered in each step? Did you count the false-positives, e.g. a useful reasoning turn get discarded, because no explicitly final answer was provided.\n\nCan you test your method’s generality on different tasks, for example, QA, general knowledge, and use different tools, e.g. web search)?\nYour ablations focus only on early steps (first 320 steps) to show instability of trajectory filtering with high importance ratios or low probability tokens. Can you report full training curves to confirm that they do not recover in the long run?\n\nCan you give a correlation between low-probability token segments and gradient spikes over many runs and report statistics instead of only a single case?\n\nDid you try to decrease the weight of late bad turns instead of dropping the whole trajectory? Did you try to filter only the suffix after the first void turn?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper links multi-turn instability to extremely low-probability tokens emerging after model processing tools, demonstrating that these tokens trigger gradient peaks. Proposition 1 and the case study elucidate this mechanism.\n\nThis paper proposes a straightforward solution: directly filtering trajectories containing invalid rounds, thereby addressing unstable updates. The algorithm description and mask selection are clear and unambiguous.\n\nTable 1 shows consistent improvements across AIME24, AIME25, Math500, AMC23 and more, for 7B, 32B and 4B bases, compared to non-TIR RL baselines and prior TIR systems.\n\nSimpleTIR exhibits smooth training curves. Ablation experiments reveal that filtering mechanisms based on low token probability or high ratio demonstrate poor stability. Similarly, filtering only “stop generation instructions” without loss masking yields suboptimal results.", "weaknesses": "Void turns are defined as those where the model fails to generate a complete code block or final answer. The training pipeline detects such instances by enforcing a final answer function and fixed tool output prefixes. The authors also note that this metric may not extend beyond the current multi-turn TIR environment.\n\nAll tasks were constrained on math problem-solving using Python tools. The paper itself acknowledges that this metric may only be applicable to multi-turn TIR scenarios. More complex agency tasks potentially require additional turns.\n\nAblation analysis focuses solely on early training (320 steps) and gives the conclusion that SimpleTIR suffers less from low probability tokens and gradient explosion. It does not mention whether the trajectory filtering with high importance ratios or low probability tokens can resolve the challenge of training instability in the long run.\n\nFiltering the whole trajectory when any void turn appears can reject hard trials and change the reward distribution. There is no quantitative report of filter rate or false positives. It may lean towards keeping easy success trajectories and discard hard trajectories.\n\nThe gradient-norm proposition is insightful, but there is no broad quantitative study that correlates measured low-probability token rates with gradient spikes across many runs and seeds. Evidence is mainly a case study and a few curves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nYhruKnOyE", "reviewer": "Reviewer_JcTa", "summary": "This paper introduces SimpleTIR, a simple yet effective reinforcement learning (RL) algorithm designed to stabilize multi-turn tool-integrated reasoning (TIR) under the Zero RL setting. The authors identify that training instability in multi-turn TIR arises from “void turns, i.e., turns that produce neither a code block nor a final answer, which cause gradient explosions through compounding low-probability tokens. SimpleTIR addresses this by filtering out entire trajectories containing such void turns during policy updates. Despite its simplicity, this plug-and-play strategy substantially improves training stability and performance across math reasoning benchmarks, achieving promising results starting from base Qwen models and enabling the emergence of diverse multi-turn reasoning behaviors.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is simple and surprisingly effective.\nThe idea of filtering out trajectories with “void turns” is conceptually straightforward yet powerful. It mitigates a key failure mode in multi-turn RL training without introducing architectural or algorithmic complexity, making it easily adoptable in other setups. These simple but effective tricks will be very elegant in practical use.\n\n- The work tackles an increasingly important and difficult problem.\nMulti-turn RL for reasoning and tool use is becoming a central direction in scaling LLM capabilities. Addressing training instability in this regime is both timely and practically impactful.\n\n- Empirical results are strong and convincing.\nSimpleTIR not only stabilizes training but also significantly improves final accuracy across multiple benchmarks. \n\n- The analysis and visualization are thorough and insightful.\nThe authors provide solid empirical and theoretical analysis to identify the cause of instability, supported by clear visualizations of gradient dynamics and token probability shifts. These figures convincingly demonstrate how SimpleTIR mitigates the problem.", "weaknesses": "- First of all: The figure captions, which are all placed above the figures, violate the conference formatting requirements.\n\n- The paper heavily focuses on “void turns” but fails to illustrate them clearly in the main text.\nWhile the appendix includes qualitative cases, the main body lacks a concrete or conceptual example of what a void turn actually looks like. Without this, readers cannot easily form an intuitive understanding of the concept or of why such turns “make no progress” in reasoning. Explicit examples or snippets in Section 3.3 would make the paper much clearer.\n\n- Discarding trajectories may lead to data inefficiency in sample-scarce settings.\nWhile filtering out void-turn trajectories works well for large-scale math reasoning tasks, it could be wasteful in domains where data collection is expensive. Discussing potential mitigations, such as partial reuse or weighting instead of full removal, would strengthen the paper’s generality. I'm curious if the authors tried other approaches that can preserve the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2JyAOA1I5", "reviewer": "Reviewer_7iMa", "summary": "This paper identifies and mitigates a key failure mode in multi-turn tool-integrated reinforcement learning under the Zero-RL paradigm. \nThis paper proposes SimpleTIR, a lightweight, plug‑and‑play trajectory filter that drops any trajectory containing a turn that produces neither a complete code block nor a final answer. \nComprehensive experiments on math-reasoning benchmarks (AIME24/25, MATH500, AMC23, HMMT) show that this paper substantially improves both stability and performance.", "questions": "1. Notation in sec 2.1 could be clearer (define H, q, K, L explicitly when introducing the hierarchical MDP). \n2. The abstract claims “AIME24 from 22.1 to 50.5 (7B)”, but 22.1 does not appear in Table 1; please reconcile and cite the exact baseline used.  \n3. Figure caption placement/style is inconsistent (some above, some below).  \n4. What is the wall‑clock overhead introduced by filtering (including sandbox latency)? Any guidance on batch sizes/turn caps (Figure 5 top) under a fixed compute budget?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The failure‑mode analysis for multi‑turn TIR under Zero RL is clear.  \n2. The “void‑turn” trajectory filter is easy to implement, orthogonal to PPO/GRPO.  \n3. Hyperparameters, prompt formats, and pattern‑labeling prompts are clearly specified.  \n4. Originality is good. Even trajectory/data filtering is not new in RL/RLHF, but the turn‑level “void‑turn” filter tailored to multi‑turn TIR is a useful.", "weaknesses": "1. All experiments use a Python interpreter for math. It remains unclear how the void‑turn heuristic transfers to other tool types (search, multi‑API plans), modalities, or noisier settings. Section 6 acknowledges this, but the paper would benefit from even a small study on search‑augmented QA (e.g., a Search‑R1 setup). \n2. The definition & detection seem heuristic. “Void turn = neither a complete code block nor a final answer.” In practice, “complete” relies on formatting and the prescribed prompt conventions. The method’s robustness to format drift (e.g., unboxed answers, partial but executable code, multi‑tool calls) is not evaluated. Clarify detection rules and report false‑positive/false‑negative rates.  \n3. Dropping entire trajectories wastes rollout budget; the paper does not quantify the filter rate over training or its impact on sample efficiency. Reporting the percentage of filtered trajectories vs. steps and wall‑clock/throughput would strengthen the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WSWfLMrOTE", "reviewer": "Reviewer_eE8Z", "summary": "SimpleTIR tackles the instability of multi-turn tool-integrated reasoning under Zero-RL by pinpointing a root cause: the distributional drift from external tool feedback that increases low-probability token sampling and triggers gradient explosions. The paper then stabilizes training with a plug-and-play trajectory filter that drops “void turns”. The paper formalizes the issue with a hierarchical MDP view and a logits-space gradient analysis, then shows strong empirical gains.", "questions": "Stronger RL training algorithms for tool using like ToolRL for stronger baseline ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper did sharp diagnosis of a concrete failure mechanismgrounded in theory, not just anecdote; \n(2) Proposes a minimal fix that is orthogonal to RL algorithm choice and easy to adopt; \n(3) Present consistent empirical wins and smooth, stable training across turn budgets\n(4) Present evidence of emergent, diverse reasoning patterns preserved by Zero-RL rather than constrained by SFT “cold starts.”", "weaknesses": "(1) The “void-turn” heuristic may miss other harmful trajectories and could be task-specific beyond code-based TIR; \n\n(2) Experimental design requires max turns are capped (≤10) and can be further extended", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)