Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning
Authors: Yulei Qin
Xiaoyu Tan
Zhengbao He
Gang Li
Haojia Lin
Zongyi Li
Zihan Xu
Yuchen Shi
Siqi Cai
Renting Rui
Shaofei Cai
Yuzheng Cai
Xuan Zhang
Sheng Ye
Ke Li
Xing Sun
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
self-imitation learning
large language model
agentic learning
llm agents
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning (RL) is the dominant paradigm for sharpening strategic tool use capabilities of LLMs on long-horizon, sparsely-rewarded agent tasks, yet it faces a fundamental challenge of exploration-exploitation trade-off. Existing studies stimulate exploration through the lens of policy entropy, but such mechanical entropy maximization is prone to RL instability due to the multi-turn distribution shifting. In this paper, we target the progressive exploration-exploitation balance under the guidance of the agent's own experiences without succumbing to either entropy collapsing or runaway divergence. We propose SPEAR, a self-imitation learning (SIL) recipe for training agentic LLMs. It extends the vanilla SIL, where a replay buffer stores good experience for off-policy update, by gradually steering the policy entropy across stages. Specifically, the proposed curriculum scheduling harmonizes intrinsic reward shaping and self-imitation to 1) expedite exploration via frequent tool interactions at the beginning, and 2) strengthen exploitation of successful tactics upon convergence towards familiarity with the environment. We also combine bag-of-tricks of industrial RL optimizations
for a strong baseline Dr.BoT to demonstrate our effectiveness. In ALFWorld and WebShop, SPEAR increases the success rates of GRPO/GiGPO/Dr.BoT by up to 16.1\%/5.1\%/8.6\% and 20.7\%/11.8\%/13.9\%, respectively. In AIME24 and AIME25, SPEAR boosts Dr.BoT by up to 3.8\% and 6.1\%, respectively. Such gains incur only 10\%–25\% extra theoretical complexity and negligible runtime overhead in practice, demonstrating the plug-and-play scalability of SPEAR.

Published as a conference paper at ICLR 2026
LEARN THE ROPES, THEN TRUST THE WINS: SELF- IMITATION WITH PROGRESSIVE EXPLORATION FOR AGENTIC REINFORCEMENT LEARNING
Yulei Qin !E
Xiaoyu Tan 'E
Zhengbao He
Gang Li
Haojia Lin'
Zongyi Li!
Zihan Xu
Yuchen Shi' Siqi Cai'
Renting Rui! Shaofei Cai Xing Sun
Yuzheng Cail4
Xuan Zhang!
Sheng Ye
Ke Li'
Tencent Youtu Lab
Shanghai Jiao Tong University
PPeking University
*Fudan University
'Xiamen University 'Equal Contribution
fyuleigin,
arthurtan
com
O Codes
Data & Checkpoints
ABSTRACT
Reinforcement learning (RL) is the dominant paradigm for sharpening strategic tool use capabilities of LLMs on long-horizon, sparsely-rewarded agent tasks, yet it faces fundamental challenge of intoraatio-exploitaatinn trade-off. Existing studies stimulate exploration through the lens of policy entropy, but such mechan- ical entropy maximization is prone to RL instability due to the multi-turn distri- bution shifting. In this paper, we target the progressive balance under the guidance of the agent's 's own experiences without succumbing to either entropy collapsing or runaway divergence. We propose SPEAR X, a self-imitation learning (SIL) recipe for training agentic LLMs. It extends the vanilla SIL, where a replay buffer stores good experience for off-policy update, by gradually steering the policy entropy across stages. Specifically, the proposed curriculum scheduling harmonizes intrinsic reward shaping and self-imitation to expedite exploration via frequent tool interactions the beginning, and strengthen exploitation of successful tactics upon convergence towards familiarity with the environment. We also combine bag-of-triee of industrial RL optimiza- tions for a strong baseline Br.BoT to demonstrate our effectiveness. In ALFWorld and WehShop, SPEAR increases the success rates of GRPO/GiGPO/Dr.Bo by up to 16.1 % %%%..6 and 20.7 7/11 1t//1..%//111 8%/13.9% respectively. In AIME24 and AIME25, SPEAR boosts Bo.BoT by up to 3.8% and 6. %, respectively. Such gains incur only 10%-25% extra theoretical complexity and negligible runtime overhead in practice, demonstrating the plu--and-play scalability of SPEAR.
INTRODUCTION
2B: KT. WEAA
BNG
Intrinsic Reward Computation
On-policy
Self- Imitation Replay
T2
&
Group-bused Advantage Estimation
Buffer
Update
Off-policy
TG
Rollont
Composition
1.k
23028
S-OM
5-205
Figure 1: Our SPEAR harmonizes the curiculum-scheduee self-imitation learning with intrinsic reward shaping for progressive exploration, improving policy performance across agentic tasks.

---

Published as a conference paper at ICLR 2026
Reinforcement Learning (RL) (Lambert et al.
2024
Guo al.
2025 Qin al.
2025b has driven
the development of reasoning capabilities of Large
Language Models (LLMs). Built upon the
reason-and-act (ReAct) paradigm (ao et al.) 2023). LLMs have powered various agentic appli-
cations such as simulated robot navigation (Shridhar et al. 2020: Li et al.
2024), mobile assis-
tant (Wang et al. 2024: Li et al. 2025a), web navigator (Furuta al. 2023 He et al.
2024). deep
searcher (Jin et al., 2025b]
al., 2022cc Tao et al. 2025), ) and GUI master (Qin et al. 2025a Hong
al. 2024). A fundamental challenge in applying RL to LLM agents is to manage the balance be- tween exploration and exploitation. The LLM agent needs to exploit both pretrained knowledge and past interactions to formalize experience that maximize rewards. At the same time, must ex- plore novel behaviors through tool-integrated reasoning and reflection. The interweaving between exploration and exploitation determines the emerging agent's competence upon convergence.
Existing studies often quantify the exploration potential through entropy (Sutton 1988 Williams &
Peng 1991 Cui et al. 2025b] Xue et al.) 2025), where the decline of policy entropy indicates over- confidence with insufficient exploration. In this case, series of regularization techniques (Ziebarl
ct al. 2008 Schulman et al. 2017b] Haarnoja et al. 2018) have been proposed to maximize entropy (Haarnoja al.; 2017 Zhao et al. [2199] Xin et al. 2020 Zhang et al. 2021 Seo al. 2021 Mehr
iet [dl])]20)) Kim & Sung [2033] Hao et al. (023). However, when comes to LLM -rivenn agents, entropy-based control is fragile: the accumulation of low-probability tokens from the environment feedback induces severe distribution shifting, often leading to mode collapse (Xue et al.] 2025)
Dong et al., (0255). Agent models may experience sustained entropy growth due uncertainty about multi-turn interactions and training instability becomes frequent (Mai et al. 20255 Yao et al. 2025
Wang et al. 20256). Recent approaches attempt to mitigate this issue by cold-start supervised
fine-tuning (SFT) (Tao et al., 2025 Qin et al. 2025al Feng et al.) [025a Qin et al.
2025c) or
hybrid schemes that combine RL with SFT (Zhang et al. 2025a). Although these methods improve stability, they compromise policy's ss discovery of strategies beyond those present in the SFT corpus. This limitation highlights the need for adaptive training frameworks that can dynamically schedule LLM-driven agents to decide when explore and when to exploit.
In this paper, we are trying to answer the following core research question: Can we schedule a smooth transition between exploration and exploitation guided by the policy's i own experience without going to extremes of either entropy collapsing or runaway divergence? We hypoth- esize that the agent should maintain its policy entropy within dynamic but controlled range that evolves over time: 1) At the early stages, increasing entropy is beneficial for broad skill-level exploration. The agent is expected rapidly develop tool-use capabilities, encounter unfamiliar observations, and engage in trial-and-errors. 2) As training advances, however, a shift toward converging entropy is required. This enables the agent to consolidate mooblem-solving heuristics and emphasize action-level exploration. The agent exploits reward signals to choose comparatively more effective actions and adapts to changing distributions for stabilizing its evolutionary path.
Emma
Outnenz Rowaed 2: M/s suasseck)
(n. Ag
Behaive Advestgg
Samplng x~
Shoest cespaseed
c/snanno 4 Flantions
Tiekaall Roward mist [L] Haskeall
Eb
g-p
f
</ooot_ puapersss
TC
rmpped 15, K.
Agenti LLMS
Intriusic Roward
Rt
Laseli nascinee Dr
Salf Imirrtee Regaletaation (Rtsslli))
[n Aa) Ag.
Fil Good Cadditans
Aduuntag: Off Peilyy Tetrutvo
Reay
relicy Upcane Foo
R.
2f Plal
for Ro,
Gool Eaperenns Canaiiaaes 1t., >s m
(1, Rt..l
Vasila fruol
Oh Oarpokay
Fop Row
Sclfllsinniin Lounnig
Figure 2: Overview of SPEAR. First, the agent interacts with the environment for set of trajecto- ries, which flow through intrinsic reward shaping and advantage estimation with on-policy updates. Second, they selected and stored in replay buffer, enabling off off-policy updates via the proposed self-imitation scheme. This dual integration allows the maximal utility of past experiences, thereby expanding the effective exploration space, while simultaneously mitigating persistent uncertainty.
2

---

2024). Reinforcement Learn- Published as a conference paper at ICLR 2026
To address this, propose the Self-imitation with Progressive Exploration for Agentic Reinforcement learning (SPEAR ). a curriculum based RL recipe for improving the exploration- exploitation balance with self- reff-mmitation and intrinsic reward. As shown in Figure [I] the core prin- ciple follows the vanilla Self- leff-Imitation Learning (SIL.) (Oh et al. 2018 Ferret et al. (020 where an independent replay buffer is prepared to store the state-action pairs only when their returns in the past episodes exceed the baselines. Such a replay buffer is exploited to encourage actions with good returns and improve hard exploration based on these successful trajectories under the sparse- reward, oong-horionn agent tasks. Specifically, we introduce three modifications to SIL tailored to the dynamics of policy entropy in agentic tasks. First, we incorporate curriculum to integrate both skill-level and action-level exploration by adjusting reward shaping and self-imitation across stages. Second, we tackle the off-policy nature of the update with experiences the buffer and avoid advantage recomputation by advantage recalibration. Third, we regularize policy updates to stabilize entropy and mitigate reward hacking. Finally, inspired by existing industrial bag-of- -r-ccksks, present a strong baseline BoT for agentic RL training. Our SPEAR brings considerable per- formance gains to GRPO/GiGPO (Feng et al. 2025bVDr. respectively up to 16.1% 16/8.6% on ALFWorld (Shridhar et al.) 2020) and 20. 11/.1..8//1..999 on WebShop (Yao al. (002)) It boosts our Dr.BoT respectively up to 3.8% on AIME24 and 6.1% % on AIME25 (AIME) 2025). These gains come with around 10% 25% computation overhead in theoretical complexity, but end up with quite comparable runtime per iteration in practice. Such compatibility and scalability enable SPEAR a allugandd-play algorithm for training versatile agents. In summary, our contributions are:
1) We propose SPEAR, a generalization of the SIL for training LLM agents. It bypasses the costly expert imitation and allows exploration under the guidance of one's own rewarded experience.
2) We bring in curriculum scheduling to harmonize SIL with intrinsic reward shaping for policy entropy management and progressive transition from skill-based action-based exploration.
3) We propose a strong baseline, Dr.BoT, which combines established RL techniques validated in industrial practice, confirming its effectiveness and superiority over existing baselines.
2 RELATED WORK
2.1 REINFORCEMENT LEARNING ALGORITHMS FOR LLMS
With the advent of large-scale reasoning models (Jaech et al.
ing (RL) (Ouyang et al. 2022) has been adopted more broadly. Proximal Policy Optimization
(PPO) (Schulman et al. (017b) leverages an actor-critic architecture together with the clipped sur- rogate objective and Kullbac--Libler (KL) divergence penalty constrain policy update. Group Relative Policy Optimization (GRPO) (Guo et al.] [2255] Shao et al.) (024) simplifies this setup by replacing the critic with group-wise baseline. Building GRPO. DAPO (Yu et al. 2025) uses dynamic sampling and "clip higher" to encourage exploration and stabilize training. Dr. GRPO (Luu al. 2025b) addresses length bias and the difficulty bias. Existing methods have greatly advanced RL for LLMss However, naively combining them can lead to conflicts or tight couplings among tech- niques. To this end, we harmonize the strengths of DAPO, Dr. GRPG, and other agent studies from research and industrial practice to establish strong baseline, Dr. as detailed in Section 4.4
2.2 OPTIMIZATION OF LLM AGENTS
Recent researches investigate how to endow models with better tool-use capabilities (Feng et al. 2025a Li al. 2025d] Xue et al. 2025). LLMs are optimized to strengthen information seek- ing from open web (Jin et al.] 20256 Tao et al. 2025| Gao et al. (025). RAGEN (Wang et al.) 20256) improves the stability of multi-turn RL through instance filtering and gradient shaping- GIGPO (Feng et al. 2025b) augments group-level advantages with additional step-level advantage estimates. ARPO (Dong et al.) 2025b) monitors entropy dynamics during rollouts to branch trajec- tories adaptively. In this work, we address the pxploration-exploittain dilemma under multi-turn tool-use settings. We introduce a curriculum-regulate RL regime that gradually shifts baiee-based exploration towards action-based exploration. We integrate self-imitation and intrinsic reward to consolidate successful behaviors (Section 4.2). Our SPEAR can work with existing algorithms in a allg-anddplay manner, exhibiting a high level of compatibility and generalization.
3

---

Published as a conference paper at ICLR 2026
2.3 EXPLORATION IN REINFORCEMENT LEARNING
Curiosity-driver methods (Pathak et al. 2017 Houthooft et al. 2016) grant intrinsic rewards for pre- diction error or novelty to actively seek unfamiliar states. Count- based algorithms (Bellemare al. Tang et al., (017) introduce pseudo-counts derived from a density model to assign count-based bonuses. Skill acquisition methods (Gregor et al., 2016) Eysenbach al. 2018) discover distinct options by maximizing the mutual information. Entropy-regularizatio methods (Haarnoja et al.) 2018 Cui al.) 2025b) maximize the expected reward and entropy of the policy. However, tradi- tional exploration techniques can lead to divergence of agent LLMs as the multi-turn interactions already result the increased uncertainty on unfamilliar observations. Under such circumstance, propose the curriculum siriide self-imitation to leverage the agent's own experience for balanc- ing exploration and exploitation. It avoids handcrafted heuristic techniques in previous studies and instead fully relies the agent itself to reinforce successful and valid patterns.
2016
2.4
EXPERIENCE REPLAY IN REINFORCEMENT LEARNING
Self- Imitation Learning (SIL) (h et al., 2018 takes advantage of past successful experience to drive its future learning (Schaul et al. 201 Horgan et al. 2018 Gangwani et al. 2018: Pan et al.)
2022 Saglam et al. 2023). SAIL (Ferret al. 2020) extends SII. to off-policy, action value- based RL methods. Tang (020) proves that SIL's return return-based update provides a virianar trade-off that speeds up learning. SILID (Pshikhachev al. 2022) extends SIL leverage both external demonstrations and the agent's own experience. GSIL (Xiao et al. (024) proposes an offline alignment framework that uses self-imitation on demonstration data. While SIL benefits long-horizon problems, its induces entropy collapsing to agent RL. To mitigate this, we harmonize both self-imitatin and intrinsic reward with curriculum scheduling for progressive exploration.
3 PRELIMINARIES
3.1 PROBLEM DEFINTTION
Given a task ~ p(X) where p(X) represents data distribution, an LLM agent parameterized by I interacts with the environment E until it completes the task or exceeds the max number of turns T. can be modeled by Markov Decision Process (MDP) where 8t, a,, and R, respectively denote the state, action, and reward at time t. Given full episode T {(S) a Rt) (82, a2, R2),... ... we aim to optimize the agent policy тy. Following previous studies (Dong et al. 2025a; Feng et al.) 2025b
Dong al. 2025b), we define three distinct types of actions (see Appendix A.2).
3.2 POLICY OPTIMIZATION
We adopt the GRPO (hao et al. 2024) which stems from PPO (Schulman et al. 2017a b) but replaces the model-based A Schulman et al. 2015) with the group- based A (Appendix A.3
group-based
TRAINING AGENTIC LLMS WITH SPEAR
4.1 PRELIMINARY FINDINGS
The extension of SIL to LLM-driven agents faces entropy collapse. Figure illustrates that the over- fitting of the few available successful experience causes irreversible stagnation exploration. In ad- dition, demonstrate that the inclusion of the tool-call reward is couble-edged sword (Figure ) where the competition between reward terms causes the oscillations to converge. To address these challenges, introduce SPEAR for progressive exploration with self-imitation (Algorithm) [).
4.2 SELF- LEARNING
We resort self-in seff-imitation to unearth past successful experience for effective action-level explo- ration, where the agent learns novel strategies along the promising decision path instead of random walk and bifurcation. We prevent policy entropy divergence by replaying rewarded trajectories.
4

---

Published as a conference paper at ICLR 2026
330
Consistont Uncortainty
0.60
Stratogic
300
Acquisition
230
ADME23 AccuraoO (00a2e3300
0.50
WWW
200
0.45
Evolution
0.40
Bottieneck
1.00
Convergent
Confidunce
0.35
Otagmantt Performance
100
0.30
50-
Hanelime
Beeelin
Exptoration
wi Drsprrtence Repbay (SI1)
Expertence Repiay 6SI1)
wi Self Euitation ISPEAR
wi Self instatioon (SPEAR)
150
200
250
300
150
200
250
(a) Entropy (aeg-mcan-tooen--uun--
(b) Accuracy on AIME 2025.
Figure 3: Effect of our self-imitation on action-level strategy exploration (Qwen2.5-32B with code interpreter). The vanilla experience replay technique (Oh et al. 2018) that enforces early overfitting of the few available trajectories in the buffer causes entropy collapsing and exploration shrinkage. At the beginning. the LLM agent struggles at tool-calling skills and fails to cultivate the transition of distribution towards frequent tool utilization and tool-integrated reasoning. The naive replay lim- its the transformation of reasoning paradigm. In contrast, our SPEAR introduces both curriculum- and covariance- based regularization into self-imitation. Its curriculum schedule with an increasing emphasis the replay data allows easy acquisition of tool-use skills at first, and stimulates strategic action plans later. The covariance clipping removes over-confident tokens, whose log probabili- ties are highly associated with their advantage gains, out of optimization. Our self- aeff-imitation gives promises to exploring novel strategies and achieves steady growth on AIME 2025.
Prioritized Experience Replay in Self-Imitation. A replay buffer is maintained to store previous trajectories, their rewards and advantages D = ((T), Rj, Aj)).j = 1, 1,2,... ..., No where Np denotes the buffer size. To exploit only good trajectories, we keep those with positive advantages:
No
JoRPo((T)) (r, ~
Σ
JéRoO 1(A, 0),
<(mo ( (le), xxp(())
where the indicator 1() equals to 1 when the condition satisfied and 0 otherwise. The past trajecto- ries not only come from the last policy Te.e but also the policies {Te... of few steps earlier.
Advantage Recalibration for Off-Policy Estimation. We propose to recalibrate the advantage of trajectories in the buffer address the underlying off-policy challenge. That is to say, the observed return of trajectory from the past policy becomes increasingly different from the current one, under the assumption that the policy keeps improving during iterations (Ferret et al. 2020 Luo et al. (02T). Under this assumption, vanilla SIL computes the advantage with pointwise max with the per-state empirical return as a baseline, which can be seen as a proxy for the upper-envelope projection of the value function onto empirical returns. GRPO removes the learned value baseline by estimating the state-dependent baseline performance through its reliance on intra-group reward averaging, but this still depends the target policy and requires extra computation resources for sampling. Dynamic adjustment on the baseline performance is performed to calibrate relative gains without introducing additional computing. Specifically, we maintain First-In-First-Out (FIFO) buffer of intra intra-group baselines for the latest NDn trajectories DR { the size of the baseline buffer. As training progresses, due to the high variance nature of agentic RL, we utilize the 50-th percentile (DDn) as conservative but robust estimation of the policy baseline with either upward or downward trends. To bypass the inaccurate estimation of intra- sirra- standard deviation, we follow (Liu et al.) 2025b) to simply remove such term in advantage computation:
Pn where Np, denotes
(2) Such recalibrated advantage enjoys three benefits: 1) the baseline performance correlates with the policy change; 2) the outdated experiences be filtered with both Å, 0 and A, 0; 3) the
R (D)(
5

---

Published as a conference paper at ICLR 2026
difficulty bias by group normalization can be mitigated.
The updated off-policy SIL objective is:
No
GRPO (T))
E (r, (r,)
JOPOO 1(A, 0 A, 0),
(3)
( xxp(X)) j=1
T
1
JOPO
- +)) ki (Tol(
(4)
T
Progressive Experience Utilization with Curriculum Schedule. We perform scheduling to 1) restrict mechanical imitation of probable-yet-immattee consistent uncertainty about the environment states and policy actions at later stage. We apply a warm-up 7 on the SIL term under the assumption that initially the transition of distribution towards diverse actions outweighs the imitation of limited solution patterns (see Equation 13 and Figure 6a
experience
early stage, and 2) prevent
Tooaal = JGRPO (TA) +
TeRPO (TB)).
(5)
0.60
Marl
ADHESS Accursoy (DA00833300
0.55
Tealuse
0.50
Dominance by Outcome Rewarr
Toakkalll
Overfong
Tendency
Reword Valkgg
Outcome Rowwrr
Distributtio Transition Fallure
withoout Coding
Beward
Reward (SPEAR)
Beward (SPEAR)
150
150
200
250
50
300
150
200
250
Steps
Stops
(a) Number of tool-call turns.
(b) Accuracy on AIME 2025.
Figure 4: Effect of our intrinsic reward on skill-level strategy cxploration (Qwen2.5-32B with code interpreter). The baseline does not consider tool-calling as a rewarded behavior and its number of interaction with the environment drops quickly due the negative feedback of bad codes. In this case, the LLM gives up coding and degrades to text-based reasoning. The vanilla tool-call reward, despite being effective in learning tool-call skills first, causes competition with the outcome ward later. Due to the limited context length, the excessive tool-call turns prevents submission of the final answer and thereafter the accuracy declines immediately. We propose the curriculum schedule as an intrinsic reward design where its strength decays over step to allow the agent to merely focus on the accuracy with wiser actions. prevents reward hacking for unnecessarily long interactions.
4.3 INTRINSIC REWARD SHAPING
We resort intrinsic reward for skill-level exploration where the agent is guided by tool-call reward to broadly investigate tool usage. Such design not only benefits tool learning but more importantly stimulates interactions that familiarize the agent with the environment for experience accumulation.
Reward Composition. A compound reward R' of each trajectory tı not only considers the final outcome but also the behaviors that are promising to achieve the goal: an outcome accuracy reward Hi R a continuous tool-call reward Ri Ri and a format reward Ri (see Appendix A.7).
Progressive Reward Modulation with Curriculum Schedule. We regulate the contribution of tool call reward to: 1) accelerate the mastering of tool usage for quick distribution shifting towards new task settings at early stage, and 2) prevent optimization oscillation and competition later

---

Published as a conference paper at ICLR 2026
stage. Although previous studies (Qian et al., 2025 Li al. 2023 Da et al. 2025 Xia et al. 2025 Singh et al.) 2025; Wei et al. 2025; Gou et al. 2023] Lin & Xu 2025) experimented with various auxilarr rewards, agent trained without the tool-call reward fails to develop tool-integrated reasoning (Figure 4 due to negative tool response: 1) missing i import of modules; 2) reference undef nnd variables; 3) unexpected indentat on inn error; and 4) forgetting to print results. The agent quickly gives up coding to run away from errors and turns to pure textual reasoning. On the other hand, the enforcement of tool-call reward stimulates an increasing number of interaction turns, leading to over- long responses that cause oscillation to outcome accuracy. We alleviate the competition between reward terms by scheduling the tool-call reward with μ (Equation 14 and Figure 6b
show that the addition of tool-call reward is double-cdged sword. The
R
R Routsome + 1 Ranot-call + R Rormat
(6)
4.4 Dr.BoT AS A STRONG BASELINE
To provide strong baseline, we refer to the existing studies (Liu et al. 2025c Sun et al. al. 2025a Cui et al. 2025b) for diverse exploration, stable convergence, and effective training. Our baseline, Dr.BoT, consists of bag-of-tricks modifications to the GRPO (see Appendix A.8
2025 Bai
Table I: Results on ALFWorld & WebShop (%). PT & FW stand for prompting & framework.
ALFWorld
WehShop Score SR
Type Method
Pick Look Clean Heat Cool Pick2
QwenZ
5-1 38-Imtinc
PT
IO
5.9
5.5
3.3
9.7
0.0
4.1
23.1
5.2
ReAct
17.4
20.5
15.7
6.2
2.0
12.8
40.1
11.3
FW
Reflexion
35.3
22.2
21.7
13.6
19.4
3.7
21.8
55.8
21.9
RL
PPO
64.8
40.5
57.1
60.6
46.4
47.4
54.4
73.8
51.5
RL
RLOO
$8.3
52.8
71.0
62.8
66.4
56.9
69.7
73.9
52.1
RI
GRPO
85.3
53.7
84.5
78.2
59.7
53.5
72.8
75.8
56.8
SPEAR
RL
DoBoT (GRPO)
92.2
75.8
81.0
81.8
728
61.9
79.1
78.7
62.9
RL
SPEAR (ours)
72.2
94.1
95.1
88.3
74,4
87.7.81%)
88.4
wisd
67.5
79.8
RL.
GIGPO win aid
96.0
76.5
91.8
91.3
71.7
79.5
86.1
83.5
67.4
RL
SPEAR (ours)
95.2
79.2
89.1
94.0
88.8
95.5
91.245.15)
90.7
DeBoT (GIGPO)
91.4
93.7
93.8
85.4
78.4
90.6
84.1
RL
SPEAR (ours)
96-4
86.5
96.
99.0
87.6
91.6
93.20224%)
90.9
81.
$1.11932299
2wwen2,5-78-Inwii
PT
NO
33.4
21.6
19.3
6.9
2.8
3.2
14.8
26.4
7.8
FW
ReAct
48.5
35.4
34,3
13.2
18.2
17.6
31.2
46.2
19.5
30.9
RI.
PPO
92.3
64.0
92.5
89.5
80.3
68.8
80.4
81.4
68.7
RL RLOO
87.6
78.2
87.3
81.3
71.9
48.9
75.5
80.3
65.7
GRPO
74.7
72.5
77.6
79.3
RL
SPEAR (ours)
93.7
62.4
97.2
78.0
83.1
75.5
85 (5.2%)74%)
92.4
RL
De (GRPO)
99.9
95.8
93.8
92.8
90.4
80.6
92.4
90.4
$0.5
RL
SPEAR (ours)
95.8
97.9
97.1
88.5
89.2
87.2
L..8%)
91.4
+4.8%
RL
GiGPO windd
97.7
82.7
98.8
83.7
89.3
79.2
90.8
$4.4
72.8
GIGPO win
88.6
90.2
75.2
RL
SPEAR (ours)
99.9
62.4
98.0
92.8
92.6
86.6
94.1-19%)
92.7
83, (3
RL
Dr Hol (GIGPO)
98.3
99.9
96.9
92.8
91.8
88.3
94.0
90.7
$1.8 8
SPEAR (ours)
94.7..875)
85.7.4395)
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Three benchmarks are used: ALFWorld (Shridhar et al. 2020). WebShop Yao et al. 2022), and
DAPO-MATH-I7K (Yu et al. (025) (Appendix(A... 10). According to these benchmarks, we respec- tively follow (Feng et al. (025b) and (Feng et al. 2025a) to report range of competitive baselines (Appendix A.II). All the training settings and paramm-praeeters are detailed in Appendix A.12
5.2
PERFORMANCE
Table I demonstrates effectiveness on ALFWorld and WebShop. It is compatible with GRPO (Shao et al., 2024). GiGPO (Feng et al. 20255). and our Dr.BTT. SPEAR brings consis- tent gains across 1.5B and 7B models up to 20% Such generalization benefits from the collection of
7

---

Published as a conference paper at ICLR 2026
successful trajectories, which acts walkthrough guide to the agent. Especially for tasks where the success rate is fairly low at the beginning. the agent has to figure out the underlying interaction logics and summarize action plans tailored specific each task. The experience replay expedites the accumulation of tactics and thereafter reduces blind trials and errors. Furthermore, our Dr. Do'BoTT boosts GRPO and GiGPO up to 15%, showcasing the validity of mixture of tricks.
Table 2: Results (mean @ 30) on AIME 2024 & 2025 (%). 1;
Official implementation already utilizes
DAPO tricks. F; Official results reported by Qwen Yang et al. 2025 PT stands for prompting.
Type Method
Model
Tool
Train Test
Context
AIME24
AIME25
MO
Qwen2.5-322 5-228-lnstic
16K
13.4
12,9
PT
D/O
Dween.55332--Mm
CI
16K
29.6
23.1
PPO
Dwven.55338-IInstwee
16K
16K
55.0
RL GRPO
Swvm2.5.328.fHae
CI
16K
16K
60.0
RL ReTool
Seen2.5-328-Inssiie
C
16K
I6K
67.0
49.3
RL. SimpleTIR
Qwen2. 5.328.
CI
12K
12K
59.9
49.2
RL ZeroTIR
Owen2. Smen2.5-328-Inssiie
C
8K
8K
56.7
33,3
RL AFM
Qwam2.5.328 Instruct
CI
32K
32K
66.7
59.8
RL DrBoT (GRPO)
Swee2.5-328-Inssee
CI
16K
16K
64.7
54.0
RL SPEAR (ours)
Qwen2 5.328 Instruct
CI
16K
16K
(6.3))))))
(0.16)))))
RL Bo (GRPO)
Qwen2.5-33 5322--nstt
CI
32K
32K
67.2
55.1
RL SPEAR (ours)
Qwen2 2.t.2255322--sseee Instruct
CI
32K
32K
71 71.0.33))) 61.0c-995)
PT
NO
Qwen3-328-
16K 68.5
53,5
PT
TOO
Owen3-3288 12sttrrtt
38K
81.4
72.9
NO
Qwen3-328-
CI
16K
31.1
24.4
RL
DoBoT (GRPO)
Qwen3-3286 -aatttre-
CI
16K
16K
81.3
74.1
SPEAR (ours)
Qwen3-328-lasteee
CI
16K
16K
81.80058)
78.8+476)
RL. Do.oT (GRPO)
Qwen3-3221 -aatttre-
C
32K
32K
82.5
77.3
SPEAR (ours)
Qwen3-328-lusruece
32K
32K
85.60318)
80.5.332)
Table Treporss the performance of Cl-integrated reasoming on AIME24 and AIME25. Dr.BoTindeed outperforms recent RL baselines. The reduced context length of Qwen3 impedes complete reasoning and answer parsing. The agent learns to exploit the CI feedback for double-check and reflecofect..n. SPEAR achieves comparable performance with Qwen3 but using a much smaller token budget. When the context is relaxed to 32K, improvement is observed on both Qwen2.5 and Qwen3, confirming our generalization with more interactions turns and reasoning tokens.
5.3 ABLATION STUDY
Table 3: Ablation on ALFWorld & WebShop. SI & IR stand for Self-Imitation & Intrinsic Reward.
Type Method
ALFWorld Look Clean
WebShop Score SR
Pick
Heat Cool
@wen2.5
S.J.2.SJJ.. 5B
Instruct
GRPO
85.3
53.7
84.5
78.2
59.7
$3.5
72.8
75.8
56.8
RI.
SI
86.8
61.0
87.4
87.7
71.1 56.6
77.3-45%)
85.1
74.2.17455)
RI.
SE IR (SPEAR)
93.9
80.9
96.4
874
88.3
79.1
88.9
90.0
77.5+207%)
GiGPO
91.8
91.3
79.5
86.1
RL SI
93.2
82.5
96.3
87.4
92.7
87.5
90.6-448)
89.4
79.0
19.0.1159)
IR (SPEAR)
95.2
79.2
89.1
94.0
88.8
95.5
91.2(-595
90.7
39.3.)
Owen2 2on22.5775
GRPO
90.8
66.
74.7
72.5
64.7
77.6
79,3
66.1
RL
SI
93.2
82.5
96.3
87.4
92.7
87.5
90.6143885)
90.4
83.4.1734)
RL
SI
IR (SPEAR)
93.7
62.4
97.2
78.0
83.1
75.5
85.2
92.4
84.6
GIGPO
90.2
85.2
86.2
RL
+I
96.1
81.9
98.4
95.3
94.5
83.9
93.61-14%
94.6
87.5
RL
SI (SPEAR)
99.9
82.4
98.0
92.8
92.6
86.6
94.1--399)
92.7
83.808395)
Self-Imitation. The SIL. improves baselines consistently across model scales (Table 3). Since ei- ther 1.5B or 7B models perform poorly at the early stage (i.e., success rate < 15%), past experiences are quite beneficial to explore promising strategies. The re-use of trajectories facilitates convergence and prevents mechanical trials especially for small agents. Table 4 shows that AIME24 dropped bit by self-imitation but AIME25 still gets improved. Such fluctuation is related to the phenomenon (Figure 4 where the imitation of samples with multiple tool calls leads to rapid increase of interac- tion turns and thereafter causes training instability. The competition between different reward terms affects the robust selection of good experience, ultimately degrading the effectiveness of SIL..
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation on AIME 2024 & 2025 (%). SI & IR stand for Self-Imitation & Intrinsic Reward.
Type Method
Model
Tool
Context Train
Test
AIME24
ALME25
RL
Dr.BoT (GRPO)
Swwen25--28--natte
C1
16K
16K
64.7
$4.0
RL
+
Qwen2.5-328-fiitte
CI
16K
16K 63.8
56.9v229)
RL
+ +SI+ IR (SPEAR)
Swven.5-328--iiie
CI
16K
16K 66.36169)
60.106619)
RL
Do.BoT (GRPO)
Qwen2.5-328-fnstin
CI
32K
32K 67.2
55.1
RL
+SI
Owen2. 1eon2.3-228-Insmmee
CI
32K
32K
66.0.129)
60.5us8.99
(SPEAR)
Qwen2.5-3228
32K
32K
71.0-389
61.0k4893)
RL
Dr.BoT (GRPO)
Qwend-33BB-Iein
C1
16K
16K
81.3
74.1
RL
+SI
Qwen3 328 Instruct
C1
16K
16K
81.2-ars)
+ IR (SPEAR)
Qwend-32B 3-s-rrstee
C1
16K
16K
a188
RL
Dr.BoT (GRPO)
Qwend 32B Instruct
CI
32K
32K
82.5
77.3
RL
+
Qwend-3225 Saend---B--nsttuerr
C1
32K
32K
81.8-179)
(8.2))
RL
+ SI + IR (SPEAR)
Qwen3 32B Instruct
32K
32K 85.6-315)
80.5
Intrinsic Reward. The rewarding of interaction turns benefit 1.5B models consistently (Table 3 Two 7B outliers are found where the self self-imitation alone brings the most performance gains. Such exception might be related to both the task definition and the RL. algorithm. One should experiment with different combinations in practice. Table + shows that the intrinsic reward is indispensable for both Qwen2.5 and because it encourages transformation from text-based reasoning to tool- integrated reasoning. It promotes frequent tool calling and such rich observation signals motivate the agent to correct coding errors, check the validity of the answer, and reflect on alternative solutions.
5.4 GENERALIZATION ON VISION-LANGUAGE AGENTS
Table 5: Success rate (%) of the visual agent for playing Sokoban.
0
0
E
Type
Method
Sokoban
0
O
Qwen2.5-VL Snen2.5-V2-38--nntrra
6
E
PT
1O
11.7
RL
GRPO
67.1
Q
@
RL
SPEAR (ours)
86.7:-188%
RL
Dr.of (GRPO)
76.0
RL
SPEAR (ours)
85.4 85.4w044)
RL
GIGPO w/ std
76.9
RL
GIGPO wlo std
81.0
RL SPEAR (ours)
87.7
RL DrBof (GIGPO)
81.3
(a) Before (step 15).
(b) After (step 125).
RL SPEAR (ours)
87.90669)
Figure 5: The agent learns to push the box.
To test whether the proposed SPEAR is still complimentary to existing GRPO-like algorithms on training visual agents, we follow (Feng et al., (025b) to conduct experiments on the popular visual game Sokoban (Schrader 2018). In this setting, the Qwen2.5-VL-3B ---3---rstttt (Bai et al.) 2025b) is adopted as the agentic LLM to solve the puzzle game where the player must push the boxes along the grid towards target positions without hitting the walls. It challenges the agent spatial comprehen- sion and long-term planning capabilities. The grid size is of x 6 and the visual agent receives both the visual (RGB arrays) and textual inputs states. As shown in Table the proposed method gen- erally improves the performance on Sokoban with either GRPO, GiGPO, and the proposed Dr. baselines. At first, the visual agent is unaware of the winning logic behind the game and wanders around for "aimlessly" exploration (see Figure 5). After optimization, it not only comprehends the spatial relationship to control the box but also learns to stop moving when the task is completed.
5.5 GENERALIZATION ON AUGMENTED QA TASKS
To evaluate the performance of SPEAR on knowledge-intensive reasoning tasks, we conduct ex- periments on saarch-augmented QA tasks, including the single-hop QA datasets (NQ (Kwiatkowski et al. 2019), TriviaQA (Joshi et al. 2017). and PopQA (Mallen et al. 2023)) and multi-hop QA
datasets (HotpotQA (Yang et al. 2018). 2Wiki (Ho et al. 2020), MuSiQue (Trivedi et al. 2022). and Bamboogle (Press et al. 2023)). We follow the experimental settings of SearchR1 (Jin al.

---

Published as a conference paper at ICLR 2026
Table 6; Results on search-augmentted QA Tasks.
Single-Hop QA
Multi-Hop QA
Type Method
NQ InviaQA PopQA HotpoiQA 2Wiki MuSiQue
Bamboogle
Avg.
Qwen2. S. 7B Instruer
Seurch-R
61.0
39.7
RL
SPEAR
35.7 62.7
34.5
46.9
43.4
17.2
44.8
40.7
5-en2.5-44B-ossraaet
RL Search-RI
48.8 67.7
48.2
45.5
47.0
21.1
51.6
49.1
SPEAR
69.3
47.8
48.5
56.6
2025b Gao et al. 2025) to launch the local wiki-18 retricval service. We adopt the Hicrarchical Navigable Small World (HNSW) CPU indexing as approximation of nearest neighbor retrieval. Our SPEAR with GRPO improves over the Search-RI baseline on average, especially on the multi-hop QA benchmarks. Such multi-hop QA datasets require reasoning for problem decomposition and several turns of information seeking. In this case, our intrinsic reward that encourages multiple tool uses for broad exploration prevents arbitrary conclusions with only or two searches. Our SPEAR respectively requires ~ 14.48 and ~ 14.42 calls for 7B and 14B models, respectively. Such behavior is expected due to the stimulation of exploration at the beginning. Despite the QA tasks are relatively nort-horiizon, the agent still benefits from the detailed decomposition of the complex queries with validation via step-by-step searching. Note that our retrieval service adopts the HNSW E5 embedding for efficient training, which slightly impedes performance (Jin et al. (025a).
5.6 MORE DISCUSSIONS
Due to the page limit, discussions theoretical analysis convergence parameters 13 qualitative analysis/A.14] training cost and complexity AA and future research directions 1.16 are presented in the appendix. One could easily adapt SPEAR to training any (M)LLM-driven agents robustly without binding to a specific optimization algorithm.
A.9 hyper-
6 CONCLUSIONS AND LIMITATIONS
In this paper, we target the pivotal challenge of balancing exploration and exploitation in RL training of L.LM agents. Our proposed solution, SPEAR X extends the vanilla SIL by advantage recalibra- tion, scheduled entropy control, and intrinsic rewards. These components work in curriculum man- ner to prevent policy collapse and excessive uncertainty, progressively guiding the policy through smooth transition between exploration and exploitation. In addition, we propose a strong base- line Dr.BoT tailored for agentic RL with existing bag-of-tricks verified from numerical industrial existing
practices. Empirical results across tasks and models showcase SPEAR's superiority methods, with performance boosts and acceptable computational overhead. The effectiveness of our SPEAR underscores the value of learning from past experiences while managing policy entropy, offering a robust framework for training L.Ms with strong reasoning and tool integration skills.
There exist two potential limitations: 1) The vague definition of good experiences under highly complex, stochastic environments with unreliable tools. In such cases, observations can be noisy and severely degrade the feasibility of the task. The sparse outcome reward cannot distinguish between good and bad experiences and thereafter the relative advantages might be simply attributed randomness instead of the agent's behavior. We We suggest a possible solution that more fine- gra--griied, stepwise supervision should be enforced. For example, a step-wise process reward that evaluates the logical consistency (Zhang et al. 2025b) of the agent's thought and action might be helpful. 2) The rigidity of entropy control which relies on prior-based scheduling and covariance- the present study, the proposed scheduling and clipping designs might not be optimal for all kinds of agentic tasks. A more adaptive solution lies in the policy's 'ss self-confidence on decisions under each observation. One might use the token-level dynamic reweighting for SIL (Wu et al. 2025) which avoids over-concentration certain low sow-rrobability reference tokens in the replay buffer. Similarly, the clipping could depend on token probability instead of the bounded random sampling. We leave the exploration mentioned above promising direction for improvement in the future.
clipping. In
10

---

Published as a conference paper at ICLR 2026
Ethics Statement. The present study conforms with the ICLR Code of Ethics. The paper does not involve crowdsoureing nor research with human subjects.
Reproducibility Statement. All datasets used in the paper are publicly accessible (see Sec- tion 5.1. All the codes are available at httttp:://ggthhie com www.///eeeetteteeeeeeeeeeeeeeeee Yout SPEAR for reproduction. In addition, we provide all the details of implementation in Section A.12
Acknowledgment. Zhengbao He acknowledges support from the AI for Science Program, Shang- hai Municipal Commission of Economy and Informatization (No. 202-G7ZL-RGZN-BBBX 02026).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.055 087774740774 2023.
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armes bandit prob- lem. In Conference on learning theory, pp. 39-1. JMLR Workshop and Conference Proceedings, 2012.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms. arXiv preprini 2rXiv:2402.114444 2024.
AIME, Aime problems and solutions. http::///attorrrooo.ooooooo...o.
index.php/Ane.. AIME Bhtp/AAMM_PPobblmss and Solutions] 2025.
Fei Bai, Yingqian Min, Beichen Zhang. Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Towards effective code-integrated reasoning. arXiv preprint arXiv:2505.24480. 2025a.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025b.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Bytedance-Seed-Fonne -oeeeed-oondtiionnCCCe Conoon-Coooee-eaaen :, Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, Bowen Li, Linyi Li, Boyi Liu, Jiaheng Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Liu, Rui Long, Jing Mai, Guanghan Ning, Z. Y. Peng, Kai Shen, Jiahao Su, Jing Su, Tao Sun, Yifan Sun, Yunzhe Tao, Guoyin Wang, Siwei Wang, Xuwu Wang. Yite Wang. Zihan Wang, Jinxiang Xia, Liang Xiang. Xia Xiao, Yongsheng Xiao, Chenguang Xi, Shulin Xin, Jingjing Xu, Shikun Xu, Hongxia Yang, Jack Yang. Yingxiang Yang, Jianbo Yuan, Jun Zhang, Yufeng Zhang, Yuyu Zhang, Shen Zheng, He Zhu, and Ming Zhu. Fullstack bench: Evaluating llms as full stack coders, 2025. httpst //tp://aaavvvvv.aaiiia org/ abs abw.aaaaa4400 21122212.0000
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Marc-Alexandre Côté. Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Textworld: learning environment for text- bastd games. In Workshop on Computer Games, pp. 41-75. Springer, 2018.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards. arXiv preprini arXiv:2502.0144 0/456, 2025a.
11

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505. 2025b.
Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent-rlvr: Training software engineering agents via guidance and environment rewards. arXiv preprint arXiv:2506. 1425, 2025.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint arXiv:1205.4833/ 2012.
Wenlong Deng, Yi Ren, Muchen Li, Danica J Sutherland, Xiaoxiao Li, and Christos Thrampoulidis. On the effect of negative gradient in group relative deep reinforcement optimization. arXiv preprint arXiv:2505. 18830, 2025.
Guanting Dong. Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. Tool-star: Empowering llm-brained multi-tool reasoner via reinforcement learning. arXiv preprint arOiv:2505. 16410, 2025a.
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507. 2025b.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: arXiv:1802.06070 2018.
Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.1 /536, 2025a.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-grond policy optimization for Ilm agent training. arXiv preprint 1095, 2025b.
Johan Ferret, Olivier Pictquin, and Matthieu Geist. Self-imitation advantage learning. arXiv preprint arXiv:2012.11989 2020.
Hiroki Furuta, Kuang-Huet Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instructionfiietunee foundation models. arXiv preprint arXiv:2305.1 11844, 2023.
Tanmay Gangwani. Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. arXiv 2018.
preprint 1805.
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint ww.Xiv:2508.07977 2025.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309. 0023309.77455 /7452, 2023.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
Yu Gu, Kai Zhang, Yuting Ning. Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your Ilm secretly a world model of the internet? model- based planning for web agents. arXiv preprint urXiv:2411.06552 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Tuomas Haaroja,, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
12

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. Pmlr, 2018.
Jianye Hao, Tianpei Yang. Hongyao Tang, Chenjia Bai, Jinyi Liu, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: From single-agent to multiagent domain. IEEE Transactions on Neural Networks and Learning Systems, 35(7):8762-8782, 2023.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.0400 2024.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint urXiv:2401.33919 2024.
Xanh Ho, Ahoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint 17Xiv:0011.01066 2020.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint 18Xi::1803.000 00933, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing sys- tems, 29, 2016.
Peter Huber. Robust statistics, In International encyclopedia of statistical science, pp. 1248-1251. Springer, 2011.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16722 2024.
Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-seare interleaved Ilm agents. arXiv preprint arXiv:2505.15117, 2025a.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. 2025b.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint 1705vv/705.03555 2017.
Woojun Kim and Youngchul Sung. An adaptive entropy-regularien framework for multi-agent reinforcement learning. In International Conference on Machine Learning. pp. 16829-16888 PMLR, 2023.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! 2019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
13

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post- post-training. arXiv preprint arXiv:2411.15124 2024.
John Law. Robust Hlaisticss-hhe approach based on influence functions, 1986.
Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. Tool- augmented reward modeling. arXiv preprint arXiv:2310.01045, 2023.
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang. Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking ulms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2024.
Ning Li, Xiangmou Qu, Jiamu Zhou, Jun Wang, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuy- ing Peng, and Weinan Zhang. Mobileuse: A gui agent with hierarchical reflection for autonomous mobile operation. arXiv preprint arXiv:2507. 16853, 2025a.
Weizhen Li, Jianbo Lin, Zhuosong Jiang. Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, et al. Chain-of-agents: End-to-end agent founda- tion models via aulti-agent distillation and agentic rl. arXiv preprint arXiv:2508. 13167, 2025b.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025c.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. 1.Xiv:2503.23333, 2025d.
arXiv preprint
Heng Lin and Zhongwen Xu. Understanding tool-integrated reasoning. arXiv preprint arXiv:2508. 19201, 2025.
Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao. Agentic reinforcement learning with implicit step rewards. arXiv preprint arXiv:2509.19199 19199, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078.. 2025b.
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, et al. Part i: Tricks or traps? a deep dive into rl for Imm reasoning. arXiv preprint 2xXiv:2508.0822 2025c.
Sha Luo, Hamidreza Kasaei, and Lambert Schomaker. Self-imitation learning by planning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pp. 4823-4829, IEEF, 2021.
Xinji Mai, Haotian Xu, Weinong Wang, Jian Hu, Yingying Zhang, Wenqiang Zhang, al. Agent scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.0772: 2025.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannanch Hajishirzi. When not trust language models: Investigating effeetiveness of parametric and non-parametric memories. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, 2023.
14

---

Published as a conference paper at ICLR 2026
Negar Mehr, Mingyu Wang, Maulik Bhatt, and Mac Schwager. Maximum-entroop multi-agent dynamic games: Forward and inverse solutions. IEEE transactions on robotics, 39(3):101-1185,, 2023.
Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-baset rein- forcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoye, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint vrXiv:2501.1939) 2025.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International conference on machine learning, pp. 3878-3887 PMLR, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Yangchen Pan. Jincheng Mei, Amir- massoud Farahmand, Martha White, Hengshuai Yao, Mohsen Rohani, and Jun Luo. Understanding and mitigating the limitations of prioritized experience replay. In Uncertainty in Artificial Intelligence, pp. 1561-1571. PMLR, 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell, Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Mcasuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, 2023.
Georgiy Pshikhachev, Dmitry Ivanov, Vladimir Egorov, and Aleksei Shpilman. Self-imitation learn- ing from demonstrations. arXiv preprint arXiv:2203. 10905, 2022.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958 2025.
Yujia Qin, Yining Ye. Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arxiv preprint arXiv:2501.12322 2025a.
Yulei Qin, Gang Li, Zongyi Li, Zihan Xu, Yuchen Shi, Zhekai Lin, Xiao Cui, Ke Li, and Xing Sun. Incentivizing reasoning for advanced essructtoo-followinn of large language models. arXiv preprint urXiv:2506.01411 2025b.
Yulei Qin, Yuncheng Yang, Pengcheng Guo, Gang Li, Hang Shao, Yuchen Shi, Zihan Xu, Yun Gu, Ke Li, and Xing Sun. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models. Transactions on Machine Learning Research, 2025c.
Baturay Saglam, Furkan B Mutlu, Dogan C Cicek, and Suleyman S Kozat. Actor prioritized expe- rience replay. Journal of Artificial Intelligence Research, 78:639-672. 2023.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: Xiv:15111005522 2015.
Max-Philipp B. Schrader. gym-sokoban.
gym sokoban 2018
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 2015.
15

---

Published as a conference paper at ICLR 2026
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning. arXiv preprint arXiv:1704.06440) 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1eXXi::707.066444 2017b.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International conference on machine learning, pp. 9443-9455 PMLR, 2021.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 14096, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-866355 2023.
Mohit Shridhar, Xingdi Yuan, Alee-Alee Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint arXiv:2010.03766 2020.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for Ilms via reinforcement learning. arXiv preprint wrXiv:2505.0144), 2025.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, al. Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent. arXiv preprint arXiv:241 1.002665, 2024.
Zetian Sun, Dongfang Li, Zhuoen Chen, Yuhuai Qin, and Baotian Hu. Stabilizing long-tem termtermm multi- turn reinforcement learning with gated rewards. arXiv preprint arXiv:2508. 10548, 2025.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3 (1):9-44, 1988.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schul- man, Filip DeTurek, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Yunhao Tang. Self-imitation learning via generalized lower bound q-learning. Advances in neural 2020.
information processing systems, 23:13964-133
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, et al. Webshaper: Agentically data synthesizing via information-seeking formalization. arXiv preprint arXiv:2507.15061 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, dean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.118800 2023.
NovaSky Team. Sky-tl: Train your own ol preview model within $450, 2025a.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning. 2025b.
16

---

Published as a conference paper at ICLR 2026
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association Computational Linguistics, 10:539-5555 2022.
Haoran Wang, Thaleia Zariphopoulou, and Xunyu Zhou. Exploration versus exploitation in rein- forcement learning: A stochastic control approach. arXiv preprint arXiv:1812.01552 2018.
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2. Mobile device operation assistant with effective navigation via multi-agent collaboration. Advances in Neural Information Processing Systems, 37:2686-2710, 2024.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for IIm reasoning. arXiv preprint urXiv:2506.01339 2025a.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, arXiv preprint arXivv2504.20073, 2025b.
Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, and Li Du. Autotir: Autonomous tools integrated reasoning via reinforcement learning. arXiv preprint arXiv:2507.21836. 2025.
Ronald J Williams and Jing Peng. Function optimization using comneciionistrriiffoceemee learning algorithms. Connection Science, 3(3):441-26,, 1991.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng. Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508. 05629, 2025.
Yu Xia, Jingru Fan, Weize Chen, Siyu Yan, Xin Cong, Zhong Zhang, Yaxi Lu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Agentrm: Enhancing agent generalization with reward modeling. arXiv preprint arXiv:2502. 18407, 2025.
Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, and Vasant G Honavar. How to leverage demonstration data in alignment for large language model? self-imitation learning perspective. arXiv preprint arXiv:2410. 10093, 2024.
Bo Xin, Haixu Yu, You Qin, Qing Tang, and Zhangqing Zhu. Exploration entropy for reinforcement learning. Mathematical Problems in Engineering, 2020(1):2672537, 2020.
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforcement learning for multi-tum tool-integrated reasoning, 2025. URL
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via eeff-improvement. arXiv preprint arXiv:2409. 12/22, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 00X9::2505.00388 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2369-2380, 2018
Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, August 2025. URL https: 7/fengyao. not on. w.w.sitee wwt.ssttftotooooooooooo
17

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-2073, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Re- inforcing multi-tum reasoning in Ilm agents via turn-level credit assignment. arXiv preprint a50iv: 1882, 2025.
Chuheng Zhang, Yuanying Cai, Longbo Huang, and Jian Li. Exploration by maximizing rényi entropy for reward-free framework. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 35, pp. 10859-108677 2021.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508. 1408, 2025a.
Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. Rlymr: Reinforcement learning with verifiable meta-rcasoning rewards for robust long-horizon agents. arXiv preprint arXiv:2507.22844 224,24244, 2025b.
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-reeulariveed multi-goal reinforcement learning. In International Conference Machine Learning, pp. 7553-7562. PMLR, 2019.
Richard Zhuang, Trung Vu, Alex Dimakis, and Maheswaran Sathiamoorthh. Improving multi-turn tool use with reinforcement learning. 2025. Accessed: 2025-04-17.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1436 Chicago, IL, USA, 2008.
18

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 SUMMARY OF THE APPENDIX
In the appendix, we provide detailed explanations on the following.
Descriptions about the Action Space
Brief Introduction to the PPO and GRPO
PseudoCode of the SPEAR
Visualization of the Curriculum Schedule
Definition of the Reward Function
Descriptions about RL Pog-of-Tricke
Theoretical Analysis on Convergence
Descriptions of the Data and Environment
Choice of Baselines
Implementation Details
Discussions and Guidelines on Hyper-parameter
Qualitative Analysis
Training Cost and Complexity
Future Research Directions
A.2 DETAILED ACTION SPACE
The following contents correspond Section 3.1 in the main text.
TextWorld Embodied Tool. The embodied actions follows ALFWorld Shridhar et al. 2020
where language-driven agent interacts with the TextWorld (Côté al. 2018). It allows the agent to take one of the following high-level actions: goto {recep]. take {obj} from (recep), put {obj} in/on (recep), open {recep), close {recep), toggle jobjHrrcepp) clean fobi} with (recep), heat {obj} with {recep), and cool {obj} with {recep), where (obj} and {recep} denote objects and receptacles, spectively.
Web Browsing Tool. The definition of web browsing follows WebShop (aoo et al. 2022 where only two actions are allowed: search [query] and choose [button] where query and button respectively stand for searching query and clickable elements such as back to search, prev/nexe page, (product title). (option), (desc/overview), previous, and buy.
Code Interpreter Tool. The code interpreter executes the code generated by the language model and return both the stdout and stderr. If the code runs correctly, the stoout contains the output. On the other hand, the compiler error messages tion. We follow (Feng et al. 2025a) to deploy a SandBox
provided for the next-round correc- SeaeenreeSSeee--Fuattttiiooo Cone--Cooeeeemmmm al. 2025) service that receives execution requests from the interpreter tool. In addition, we add a reminder in the stouut for empty output when the LLM forgets to print computation results: Empty stdout! You might forget to print the answer. For non-empty stderr, we also add an instruction as hint: Errors occurred! Check your code.
A.3 DETAILED POLICY OPTIMIZATION ALGORITHMS
The following contents correspond Section 3.2 in the main text.
19

---

Published as a conference paper at ICLR 2026
Proximal Policy Optimization (PPO). Typically, PPO optimizes the following:
(TO) = Re-p(X)) R(x,s,a) - (xxx.. [mo(- (er( s)]
where R(I, a) = Ei= I' (1,, St, ar) is the return (Sutton et al. 1998) for the trajectory and we the reference policy model. The KI. divergence proposed (Christiano al., (017) to prevent the policy Te from deviating greatly from the reference Tret (3 > 0). In consideration of the simplicity, we follow TULU3 3 (Lambert et al. 2024) to adopt RL with the verifiable reward where the rule- based verifiers are designed to provide the outcome reward signal instead of the reward model re- In addition, we follow (Liu al. 2025b) to drop the KL term by setting 5 0, which not only emphasizes agent performance but also saves memory and computation during training.
Group Relative Policy Optimization (GRPO). Specifically, the policy model Teae from the pre- vious iteration generates group of G individual trajectories (r:). GRPO updates the policy me by maximizing the objective below.
G Σ TORPO
(To) =
~ (12) G 1=1
(8)
Ti [ss. Ri), (s) az. R3), ...,, (s aiy, RT)),
T
TORPO
T Σ min r(()) clippr (r), - + c d/A - (TTo|
(9)
r Tol a
rr
A std( R [
R¹ -ーー R
R mean (1aam(1F) (R' (
(10)
nu (a x st)
Trer Ir, s())
Terr |z, si)) 1. (a [r, s() To (a) x st))
Diu (Tol (Te)
log
(11)
A.4 PSEUDO CODE
The following contents correspond to Section in the main text.
Algorithm D summarizes the full training procedure of the proposed SPEAR. It is noted that our SPEAR is compatible with various baselines such as GRPO (Shao et al. 2024) and GiGPO (Feng et al.) 2025b), enjoying high-level of generalization. Specifically, the algorithm is featured by: 1) Maintenance of a replay buffer and a baseline buffer that respectively stores the trajectories for good experience replay and estimates the current policy's s average performance; 2) Recalibration of the previous advantages for off- poolcy update; 3) Regularization against the pre-mature entropy collapsing: 4) Shaping of the composite intrisic rewards for dominance of the outcome reward.
Compared with the vanilla GRPO-like training, the proposed method only introduced: 1) Additional policy update iterations positively associated with the number of No in terms of computational complexity: 2) A replay buffer of the size No and a baseline performance buffer of the size ND in terms of space complexity.
Since we re-utilize previous trajectories without completely re-computing the rollout generation, log-probability estimation, and the advantages, such operations are light-weight and incur minimal computation overhead. In the present study, we empirically set ND = 2048 without meticulous hyper-parameter tuning. For both ALFWorld, WebShop, and Sokoban, the number of trajectories per data batch is the product of train batch sizexn samples per prompt=256 and there exist around 4K turn-level training samples under the VeRI--agent (Feng et al. 2025b) framework. For the DAPO- MATH-17K, the number of trajectories per data batch is 2048 and there exist exactly 2048 trajectory- level training samples under the VeRL (Sheng et al. (024) framework. In this case, our replay buffer reaches its full capacity around every two or three training steps on average for all experiments. For each policy update by self-imitation, the number of iterations is comparable to that of the vanilla policy update by GRPO under the present settings. The detailed analyses on the training cost and complexity can also be found in Section A.15
20

---

Published as a conference paper at ICLR 2026
Algorithm Training Agentic LLMs with SPEAR
Require: Initial policy ma data distribution p (. clipping bounds élbs cub, KI. penalty / (3 0), replay buffer D with buffer size ND. intra-grout group baseline buffer DR with buffer size ND,: the warm-up factor y with the number of warm-up steps Twam--up. covariance clipping bounds w/bs Wabs the covariance-based clipping ratio À (A 0.02), the decay factor μ with the number of decay stepsTaecayy the group size G, the maximum allowed interaction turns T.
Ensure: Updated policy Thg
1: Initialze D = 0 and DR =
2: for each training step liter do
3: Update the old policy model: fold + 0
4: # Repeat batch sampling and rollout generation for trajectories
5: Sample data batch with each unique sample r ~ p(X)
6: # Sample G' trajectories (Ti G for each
7: for = 1 to G' do
Initialize environment states sj 9: # Sample at most I actions
8:
10: for f to do
11:
Sample action af TT(- (x) Execute actions, receive rewards Ri observe the new states s/+1
12:
13:
end for
14:
Organize the trajectory 7, = {(s} ai, RJ), (sh. al, R2), (s/ a R2 end for
16:
# Apply intrinsic reward shaping for advantage estimation
Compute the vanilla objective JaRPO(TT) via Equation with the secay-scheduled R via
Equation
18:
# Maintain the replay buffer and the baseline buffer
19:
DR < Dn U (R), , Rt = mean( [[+)
20:
while DRl > NDx do
21:
Pop the oldest baseline DR + DR {Ro}
22:
end while
23:
if |D] ND then
24:
# Add trajectories into the buffer only when their advantages are positive
25:
D + DU (Tg|A; 0}
# Apply on-Policy update with the vanilla GRPO
27:
Update policy by maximizing objective (
else
29:
# Recalibrate the advantage
30:
Compute the newly estimated advantage A, for all Tj E D via Equation
31:
Only keep T, with positive A, D (TylAy > VT, E D)
32:
# Apply regularization on self-imitation learning
33:
Compute the self-imitation objective JGHL-B (mO) via Equation 18 with covariance
clipping via Equation 19
34:
Apply the warm-up schedule for the total objective Trocal (To) via Equation 5
35:
# Apply both the on-policy and the off-policy update for self-imitation
36:
Update policy by maximizing objective T (TT)
Reset the replay buffer D
38:
end if
end for
40:
return :
21

---

Published as a conference paper at ICLR 2026
A.5 POLICY ENTROPY
The following contents are mentioned in Section 4.2 in the main text.
The policy entropy quantifies the confidence inherent in the actions triggered off by the LLM. Under the context of agent tasks, we measure the average entropy of the entire trajectory т for the policy model via seauence-man-tooeee Given the training data batch DB, the entropy is defined as:
in accordance with the Dr. GR.G technique (Liu et 2025b).
H(mo, DB) = -ED llg Dog (T(T)) = |DB
E
E
I. Eayaa st)
xEDD, (Sua)e)
(12)
A.6
CURRICULUM SCHEDULE
1.2
Mintzum Rrward of Oooedt Theps
6.3
Reward Rarge of Comest Toxs
Staisum Benart hocees Togs
Jaward Bange dicomm Toga
150
(a) Visualization of y for the SII. term with T Twarm-ap 200. The weight of SIL loss gradually increases from 0 1 the first T Twammup steps.
(b) Visualization of the composite intrinsic reward (Taeaa = 200). The tool-call reward gradually decays from to in the first 200 training steps.
Figure Visualization of the curriculum for progressive exploration.
Self-Imitation.
The following contents are mentioned Section 4.2 in the main text.
The schedule for strengthening SI is defined below:
2 (1 cos
1
fue
)), fnerr S T warm-uu-
Toxnop
(13)
1,
faerr >> T warm--up
where Uner and Tarm-up respectively denote the training iteration step and the total warm-up steps.
Intrinsic Reward. The following contents mentioned in Section 4.3 in the main text.
The schedule for decaying IR defined as below:
(cos(e π Avee
+ 1), liter Taecays
Tasey
(14)
0,
Uieer Taccay,
where Taecay denotes the number of decaying steps.
A.7 REWARD DEFINITION
The following contents correspond to Section 4.3 in the main text.
Outcome Reward A binary signal is assigned at the end of episode according to the pre-defined verification rules.
1,
7: succeeds,
Ri
outcome
(15)
-1,
otherwise.
22

---

Published as a conference paper at ICLR 2026
Tool-call Reward. To incentivize multi turn interactions, Petionnbasedd reward that is propor- tional to the number tool call turns is added. To avoid reward hacking where the LLM repeats meaningless tool calling, the action reward is confined smaller than the outcome reward.
Ribot-call = min (1, 0.1 Prool-call, Prool-call 2 0, where Thool-call denotes the number of valid tool call turns in the trajectory 7;-
(16)
Format Reward. A negligible reward is assigned to the trajectory if the model's output contains valid wrapping format given the task descriptions (e.g.. <think> ... wactiinnooaction................ w...////aiiin.....
{
0.1, if aj is wrapped correctly, V(si, ai, Ri) E m otherwise.
R $10
(17)
A.8 BAG-OF-TRICKS FOR Bo.
The following contents correspond to Section 4.4 in the main text.
Removal of KL Divergence. We follow (Yu et al. 2025 Liu et al. 2025b) to simply remove the KL divergence by setting 3 = 0. This allows the distribution of the LLM to diverge from the initial policy To for adaptation to iool-integrated reasoning under the agent tasks.
Clip-Higher. We follow (Yu et al. 2025 to raise the upper clip bound cub 0.28 and keep the lower bound Ems 0.2 as default. The decoupled lower and higher clipping range leaves more space for the increase of wwwprobabilltyy tokens. It relaxes the exploration of the policy which benefits premature entropy collapsing.
Removal of Intra-group Normalization. We follow (Liu et al. 2025b) to drop the advantage normalization term where the standard deviations lead to difficulty bias in optimization. It has two benefits: 1) The samples with smaller intra-group standard deviations contribute more to the policy update and the removal of normalization allows balancing between samples of various difficulty; 2) The estimation standard deviations are inaccurate for the off-policy advantage recalibration of replay samples. It is challenging to measure the sampling diversity of specifie group.
Removal of Length Normalization. We follow (Liu et al. 2025b) to drop the length normaliza- tion terms. We choose the token-level sum and sequence-level normalization the aggregation approach for both loss computation and the entropy monitoring.
Filtering of Over-long and Void-turn Samples. We follow (Zhuang et al. 2025 Yu et al. 2025) to mask out the loss for rollout samples that exceed the predefined maximum response length. The improper reward shaping for overlong samples introduces noise into training, which causes insta- bility of training. Besides, it prevents from test-time scaling when the context length of evaluation longer than that of training. In addition, we mask out all the trajectories with void turns (Xue) et al. 2025). where the LLM fails call any tools in the response. Such void turns often accom- panied with the occurrence of repetitive reasoning contents, wrong chat-template formatting, and nonsensical tokens. The filtering of these void-turn samples prevents mode-collapsing where their distribution deviate severely from the initial policy.
Filtering of Low-variance Groups. We follow WWag et al. 2025b) only keep groups with high intra-group variance for each batch of training samples. The bottom 25% samples with small intra-group reward standard deviations are removed to keep the policy update informative. High intra-group variance indicates diverse agent behaviors and the contrast between different actions is beneficial to exploitation.
Regularization with Covariance-based Clipping We introduce the covariance- clip- ping (Cui et al. 20255) to the trajectory-Ievel entropy control. The changes of output logits that highly associated with advantage gains greatly decrease the entropy. We remove tokens with
23

---

Published as a conference paper at ICLR 2026
high covariances (Cui al. 2025b Wang al. 2025a) out of loss contribution for JeRB TT),,
preventing aggressive changes of log probability for advantage acquisition.
No
GRPO (m²)
JaROO 1(A, && A, > 0) MM,
(18)
(rr))=i (mea (1a),, (1a)) axp(X)}
0, t l'i
Mf
(19)
1, 1 & leii
Telip Ind Uniform(IJwn
Cov(log no (ail,,,))), A;) S whh, Neti))
(20)
Cov To [o(a] At) (log No (a(a,,,
log To (a (i,,,,s))))) (A) Σ j=1
G
1 G E G j=1
A1), (21)
where the lower bound and upper bound for determining the range of high-covariance tokens are respectively represented as Wp and Wub- The operation Uniform(t]- NNtip) refers to the uniform sampling of tokens with high covariance until budget of Neup tokens. The indices of the selected tokens for loss masking are represented as /nd. It noted that such masking introduces randomness which benefits the convergence of RI. The detailed settings of Wih, Whh, and Nelip are subject to both the LLM and the task. We empirically set the rounded integers of the mean covariance the range of top 20% and top 0.02% respectively for Wib and Wab. and set Nelip XN* with N' being the total number of learnable tokens of 7; and denoting the clipping ratio.
A.9 THEORETICAL JUSTIFICATION
Claim 1. The self-imitation, with warm-up schedule coefficient ((uur) ) that increases from 0 to (Eq. 5.. implements a constrained projection onto the distribution of good responses, ensuring monotonic improvement of the surrogate objective.
Theorem (Surrogate Objective Improvement Bound). Let To. be the policy at iteration fiters
(liter) [0 the warm-up coefficient, and r(a) To (a)
*o (a) the importance weight ratio with
its clipped surrogate i(a) clip(r(a), 1, c). We define the good experiences for group sample j as 1 = 1(A, advantages. Under the assumptions that: (I) the policy change is bounded by the clipping range, and (2) the advantage estimates are unbiased, the surrogate objective improvement satisfies:
A, > 0). where A, and A, are the estimated and baseline-corrected
I (10 I (T Ba Bea
f(a) ATH (a) +7(ter)))
BjnD (T, log r(a;)) - RRmax
SIL improvement
GRPO improvement
(22)
where Rmax is the maximum per-token reward, and J denotes the surrogate objective function.
Proof 1. Consider the combined objective (Eq. 5 we can decompose the total improvement by linearity:
AJwal I [TT (TTO...)) AJGRPO (tuer)
AJGRPO
(23)
The GRPO component has a lower bound from the clipped surrogate theorem (Schulman et al. 2017b);
AJRrO Ea ((a) Are (a) R.max-
(24)
For the colf-imitation term, under the assumption of small policy changes (10 - 0ell bounded), we approximate the finite difference via gradient integration:
Vomo(a,)
Jo = ESnD T
(25)
To (o(a))
24

---

Published as a conference paper at ICLR 2026
Using the mean value theorem and assuming smoothness of the objective, integrate from P to +1
T (ay)
A STL
GRPO
1/-log
13
1EjD U1 log ro,)(
(26)
(aj)
The coefficient (thee)) scales the SIL contribution gradually. Combining terms yields the final bound:
((a)) A (a)) + (liter) RyND 1 log r(a))) Rmax-
Aooall 2 Pa
(27)
Under trust region constraints, improvements in the surrogate objective I translate to improvements in expected return (Schulman al. 2017b).
The choice of median (Pso) as the baseline estimator is grounded in robust statistics and variance minimization agentic RL with heavy-tailed relurn distributions.
Claim 2.
Theorem (Robustness to Outliers). Let {R1, R2, R,, set of returns baseline buffer DR. The median Pso minimizes the expected absolute deviation and has bounded influence function, making it robust to outliers compared to the mean.
Proof 2. For any estimator b, the loss minimization objectives are:
Mean: argmin Ei= (R: - 6)² = b b= E, b
Median: argmin Σ R4 - bl = = (R)
The influence functions characterize robustness Huber 2011
Mean: IF[R; mean) = R - R[] (unbounded)
Median: IF(R; median) sgn Pso) 2f(35))
(bounded when f(Poo) > 0)
Thus, the median is robust to outliers while the mean is sensitive. This property extends to advantage estimation since advantages are linear functions of returns.
Claim 3. The Pso achieves a balance between robustness and informativeness. Compuratively, the P25 and P75 percentiles are either overly conservative aggressive during advantage-based replay filtering.
Theorem 3 (Minimax Risk). For the class P of symmetric unimodal distributions, the median minimizes the minimax risk for absolute error loss among -qquvvvaannn estimators:
inf sup E11 u(p)l] pEP
sup
Poo p(p)
(28)
à PEP
Proof 3. This is standard result in robust statistics (Law 1986 Huber 2011) For symmetric uni- modal distributions, the median is minimax for absolute deviation loss among cewolaatooneqiivaiian estimators.
The dual filtering mechanism using both historical advantage Å, and recolibrated ad- vantage Ag ensures robust policy updates and leads to better convergence properties.
Claim 4.
Theorem (Dual Filtering). The combined condition A, > 0 and A, > 0 in the SIL objective reduces the variance of gradient estimates and promotes stable policy improvement.
25

---

Published as a conference paper at ICLR 2026
Proof 4. The dual filtering mechanism provides two benefits:
1. Variance Reduction: By filtering trajectories that were both historically good (A, 0) and re- main valuable under the current policy (A, >> 0), focus on a higher-quality subset of experiences. This reduces the effective sample size but increases the signal-to-noise ratio, potentially lowering gradient variance.
The exponential decay in the probability of reusing old trajectories (Eq. 39) prevents over-reliance on outdated experiences. Under appropriate importance weighting and assuming the advantages are estimated correctly, the policy improvement follows the standard off- off-policy policy gradient theorem (Degris et al., 2012).
2. Stability:
The combined filtering ensures that policy updates are based on relevant, high-quality experiences, promoting monotonic improvement under trust region constraints.
A. DETAILED DATASETS AND ENVIRONMENTS
The following contents correspond to Section 5.1 in the main text.
ALFWorld is an interactive environment created to evaluate how well LLM agents can handle multi- step decision-making tasks. In each scenario, the agent is given a textual goal and must achieve it by engaging in multiple rounds of interaction with the environment. The platform offers 4,639 task examples spanning six typical household activity categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2).
WebShop, on the other hand, is a sophisticated web-based platform aimed at assessing LLM agents in authentic online shopping situations. Agents are required to interact with a simulated HTML. shopping site to search for products, browse items, and purchase an appropriate product. WebShop supports a broad and varied action space, featuring more than 1.1 million products and 12K user instructions.
DAPO-MATH-I7K is rigorously engineered, competition-gratt benchmark designed to stress- test large-scale RL on LLM agents. The agent must develop multi-step mathematical reasoning, perform strategic tool-calling for code verification, and reflect on feedback from the sandbox before submitting its final answer. It contains 17K manually-curater prompts sourced from olympiad- problems, each transformed that every sround-truth label is an cnngerr-llmminatig symbolic- parsing noise and yielding a clean, deterministic reward signal.
For ALFWorld, report the average success rate for each subtask well as the overall results. For WebShop, we report the average score and the success rate (SR).
A.11 DETAILED BASELINES
The following contents correspond to Section 5.1 in the main text.
ALFWorld and WebShop. We compare with basclines such prompting-based method (i.e.,
direct 1/O) for the proprietary models GPT-4o (Achiam et al. (023) and Gemini (Team et al. 2023).
framework-based method such ReAct (ao et al. 2023) and Reflexion (Shinn et al.] 2023). ) RI. methods including PPO (Schulman et al. 2017b), RLOO (Kool et al. 2019 8019]/Ahma Ahmadian et al. (024))
GRPO (Shao al. 2024 Guo al. 2025). GIGPO (Feng et 2025b). and our proposed strong baseline Dr.BoT.
DAPO-MATH-I7K. We compare with baselines including domain-specific experts (e.g.. Qwen2.5-Me Math (Yang et al. 2024)), existing reasoning models (e.g.. Sky-T1 (Team] 2025a) ol (Jacch et al. 2024). J sl (Muennighoff et al. 2225)), and the tool-integrated RL counterparts (e.g., ReTool (Feng et al.
Qwen 32B (Guo al. (025), ), QwQ (Team 2025b), and
2025a), SimpleTIR (Xue al. 2025 ZeroTIR (Mai al. 2025), and AFM (Li al. 2025b)).
26

---

Published as a conference paper at ICLR 2026
A.12 IMPLEMENTATION DETAILS
The following contents correspond Section
5.1 in the main text.
For ALFWorld and WebShop, we follow (Feng et al.,
2025b) to use Qwen2.5-1.5B-Instte and 2024) as our base models. For DAPO-MATH-I7K, fol- low (Feng et al. 2025a) to use Qwen2.5-32B-Instree addition, we use the latest Qwen3- 3-----2BBB---e- (Yang et al. 2025) for generalization studies.
Owen2.5-7B-Innttun Yang
(Yang et al. 2024) for fair comparison. In
The implementation of the present study is based on VeRL (Sheng et al.
2024) and its extension
VeRL-Agent (Feng al. 2025b). We use the vLLM (Kwon al. 2023 as the inference engine during online rollout generation.
Table 7: Descriptions of the hyper-parameters for training and inference.
Config
Explanation
vriinbbtch.ssiz'
The batch size for training
val. data_
The batch size for validation The mini batch size for actor update iterations The maximum number of tokens on each GPU for training The miero hatch size each GPU for training
ppo_mini mno_mini_batch_sii batch_size
ppo max token len per gpu ppo_micno_batch size.per-gpi
The maximum number of tokens each GPU for log-probability
rogproob micro batch size www.ssteeeeeeeeeeeeeeeeeeeee
wwt.pperr.g....... gpu
The miero batch
each GPU log- nog-prbbability
use vee.llnmiicc.cm bsz
Whether use dynamic batch size load balance
ulysses sequencee wequence.parallll.cc..... parallel_size
The sequence paraliel size for training efficiency
tensor model. parallel size
The tensor parallel size of model deployment for rollout generation
temperature
The temperature for decnding in L.LM generation
top-p
The top-p for decoding in LLM generation
samples per prompt
The number of generated samples per prompt
lettnnee rate
The learning rate of the actor
max _eax_eppchs
The maximum number of epochs
num_steps
The number steps
Twammup
The number steps
Taway
The number of steps
use kl in reward
Whether to use the Kl. term in reward
cloo
The coefficient for the KL divergence term Whether use the KL loss
use kl loss
The coefficient of the KL loss (i.e., kl_loss_coef)
The maximum length of input prompt
max max_ressoose length
The maximum length of output generation
multi turn. max turns
The maximum number tool-call turns
Gp
The lower bound of the policy ratio clipping (i.e., The upper bound of the policy ratio clipping (i.e., clip ratio high)
Cab
N/p
The replay huffer size for self-imitation learning
NDR C
The baseline buffer size for storing the intra-group averuge performance
The lower bound of the value for dual-clip PPO/GRPO (i.e., clip_ratio_e)
Will
The lower bound of the covariance-baan based clipping
Wib
The upper bound of the covariance- The ratio of the covariance based clipping
clipping
colluut_ilttertype
The type of filtering based on intra-group variance
rollout.filer.rate loss agg mode
The ratio of filtered group
The aggregation technique for loss
norm_ady.by std. inddinnggmm grpo training strategy
Whether drop the advantage normalization
The strategy of training (e.g., FSDP, megatron)
A.12.1 PARAMEERRS
We present the details of the hyper-parameter settings in the present study. Table 7 provides the definitions of the sapee--rrrmeters used in the present study. We follow (Sheng et al. 2024) to keep most of the default empirical settings unchanged for comparability. For the covariance based clipping, we follow (Cui et al. 2025b) to set the clipping bounds wh. Wab respectively as the mean value of the top 0.02% and top 2% covariance. is noted that the token-level covariance differs from
27

---

Published as a conference paper at ICLR 2026
Table 8: Hyper-parameters of ALFWorld, WebShop, DAPO-MATH, Sokoban, and SearchR
Config
ALFWorld WebShop DAPO-MATH Sokoban
SearchR1
train batch
32
32
128
32
128
data size
ppo batch size
1024
256
32
64
ppo_max_token_leee
18432
ppo upo_micro_bater ptt.ssiiieeeeegggppe
batcoo_batc_siieeee. -pr
8
4
8
log log-prob.man ma..bobbaxx..om booo_max__oobl www.eo.eeeeenn..oee wew.:ee-epppeeeeppp gpu
73728
73728
log rog_prob_me prt.probccooo._.. wwcmmmccoobaaa wwwccooaatchhtieeeeeeeee wwwpo/attcctctpprrpeoeoooo ppp
8
4
8
use.dyuamic_bss
False
False
True
False
True
daysses_sequene sequence.parallll.siz spar
8
8
tensor model parallel size
2
2
4
2
temperature
0.4
0.4
1.0
0.4
1.0
top-p
1
1
0.6
1
0.6
nasamples per prompt learning
8
8
16
8
16
lc-6
epachs
200
350
200
num_steps
300
300
Toumuu
100
200
100
Taeyy
use iiserrarr
False
cloo
kllsss
False
3
max pwwppt..eenntttt
2048
4096
2048
1024
2048
max_response.leut length
512
1024
16384/30000
1024
30000
multi turn max turns
50
15
8/15
15
32
0.2
0.28
Ny
2048
Nrg C
10240
10
web
2
2
2
wab
60
60
60
0.02
rollout filter type
std.
rollout rollout,filter filter ratio
0.75
loss_agg mode
norm adv by std in. grpo
E False
training strategy
FSDP
28

---

Published as a conference paper at ICLR 2026
task to task. Therefore, perform statistics analysis on the covariance between action probability and the advantage with the initial model at the first training step to determine the clipping bounds.
All the settings of their values can be found in Table 8 Without loss of generalizability, do not perform meticulous fine-tuning of the arper-parameter.. One would expect better performance with grid search for the optimal rrper-parameters.
A.12.2 COMPUTING RESOURCES
All experiments performed on workstations with 380 CPU cores, 2.2TB memory, and 8 GPUs of 96GB memory. For both 1.5B/7B LLMs and 3B VLMs, the training is performed on four worksta- tions with 32 GPUs in total. For the 32B models, the training is performed sixteen workstations with 128 GPUs in total.
For ALFWorld, Webshop, and Sokoban, it takes less than 60 hours for optimization of 1.5B and 7B models. While for the DAPO-MATH-IT7 17K, it takes around a week for training the 32B models.
A.13 DISCUSSIONS ON HYPER-PARAMETT
The following contents are mentioned Section 5.6 in the main text.
(a) Replay Buffer Size Np.
(b) Baseline Buffer Size NDR.
(c) Clipping Ratio X.
012
T
Warm-up Steps 1 T'ummup.
Decay Steps Taceay.
Figure 7: Effect of hlyee-parametern of Dr.BoT (GRPO) with SPEAR on WebShop (Qwen2.5-7B- Instruct).
A.13.1 EFFECT HYPER-PARAMETERS
We investigate the following key hyper- parameerrssstrrs (see Figure 7 of Dr.BoT (GRPO) with SPEAR on WebShop Doven2.5-7B-Instrene while keeping the value of others fixed (see Table g
Replay Buffer Size ND. As the buffer size increases, the performance first improves due to the improved diversity and impact of the collected trajectories in the buffer. However, when the buffer continues to expand, trajectories in the buffer might come from earlier training batches and thereafter causes more severe degree of off-policy. The self-imitation of excessively outdated experiences becomes detrimental to the update of current policy. In addition, the large replay buffer takes more iterations to refill and thereafter the policy update frequency from self-imitation is lower than that of smaller buffer, further diminishing its intervention in agent exploration.
29

---

Published as a conference paper at ICLR 2026
Baseline Buffer Size ND When ND, = 0, the original advantages used without recalibra- tion and filtering (see Equation It shows that the direct imitation these experiences can be suboptimal where certain trajectories are outdated for the current policy. By timely adjusting the advantages and removing inappropriate experiences (A, 0), reduce the inaccurate estimation for off- off-policy update. It is noted that using advantage rather than reward in the baseline buffer helps mitigate learning bias, it allows for contributions from samples with negative rewards as long as there is variance within group. The removal of the standard deviation of outcome rewards is crucial for reducing difficulty bias. Furthermore, our double-positive advantage gate for replay filtering is essential for off off-po learning. We also find that ND,, should not be set too large such 50-th percentile reward deviates from the latest ones, decreasing the effectiveness of recalibration.
Covariance-based Clipping Ratio X. The clipping ratio can be viewed as the degree of regulariza- tion for policy entropy, where larger ratio causes more tokens be ignored during policy update. In this case, the contribution of self-imitation gets weakened. A modest range of clipping ratio (c.g., 0.0002 0.02) not only suffices the entropy management but also allows proper exploitation of the collected experiences.
Warm-up Step T warm-up- A smaller warm-up step implies earlier self-imitation of the premature, suboptimal experiences during RL. Especially when the distribution of the task and environment differs greatly from the pre-trained knowledge, the overfitting of the initial trajectories hinders ex- ploration of low-pre ww-probbble solutions and leads to action-level local optima. Intuitively, Twamnuu can be first set the same as the total number of training steps and then adjusted according to the task and the model for the improved performance.
Decay Step Tecay. A smaller decay step reduces the stimulation from the intrinsic reward for acquisition of tool-use skills. If the LLM already excels at interacting with the environment (e.g., use of tools and comprehension of observations), Taecay can be set close encouraged as the interference with the outcome reward causes inconsistent policy optimization for convergence.
large Taecay is not
A. 13.2 GUIDELINES ON HYPER-PARAMETE PARAMETERS TUNING
In this section, we provide guidelines on the choice of these It is noted that most of the hyper-parameter share the same value settings across benchmarks of various domains and tasks. One would expect performance gains without meticulous fine-tuning.
for practical usage.
Replay Buffer Size ND. It should not be set too large to avoid severe off-policy deviation. A modest size of 2K 4K proportional to the training batch size of 128 and group size of 16 (128 16) is expected to work well for frequent refilling and policy update. In other word, Np cam be set as 2x/4x of train batch size x samples per prompt.
Baseline Buffer Size ND An appropriate setting between 2K and 10K prevents outdated and untimely estimation of current policy baseline performance. other word, ND, can be set as Ix/4x of Np.
Covariance-based Clipping Ratio X. The percentage of clipped tokens should be controlled be- tween 0.02% and 2%. A smaller percentage would reduce the effect of anti-overlitting while larger percentage slows down the policy exploitation of experiences.
Warm-up Step warm-upp- The self- self-imitation should be scheduled to reach its maximum after 200 steps. For difficult tasks, should be increased to allow exploration of diverse trajectories without convergence to local sub-optimum. One could first try T Tarm-up num_steps.
Decay Step Tdecay. A decay step between 100 and 200 would be sufficient. If the tool is hard to master (c.g., complex slot-filling), the decay step should be increased to allow more stimulation of tool-calling behaviors. One could first try Taecay num steps.
30

---

Published as a conference paper at ICLR 2026
14 QUALITATIVE ANALYSIS
The following contents are mentioned in Section 5.6 in the main text.
goonetric o
probl en-sol
vingg
Voriiootion verification
computation
solution
finding
g
compuatiion
calculation computation
consutaciin
colulieg yerk
ca
solution finding
c0 Lotton
pro bobbleenn wwwolem-oooe SolvinSolnng
geome Crin pro bl -lm--slll ViLLg
Problem-solvy Callae
(a) Before RL training.
(b) After RL training.
Task Input (AIME24)
thik aeturfy fellowrs yote mattimn:
Viile-- wtw.-ottttttl ln scturighs Voen-3
Thes the value Staet
sigiis froco (es vere ind vatiially priee pati liee Emagees Find levil anaerr feros mut Voodd foal antw gia Bon.
Outpu (Before
Outpus (Afer RL)
wilee this uilhow using our mericcs live seletitenten diternet withes
Let's momastnng each saartin teme
ieler the
C- Vaaccal
Slve moual
Curt Mfrec (ythinige))) Viee 1212
3. kidtig Caft Vieeo Vi
ae coe stlve system Straer equitione to Piatt Voko Va0l, and No00, Lat's and three
is deread reckod fire strie
Varu
MMee crmee foctora other the
di De 10 1i a Meeeet) 10
Therstore
But crick
didhtt
The right coputed
entare
aoaattnn actil fiom soliv than progremntion aoperctiattyy
Them "oode pwwrrreeeeent "crpuerra": Coodee" "Troe froction topert Troesteteirnne frections
be ua TiidOoe Deve oete Sehmooate 3 Sen 030 poeertee beeg 20 (wwallgey
<hool reppompi: 16 be..
(c) The evolution of efficient coding from the purpose of computation verification (best viewed magnified).
Figure 8: Development of the agent's coding skills.
A.14.1 TOOL-INTEGRATED REASONING
Skill Development. We follow Feng et al. 2025a to analyze the coding capabilities of the agent before and after RL by classifying the purpose of the code snippets. Specifically, we em- ploy Hunyuan-Large (Sun et al., 2024) to interpret reasoning contexts before each tool-calling and judge the intention of the codes passed into the code interpreter on DAPO-MATH-I7KK dataset. The external LLM first performs intent classification with open-ended categories in free-form manner. Then, we manually deduplicate these categories and only keep the top 20 frequent ones: calculation, computation, solution finding, wroblem-solvingg geometric problem-solvine calculation, solution verification, modular calculation, probability calculation, geometric analy- sis, analysis, opfimization, intersection finding, solution search, function analysis, configuration validation, data computation, game analysis, data processing, game strategy analysis, solution ex- ploration, data analysis, list validation. We further use the LLM classify each code snippet into at most three categories. The agreement between the LLM and manual classification is above 90% on 50 randomly chosen samples. The code purposes with their frequency over twice are kept and illustrated in Figure 8 We find that the after RL, the agent becomes proficient in writing codes for solution finding, problem-solving. and solution verification, which quite advanced compared with the basic calculation and computation before RL. Our case study shows that after training, the agent learns to master matrix computation with numpy for higher efficiency, confirming the improvement of coding skills.
verification, geometric
Action Refinement. As shown in Figure 9 the agent initially aims at finding the target product that satisfies all the constraints simply by searching. However, such continuous choice of the action search is trapped by the unqualified retrieval results. The attributes of product such as color and size should be determined only at the product page. After RI., the agent jumps out of the
31

---

Published as a conference paper at ICLR 2026
Task Feppa (WeisShop)
Your took fraer aornees farh, fire kim dintt stonyy bag slene with edler wwweerceieee chiddacy and siert sel and price Tiee the
cisemottor SSard
Nen 18's kamm octton for the correnn strp.
fines reasos dect
then callaaty which obilntite lec, alcaping This tegh.
Nai aecload ploin atiiio
yourrr wiritible action air prciee
Outpat (Before RI.)
Quipur (Afler RLJ
actima de gndl of fieding
cagt tess with elart siews loog
firet in create deaa and anacio esend awryy Tike voelise ledlulig
steee,
than centrict
on the the finst mestt is in clese the mecr them sestiite redectnn med for
thaut inclule
fem,
tode
vitin
Gatar: 23-11ac
Smill
taps twos Minees
Observation
The "ooc to Serrch noarer vettrree twe the first seech oritento Oveen's come chitte parsentans,
Loaecs Lense waen viet, tewe bicosse,, dlae unddl ohriiad, soes cleee and log simee MC
Whis verticl
Observition
siace the first
extore ofcommntion
saacs, the sholld cun the ssacs them that
tnttia Citle
Uiven the sorrch ortooita Clone the ween's fap,, fors tie ueu, stoe sed stoce, shart
wwen's stans
wars, costan prtoo nomrow fhem that right march onitemo sonder stoy.
portialiy eauctin (liaa
naton oilee sue unitenia with star by clicking the furne Lsen cleck autowss
Observation
Observation
Mimaes stree sives,
action saureh' as actior that token wdis dlrddy with tritien ertnents.
slack calirry tise shall, lind ander sid
tolie citcking
becuouing the darnent dtermatiin,, stmes listed aquch critanse, Hust than the
The withow sboutd rakr selert oster first, whick the rect Teated yoe in the The
Item
Dhoper Sude Sile Shirts Site The Seatakirt Spring S0 Sumeer Shimrtoreyy pricce 35, hut
clan moy shall Pirnt colect the seriic colorr
t've diiced on ta "OALUTT Camer Taps for Mone' which danet' the lnisia he'o
Wiice the ourness page dass net corsais sucuaile tiee, thes bes oction is diice "Bcck
worth
Observation
Observation
Usei stene uall Long stwwe. in the cac slaad, saa cauii ane arscnd tren
clock action. The caree actioe la selection of the side, which loginal not Hea.
search
Stece I've medy the cator as "cl detcc cerregpode a
tecludes mcsssoy Fittons:
ferp', lergyy
Terk. Soserr
Moen berg
sholld astect the and slze woich "andlt slos, tiik cctian Unitiete the process nerreiing wornngg dow ebact come.
qur seall
sill usa the Tracnd cyour auayy] accion treut this refined saanch dary.
ol Direer' tiies shanies Meen stave sheve, ciittrk, slee sedll
Observation
Observation
acciss tover war
"eall" cio,, antich the action oction teitutes partlese
Figure 9: The advancement of strategy from the search query perfectionism to goal-oriented active progression (best viewed magnified).
32

---

Published as a conference paper at ICLR 2026
perfectionism for the search queries and tries to break the task step by step. It learns choose the action wisely for persistent focus on the task.
A.14.2 CONTEXT BUDGET
Table The number of tool call turns and response length of SPEAR on Qwen2.5-32B and Qwen3- 32B under 16K and 32K context budgets, respectively.
Model
Turns @ 16K Responses Len @ 16K
# Turns 32K
Responses Len 32K
Qwen2.5-32B
7.18
4855.48
7.13
7502.59
Qwen3-32B
3.23
10522.38
4.43
12371.95
In this section, we provide more analysis the differences of reasoning behaviors between 16K and 32K token contexts. T shows that for Qwen2.5 models, the number of tool call turns does not increase abruptly from 16K 32K. Two reasons are possible: 1) The tool call reward (Eq. [6] allows maximum of 1 score which corresponds to 10 tool calls. More tool calls (2 10) will not be rewarded. 2) The intrinsic reward design is targeted at stimulating exploration the beginning and the dominance of outcome reward is guaranteed via scheduling. The mechanical increase of tool use for reward hacking will penalized to promote reasoning accuracy. For Qwen3-32B, the number of tool calls increases a bit but still falls behind that of Qwen2.5-32B. This is because the Qwen3 series are reasoning models and tend to develop sophisticated solution patterns via pure text. In this case, the agent mainly uses the tool to double-check its previous textual reasoning and computation. The context budget from 16K to 32K allows 2K more response tokens and accordingly follows one two more rounds of tool calls for verification.
Examples on the reasoning patterns of Qwen2.5 and Qwen3 under 16K and 32K contexts are re- spectively provided in Figures TO 11 12 benchmark. It shows that for both Qwen2.5 and Qwen3 models, the number of tool calls does not increase drastically, which is consistent with the Table We believe the AIME benchmarks of reasoning-heavy tasks which challenge the agent the most its complex reasoning capabilities, In this case, our SPEAR balances the tool call frequency and the final outcome by: 1) stimulating ex- ploration at an early stage with bounded tool call reward (maximum of 1), and 2) guaranteeing dominance of the outcome reward via scheduled adjustment. Such design prevents the agent from hacking reward simply by frequent tool calling. Instead, the agent learns to reason deeply in text, and uses the tool observation as feedback to cross-validate its previous deduction and computation. The increased context budget allows longer thinking and reflection process, leading to performance gains.
We randomly choose one sample from the AIME 24
A.14.3 ADDITIONAL ENTROPY MEASUREMENTS
Figure 13 illustrates the variance of entropy of the proposed Dr.BoT with and without SPEAR. We can observe that:
1) For most tasks and model scales, the policy entropy of the vanilla Dr.oTT does not converge. This in line with our findings in Figure where the consistent uncertainty about the environments and actions causes policy entropy divergence.
2) Due to the curriculum scheduling of self-imitation, the policy entropy maintains a steady trend across stages. The SPEAR allows sufficient exploration at the beginning and gradually strengthens imitation of self-generated promising experience. Therefore, the entropy varies smoothly during training.
3) Due to the curriculum scheduling of tool call reward, the interaction with the environments is en- couraged and therefore the policy entropy of SPEAR can even surpass the baseline (e.g.. ALFWorld 1.5B and WebShop 1.5B). However, such exploration about the environment does not necessarily correlate with entropy variation. We believe the distributional gap between task domains and the pre- trained knowledge LLMs plays a critical role. For larger models (7B), its internal parameterized knowledge is richer to handle the observation states properly.
33

---

Published as a conference paper at ICLR 2026
Tonk Tepal (AIME24)
1es pectise res oaners Mut satully the Fet Fellowing lyste of mosiionn
yolrigh
Vitog.sttets 1 loee- Servige love AlC
Ouspur (Qwm22 2 80 [6X 1381 Tokens; Toolscills
Outpur (Qwen2.5 @ 32K7 1987 Vileens; Vookerals
eslue glver syttow of Sager thet: eeant-on cast ctart by conwrrting eath
equations tevblting tegeriches, atare by coneerting the legaritunic equattore tato esperaniiat fome ftree reertee each aquettor:
To selve goven
(llg Stered//addollolloorooo
Shng./iletrr/feeteelll vyclviintt - (2
Dilgg- (reddy batl lingett (ra
Clog 20terng/raad bolliigg (11
4 revrice acch eqaction ce senas of exporenta:
(traadd - 22/244
Let's oppeas ext eputtin none conopadiie form:
let's eqeess exch mastice in toms of Oe, vi and DO1
Q+ 200/2) vake vi
Oy 2401/) Yake xt3
vodet 2-21/553
Dr sy soder 2401/40)
howeery bris eight get corolicated diperatoonlyy by. Alzeresteelyy one take the
solve
(t).
Nest, nasetturee theee ceantion each wther natue Cs, 31) (O).
Alteratively
directly, Lit's sake original legastbeic craatiess and express thee tome
tet's set (o Slag2 O. Dx Slep22 1), and (c "logu? 1). Thes er can resrite the
Chiogt 0l y2,
original
Gremn:
(lag 2eg.22 y stag.3 x Mrad ahich becones Co- c- MreccS
Ciing www.ett bethigggt -Mreel21233>
Cliag2 Megg2 - 2lag.2 a MiactaD01) whtch beconer Di a- MrrcDD
Ding to berviggt) - -thacD/DS)
Cling esterie bom.emmmee -vreo(i46))
"logut (99)
VfrecD)
(4)
Dilgg Moyud Maaud a MiadDD
hove the follomingg system Ltrear spotiors:
Let's sentt Moggd co Vlouu to <Segg2 Then, reirtee
system epattion tarm (00, (b), and (c):
00.
(O. mollazll
symyy inport
face a 1/31/e23 a sapsevaritton solo odl eid), (s,
-i aaa0544046666 b. 4.37500000MM000 -4 4488602200077 cttall repposte
Co Vrrec3003)
ca- -20re07724 Yaud -Mroot2303 Seaad a- (1rrett/023 inserties ligarithe::
Cieetthiing entiiiget))
Nr solve this easter of sputiorss 1o find (O), ()), and (O. Then, me can fire yi (n2ovight))
slop 2014 yr sr2) 40 36 +
30
("rene's seort getala, bo aoles,
Let's whstirabe the volusa (o), (80, trto this egrsastie comput the
100e teoe Sentinell) 42linasattoo 3 wits reell cesgraalt) (rtoright is (Vrec vestiment it whe of Coletti yi shore relestweyy pasttive Hers 22 stree retuttvely prime cn find 10:
foom guaved
fracciin
tool, www..repmeeett sueedls ol tissstert tivestsaant The walse of de 20
teel
(1re:/25:1)) The fractier curnect2i (21)-122-13 La cleedy tx tte stglest fore since 25 and rebativily
that
(Viradted (ri)
Therofore, the fiosl, stseer is 35.
Mlood (ot.ele
scuraay. sye slo, los Ratlond,
Ruturr Shand ltyle
sat), Do aal)) saflied Ennoes
sslution
selveccad ng. ecD) Oo, 2)
there was crrer seceuse the sie caction won't reoug tned twe oeneet. let ee ecastions step stop sgote directly to Sychon:
Green
their expoential fores
solve for (o), sod to
sestitute into the exesssies Cllefe 2005 ynd elDiiggee). rotel to syeeclaC el paittveerss, razletrad anvevva up by O. 22 so Cadd ( 20, 3) as Aatioe Quisenittte eh doasloe seD 22ersslt 16 dos sute foo wesul sind bycrralllattt
unubbrt suner sball and
25/4 orhel masoorse-iii to andis diraatert
constatent The reaute contired to be (1mrep22)0)) az da 25) and on R),
30).
This, the rinel antwer io (oosep
Figure 10: The qualitative results of Qwen2.5 reasoning between 16K and 32K contexts.
34

---

Published as a conference paper at ICLR 2026
Task Impot (AIME24)
Let
met Mott wattlyy the Pellowing systm of contron
relatioely Eetagerss
Oulput (Qwen? @ (6K) 41.00 Tokent;
-hol-calll
Outpur (Qren? G332K) Tokenns coll-calls
1845 shep ctey sstrg
The snallow fied wartiss 250 L/vildieee oil posttivn have vorry
wittill
logariches
pouitive
1/3 (ol 20 Sunale sedd to fire (Tegi= yi (1) and
these ina form
ure
First
larly for the and find trattomn
a3 they thet.
Vistlii
frections.
233 curce e2) colos 22 (01//3 2013/5)
Fire derentncte..
wilthenettre teft did
pegto/2
1/3 1/5
is
takin pugng
2/43
inplitz
parpas
maybs
find
Worler Let't
vistin, lep-ly 2
1/2
(1eg
cputioos: (arCoo)) 1/3 lopctt/Cyy1)
this
viat,
souedd
cgutiio
shird
Vislouatio
1/2 viaz.
Wisleatter (1)
1/2
atorirete veriedia.
(1)
(b) co. tinee (1)
db) -2
C-3 Tavviits//ee-d
wirdnne
see
cpation 2/22 de
(3)
viddeeiaas castice
4/12 1/4
vnweiam: te wass that
with
hean
temm
visdo
creuts it out
on
26 C-73306
but
let
take
equttions
1/2
to get
1804 wewttic
(-23304 visder 4012
(-08
ovr the cnde the
warteiten o, ant aggattions
1/2 Vniiioustees: -1/3 a
deveiiattr Decuntn 18/20 24
Figure 11: The qualitative results of Qwen3 reasoning between 16K and 32K context part).
35

---

Published as a conference paper at ICLR 2026
Yook Iepust (AIME24)
Les 100 partiee reel nanerr thi sestary the fillovin syoue ausctoon V.waa l Muen soourgght Yoee
Thes wik 1in.2/
Ext shere rebetteely prie praitive tetrerry
Forett mot Woedd fivel pors
Outpur (Quen I6K)
Tokens: Tool-calls
Output (Qwen? 3232
Tokens Tost-calls
Veldaaler oyttem sobes fanciio code thio vevelloo, computs the diselune lane The systam Nonee viidde the spuctirss be writtan aacleeeeer viniiaationll
thorefore values for correct. Viatherefere veluo of Cane ynd ynly npell -255, dbsilus vilue Ih's to som
cantiors sulnuct that for wer Friiitttion boist will serills is L/6-d tetaa wwl.taeettieeyyieeeeeee
sasts
voutd becorva.. Viveggt 3/3 seleeh: Intappataer
Lif Vives
feom
viriin cade
this vide usit
yyodue made enderleviet samy
somy
start
tneate tyntacebrvers
ustng comyyooneeete somels civaaauttten:: viseel
etxeed des beftne eget to. selt in code entry here pereess foretting trofect
impori fy. solvenne,
e' befine tood the
rold preserty.
- Jejivastution adlittoess spo Co b, 47%
imsiie
the has made too rechecting Vookkcall
saserrsttel resassevvieely asdaut You wight farge to prict the resistantireel
statownss Let
abject
410000 confieee previsus fractions water 2e
compate ustra
up coleelarrres i ente0seeeeer 30 seeees 100000
collowe
inseesl whicch 12/5 sime castts
soluttie to wiie 7704 a 2.30000. 220 375 vits
Tharefors 2515 aoob aalsiig
vivlier coeatte scchovee C3//gg 30D-30 HCC/i)yaaa
coputing
20r26 widden PC antp 1ie C200 vind to triitebal:
C300
and se
wishrocl))
dbstlite velue
inneeetter code
dhecciventtn ailttt
sstved vnniue na cade
replaing the Left-oard sodes and right-nord acsecvetiio (ar Wande Wieetsbbey
that unilliso, Wishermter ond
frersians
ieport
Walg (130 Wilttaals (i).
vilvee
Readtt (routt). Mocluce
soloe besttenn dodoiiiig
60. so andy
matertntted reaanee-veaaaai -250 Aboolate valle: 25/
vddvdian Co0 (vrractet)iod vorecic/) viggetl (10 wawhannn ynlooee eddepp E: Piool Bes seree 5 and 5, to 5
dosline whet 20% the chre dt 8 56, vieenna wower 16 wwwd
(c) (n) wattweeeei
Figure 12: The qualitative results of Qwen3 reasoning between 16K and 32K context (second part).
36

---

Published as a conference paper at ICLR 2026
Busiaa SPONR
Seadine
2500
SPEAR
1280
2003
100
125
150
177
200
1.0
Strps
Staps
(a) ALFWorld 1,5B Dr.BoT (GRPO).
(b) ALFWorld 1.5B Br.oTT (GiGPO).
Doceline
Jaselar
2500
2003
2000
1500
1000
1003
300
100
125
150
175
100
Stogo
Stops
(c) ALFWorld 7B Dr.ofT (GRPO).
(d) ALFWorlld 7B DrBoT (GiGPO).
1050
SPEAR
2002
1000
(c) WehShop 1.5B Dr.BoT (GRPO).
(f) WehShop 1.5B Dr.oT (GiGPO).
003
Saading
3200
STEAR
s/ SPEAR
302
2500
2000
1.50
300
1000
103
135
200
202
220
2000
Stege
Stage
(g) WebShop 7B Br.BTT (GRPO).
(h) WebShop 7B Br.BoT (GiGPO).
Figure 13: Entropy (seq mean token- sum norm) across tasks, algorithms, and model scales.
37

---

Published as a conference paper at ICLR 2026
A.15 TRAINING COST AND COMPLEXITY
The following contents are mentioned in Section 5.6 in the main text.
Table 10:
Comparison the complexity of the vanilla GRPO and the proposed SPEAR. PG, FW, BP, RB, and Adv respectively stand for the policy gradient loss computation, forward, back- propagation, replay buffer, and advantage. Out of simplicity, we the (M) denote the forward FLOPs which is positively associated with the model size and the input length. O(P) denotes the BP operations proportional to the number of LLM parameters. We use IISIL refer the equivalent number of off- off-policy update (by SIL) per on on-policy update. After filtering by A, > & A, > (Equation the number of samples in SIL is represented as K, K S ND.
Computation of GRPO (vanilla)
Additional Computation by SPEAR
Training Stage
Description
On-policy Rollout 2CTO(M)
FW & sampling w/ Toa Copy operation (negligible).
RB Update
O(GT)
On-policy PG
GTO(M)
FW w/ my (w/o KL wa,a)
On-policy BP
O(P)
BP me.
RB Filtering
O(N=)
Look-up operation (negligible).
Adv Recalibration
O(No))(((NNa
Additive operation (negligihle).
nsu KTO(M)
w/ Rt. token- wise
Replay PG
O(KT)
clip& min (negligible).
Replay BP
msm O(P)
BP w/ mo.
In Total
3GTO(M) OPP) vmm. (KTO(M) + O(P)) Dominance by FW & BP
2000
4.00
999
DrBuT (GRPO)
5AA
Dr.BoT (GRPO)
SAL
1.8
L33
1.3
(35N)
(3R20)
(a) ALFWorld.
(b) WehShop.
(c) DAPO-MATTH-17K.
Figure 14: The averaged policy training time (s) per step with and without the proposed SPEAR.
We compare the computational complexity of our SPEAR with the vanilla GRPO algorithm in Table [O, Most of the computation comes from the forward and back-propagation of the filtered samples in the replay buffer. The memory operations such as the update and filtering of the buffer are liigt and can simply ignored. Given the current experimental settings (see Table 8) observe that nsn x 0.5 for ALFWorld and WebShop, and nsı 0.33 for HAPO-MATT-177 In this case, our SPEAR additionally introduces around 10% ~ 25% computation overhead with K G. Such computation complexity is acceptable in practice the time of each training iteration is dominated by that of on-policy rollout generation.
Figure 14shows the runtime per iteration step with and without the proposed SPEAR across differ- ent tasks and model scales. the total optimization procedure (including the rollout generation, ad- vantage computation, log-probability inference, reward computation, and the actor update) is quite similar on average for ALFWorld, WebShop, and their SPEAR counterparts. For ALFWorld and WebShop, the 1.5B models exhibit larger variance than 7B models in training time. We believe such variance is associated with findings of the previous study (Havrilla et al. 2024) that the size of L.Ms matters to the exploration diversity. Smaller L.LMs are less diverse in exploring strategies due to their shallower reasoning nature, and are therefore prone to suboptimal policies with relatively increased stochasticity in training dynamics. For DAPO-MATH-I7K an increase around 5% and 26% is observed respectively on Qwen2.5 and Qwen3 models. Since the time per step is dominated by the rollout generation and actor update, we believe such increase in time is caused by the longer reasoning traces, more tool call interactions, and the additional iterations from the replay buffer. Such encouraged exploration by SPEAR is exaggerated on the reasoning model Qwen3 and leads to longer training time.
38

---

Published as a conference paper at ICLR 2026
It is noted that the proposed SPEAR does not increase GPU memory usage. The introduction of the experience replay buffer is equivalent to increasing the training batch size per step. Due to the current sequential implementation that uses gradient accumulation with fixed mini training batch size, we can achieve policy optimization on batches of any size without OOM issues.
A.16 FUTURE WORK
A.16.1 DYNAMIC SCHEDULING
In the future, one of the promising research direction is model and adjust the scheduling param- eters dynamically. It is noted that there exists no clear-cut line between exploitation and explo- ration during training (Snoek et al. 2012) Wang et al. 2018). The exploitation and exploration are intertwined and optimized together, which is often context-dependent (Bellemare et al., (016) or guided by the policy itself (Pathak et al. 2017). Therefore, the scheduling should be progressive and smooth. We believe three kinds of techniques can be utilized for guiding the exploration:
Entropy as the medium. Following ARPO (Dong et al. 20255), we could schedule the self- imitation and intrinsic reward with monitoring of the entropy itself. It is direct and intuitive, and allows flexible and frequent adjustments. However, the modeling of the relationship between policy entropy and scheduling itself is often task-dependent and parameter-involved introducing additional computation. In addition, the entropy prone to noise where outliers of certain tokens might interfere with the scheduling negatively.
Performance as the medium. One could also adjust the scheduling by the performance-related metries (Agrawal & Goyal| (012)) such the task completion rate and the number of tool-calls. The association between exploration and success rate can be utilized. Furthermore, the number of tool-calls often indicates the degree of exploration with the environment. Nevertheless, the metrics might be deceptive as an early stop of exploration stimulation could lead to suboptimal convergence.
Curiosity Self-confidence the medium. One could intensify the exploration when the policy exhibits uncertainty (Pathak et al.) 2017]] Ladosz 2 et al.) 2022) about its actions or confusion about the transition of environment states. The policy's familiarity of the environment and its action reflects the exploration status. But it often requires parameterized learning of the curiosity or confidence via quantification of the inconsistency between the expected state transition and the real one.
A.16.2 STEPWISE CREDIT ASSIGNMENT
In extremely noisy tool ecosystem, the discrimination between good and bad experience is rather challenging merely with the outcome reward (Deng et al.) 2025 Zeng et al. 2025). Under such circumstance, a process reward model (PRM) would be beneficial to provide fine- fine-grained, stepwise supervision. However, remains prohibitive to conduct manual evaluation and preference annota- tion for training online PRMs. Very recent studies highlight few potential directions:
The usage of meta-reward via L.LM-as-a-Judge Instead of training a process reward from scratch, one could directly use an off-the-shel. L.M to assess each step not from the accuracy but from the aspect of meta-rcasoning (Zhang et al. 2025b) behaviors (c.g., planning, exploration, and reflection).
The employment implicit PRMs.
One could derive an implicit PRM (Cui et al. 2025a) by reparameterization of the outcome reward as a sum of log-likelihood ratios of two LLMs over steps. Therefore, the step-wise reward can be approximated as the differences between two adjacent steps (agent actions) (Liu et al. 2025a)
The introduction of world models. The noise from real-world tool ecosystem might be inevitable and therefore it is reasonable to perform model-based sim2real RL (Moerland et al. 2023). One could prepare an internal world model to deliver reliable state transition (Gu et al. 2024) for tool- based interaction, which help the agentic LLMs develop strategies via RL. Then, the trained LLM further adapts to real environment after a few more steps of training to gain robustness against noise.
39

---

Published as a conference paper at ICLR 2026
A.17 THE USE OF LARGE LANGUAGE MODELS
In the present study, we use the LLMs for the polishing of the manuscript writing and the discussions for analysis.
40


Reviews:
[{"review_id": "P0hSHEDkp7", "reviewer": "Reviewer_uAiT", "summary": "The paper proposes SPEAR (Self-Imitation with Progressive Exploration for Agentic Reinforcement Learning), which addresses the exploration–exploitation trade-off for large language model (LLM) agents in long-horizon reinforcement learning. The method builds upon a curriculum-based self-imitation learning (SIL) framework to progressively control exploration during training.", "questions": "See the \"Weakness\".", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The direction of agentic reinforcement learning for LLMs is highly relevant and important.\n2. The paper demonstrates solid engineering effort, with comprehensive implementations and comparisons.\n3. The writing is clear and easy to follow.", "weaknesses": "1. From my perspective, the paper seems to be more of an engineering combination than a foundational innovation. The paper reads as a well-crafted integration of existing techniques, e.g., self-imitation, curriculum learning, intrinsic reward shaping, and covariance-based regularization, etc., rather than a conceptually new algorithmic contribution. While the integration is effective, it feels closer to a sophisticated *recipe* or *best-practice collection* for training LLM-based agents than a novel RL framework with theoretical depth.\n2. The paper defines exploration primarily as maintaining policy entropy to avoid collapse or divergence, and ties it closely to tool-use behavior. The notion of “skill-level exploration” is implemented through intrinsic rewards that encourage more tool calls. This design works well for tool-using environments but doesn't generalize to settings where exploration challenges arise from reasoning complexity, world modeling, creative planning, etc. As a result, SPEAR seems to address a narrow sub-problem of exploration–exploitation, rather than offering a broadly applicable framework.\n3. SPEAR’s success depends on several manually set hyperparameters, like warm-up steps, decay schedules, clipping rates, covariance thresholds, and so on. The paper states that these are empirically chosen, and its own ablation results show significant sensitivity to them. This dependence undermines reproducibility and may limit generalization across tasks or models. Without principled guidance or adaptive mechanisms, achieving the reported results could require substantial tuning effort, reducing its appeal as a “plug-and-play” solution.\n4. The paper states that the progressive entropy scheduling is one of main innovations, but all scheduling parameters (stages, thresholds, ratios) are fixed manually, with no criterion for determining when the policy is “ready to exploit.” This makes the method appear heuristic and task-specific, rather than a self-consistent or theoretically grounded algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vAPFErTREa", "reviewer": "Reviewer_K8qo", "summary": "This paper proposes SPEAR, a curriculum-based self-imitation learning recipe for training agentic LLMs that progressively balances exploration and exploitation by (i) shaping intrinsic rewards to encourage early, skill-level tool use, (ii) strengthening self-imitation later for action-level refinement, and (iii) stabilizing learning via advantage recalibration and covariance-based clipping to control entropy and policy drift. SPEAR stores only promising trajectories in a replay buffer (positive-advantage episodes), then: (1) schedules intrinsic \"tool-call\" rewards that decay over training so outcome rewards dominate; (2) recalibrates off-policy advantages using a rolling, median baseline to avoid costly recomputation; and (3) clips tokens whose log-probabilities covary strongly with advantage, preventing over-confidence and entropy collapse. Together, these yield a smooth transition from broad tool-skill exploration to focused action-selection exploitation.", "questions": "Please refer to the questions in Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper targets a concrete failure mode in agentic RL for LLMs—entropy collapse from naive self-imitation—and designs a remedy that explicitly balances exploration and exploitation. This focus is well-motivated and illustrated (Figure 3) with clear problem framing.\n\n- The method SPEAR is a cohesive recipe rather than a single trick: curriculum-style intrinsic tool-call rewards for early skill exploration, action-level self-imitation later, advantage recalibration for off-policy stability, and entropy/covariance-based clipping to prevent over-confidence and policy collapse. The components and their roles are specified precisely.\n\n- The paper identifies and mitigates a practical reward-design pitfall: competition between outcome reward and tool-call reward causing oscillations. The scheduled intrinsic reward is a simple, effective fix.\n\n- Empirical results are broad and positive across distinct action spaces (ALFWorld, WebShop, and DAPO-MATH-17K), demonstrating that the recipe transfers beyond a single domain. Metrics and evaluation protocols are clearly stated. \n\n- The authors provide a strong, modern baseline (Dr.BoT) assembled from recent \"bag-of-tricks,\" improving the fairness and relevance of comparisons; SPEAR then yields additional gains on top.", "weaknesses": "1/ The method hinges on a fuzzy notion of \"good experiences\" in stochastic, tool-heavy settings; sparse outcome rewards can make positive advantages look like noise rather than genuine policy improvement. This undermines the replay filter’s reliability. Would step-wise process rewards be mandatory in noisy tool ecosystems?\n\n2/ Entropy control depends on hand-designed schedules and covariance-based clipping, whose optimal settings likely vary by task. This rigidity risks under-/over-regularization. Can the paper provide principled guidelines or adaptive rules beyond fixed schedules?\n\n3/ The advantage recalibration and replay-filter condition $\\hat{A} > 0, \\tilde{A} > 0$ may bias learning toward \"easier/high-return\" slices of the data, potentially narrowing exploration and reducing robustness to hard cases. How sensitive are results to relaxing this double-positive gate?\n\n4/ Parts of the qualitative analysis rely on an external LLM to label code intent, which could introduce evaluation bias. Is there an annotator-agreement check or a non-LLM rubric to validate these labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d5q77oIaAV", "reviewer": "Reviewer_Zk4g", "summary": "The paper tackles the exploration-exploitation trade-off in reinforcement learning (RL) for large language model (LLM) agents on long-horizon, sparsely-rewarded tasks. It proposes SPEAR, a curriculum-based self-imitation learning (SIL) method with intrinsic reward shaping and regularization, to progress policy entropy smoothly. SPEAR achieves performance improvements over baselines like GRPO, GiGPO, and Dr.BoT on ALFWorld, WebShop, and AIME24/25 benchmarks, with manageable computational costs.", "questions": "1.\tHow does SPEAR’s curriculum schedule adapt to different LLM sizes and task complexities? For instance, does a larger LLM need a different entropy progression rate, and does SPEAR adjust this dynamically (Sec. 4.1; Fig. 3)?\n2.\tIn AIME24/25 experiments, how does SPEAR balance tool-call frequency and final accuracy under context length constraints (Table 2), and are there qualitative reasoning path examples illustrating this trade-off?\n3.\tHow does the 10–25% extra theoretical complexity of SPEAR translate to actual training time and GPU memory usage across different benchmarks and LLM sizes (Sec. 1; Sec. 5.2)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Innovative Method**: Proposes SPEAR, extending vanilla SIL with curriculum scheduling, advantage recalibration, and covariance-based clipping to address entropy collapse/divergence in LLM agents (Sec. 4; Eq. (5); Sec. 1). Integrates intrinsic reward shaping with a curriculum, distinguishing skill-level and action-level exploration tailored to LLM dynamics (Sec. 4.2; Fig. 4). Introduces Dr.BoT, a strong baseline combining RL tricks, enabling robust comparison of SPEAR’s performance (Sec. 4.3; Table 1, Table 2).\n\n**Curriculum Design**: The integration of curriculum-based SIL with intrinsic reward shaping enables progressive transition from skill-level to action-level exploration, addressing entropy collapse (Sec. 4 and Fig. 2). Advantage recalibration using a FIFO buffer and 50th percentile baseline mitigates off-policy estimation drift and aligns with policy improvement (Eq. 2 and Sec. 4.1). Reward composition with curriculum scheduling (μ) for tool-call reward prevents competition with outcome reward (Eq. 6 and Fig. 4).\n\n**Extensive Experimental Validation**: Evaluates on diverse benchmarks: ALFWorld (multi-turn), WebShop (e-commerce), AIME (math with code), covering different action spaces (Sec. 5.1; Table 1, Table 2). Compares against multiple baselines (GRPO, GiGPO, etc.), showing consistent gains (Table 1, Table 2). Uses ablation-like figures (Fig. 3, Fig. 4) to demonstrate self-imitation and intrinsic reward’s impact on entropy and performance.", "weaknesses": "**Mathematical Formulation Ambiguity**: Equation (2) for advantage recalibration uses the 50th percentile without theoretical justification; it is empirically described as \"conservative but robust\" but lacks derivation or comparison to other percentiles (Sec. 4.1). The SIL objective in Eq. (3) and (4) introduces multiple conditions, but the interaction between these advantages and their combined effect on policy convergence is not analytically explained.\n\n**Limited Generalization**: Experiments focus on specific tasks (ALFWorld, WebShop, AIME) and models (Qwen series), with no main-text results on broader domains like vision-language agents, though mentioned in Appendix A.17 (Sec. 5.3). The method assumes that successful trajectories are clearly identifiable via positive advantages, but in stochastic environments with sparse rewards, this may lead to noisy experience selection, as acknowledged in Sec. 6.\n\n**Vague Definition of Good Experiences**: The paper notes the vague definition of good experiences in highly complex, stochastic environments with unreliable tools, limiting generalizability (Sec. 6). No concrete examples or analyses in evaluated benchmarks illustrate this vagueness’s impact or occurrence frequency. The proposed stepwise supervision solution is mentioned but not elaborated or evaluated (Sec. 6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0C6ERowJs", "reviewer": "Reviewer_pTe3", "summary": "This paper aims to address the exploration-exploitation tradeoff for agentic\nLLMs that use tools to complete certain tasks. The paper proposes a method\ncalled SPEAR that enhances existing RL methods for finetuning LLMs. The main\nfunction of SPEAR is to control the entropy of the policy to be within a narrow\nrange. It does this with methods such as intrinsic reward shaping and\nself-imitation learning. Results on various benchmarks demonstrate that SPEAR\nimproves performance (success rates) when integrated with various contemporary\nRL algorithms.", "questions": "The following are my additional comments and questions on the manuscript.\n\n1. The axis labels in all the figures tend to be quite small and hard to read.\n   In particular, the plots in Figure 1 are illegible.\n1. The main performance metric seems to be the success rates on the tasks; these\n   success rates are treated as validating the changes introduced by SPEAR. Are\n   there any measurements showing that SPEAR controls the entropy across a wide\n   variety of tasks? Figure 3 seems to only show SPEAR's capabilities with\n   respect to entropy in one task.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper seems to achieve high performance on the benchmarks presented.\n1. My understanding is that the SPEAR method is achieving what it claims to,\n   i.e., it is managing the entropy. This understanding is based on results like\n   Figure 3.\n1. The paper provides good low-level details of how exactly the method works.", "weaknesses": "As this paper is outside my field, I am unable to comment on the novelty of the\nmethod or the validity of the low-level details, so my comments mainly focus on\nthe presentation of the paper:\n\n1. While this paper excels at providing low-level details, I find that too much\n   time is spent on these low-level details. As such, I overall had a hard time\n   reading this paper and understanding the insight that it provides. For\n   example, the abstract and introduction both dive quite deep into the\n   low-level details of SPEAR, which causes them to be far too long. Meanwhile,\n   in Section 4.1 and 4.2, it is unclear what the purpose is for each of these\n   components, as the low-level details are just presented immediately.\n1. Overall, the paper seems to be written for a highly technical audience who is\n   already familiar with RL finetuning for LLMs. I think revising the paper to\n   be written for a slightly broader audience would make it much easier to\n   understand. For example, the abstract and introduction should seek to explain\n   the high-level insights of the paper, rather than directly describing the\n   modifications introduced by SPEAR.\n1. Writing the paper for a high-level audience may help in shortening it. One\n   aspect that seriously hurts this paper's readability is that important\n   sections like Related Work and Discussion are left to the appendix. If all\n   the sections truly need to be as long as they are, this paper may be better\n   fit for a journal. That being said, I think this paper can be shortened a\n   considerable amount by optimizing the usage of space. For example, there are\n   many bulleted lists in the paper that could be switched to paragraphs.\n\nI think a good starting point for addressing my feedback is: What is the key\ninsight of this paper? That is, if you had to summarize the contributions of\nthis paper in one sentence, what would you say?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)