Agentic Reinforced Policy Optimization
Authors: Guanting Dong
Hangyu Mao
Kai Ma
Licheng Bao
Yifei Chen
Zhongyuan Wang
Zhongxia Chen
Jiazhen Du
Huiyang Wang
Fuzheng Zhang
Guorui Zhou
Yutao Zhu
Ji-Rong Wen
Zhicheng Dou
Venue: ICLR 2026 Poster (2026)
Keywords: Agentic Reinforcement Learning
Large Language Model
Agentic Reasoning
Tool-use Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: ARPO proposes an entropy-based adaptive rollout mechanism to adaptively manage branch sampling during high-entropy tool-use steps and employs advantage attribution estimation to assist the LLM in understanding step-level tool-use behaviors.

Abstract: Large-scale reinforcement learning with verifiable rewards (RLVR) has proven effective in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks. In realistic reasoning scenarios, LLMs often rely on external tools to assist in task-solving processes. However, current RL algorithms typically employ trajectory-level rollout sampling, consistently neglecting the fine-grained exploration of multi-turn tool-call steps. To bridge this gap, we propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents. Our preliminary experiments reveal that LLMs frequently exhibit increased uncertainty after tool-call steps, as evidenced by higher entropy in the distribution of generated tokens. Motivated by this, ARPO incorporates an entropy-based adaptive rollout mechanism, encouraging the policy model to adaptively branch sampling during high-entropy tool-call rounds, thereby promoting step-level exploration of latent tool-use behaviors. By integrating an advantage attribution estimation, ARPO enables LLMs to internalize advantage differences in stepwise tool-use interactions. Experiments across 13 challenging benchmarks demonstrate ARPO's superiority over trajectory-level RL algorithms. Remarkably, ARPO achieves improved performance using only half of the tool-use budget required by existing methods, offering a scalable solution for aligning LLM-based agents with real-time dynamic environments. Our codes are released at https://github.com/RUC-NLPIR/ARPO.

Published as a conference paper at ICLR 2026
AGENTIC REINFORCED POLICY OPTIMIZATION
Guanting Dong' Hangyu Mao?*, Kai Ma², Licheng Bao Yifei Chen', Zhongyuan Wang" Zhongxia Chen Jiazhen Du Huiyang Wang', Fuzheng Zhang", Guorui Zhou Yutao Zhu', Ji-Rong Wen', Zhicheng Dou
'Renmin University of China, Institute of Microelectronics, Chinese Academy of Sciences "Kuaishou Technology (dongguanting, dou)âruc. edwic.eddd.ccc machangyuelme.acc.m
ABSTRACT
Large-scale reinforcement learning with verifiable rewards (RLVR) has proven effective in harnessing the potential of large language models (LLMs) for single- turn reasoning tasks. In realistic reasoning scenarios, LLMs often rely on external tools to assist in task-solving processes. However, current RL algorithms typically employ trajectory-Ievel rollout sampling, consistently neglecting the fine-grained exploration of multi-turn tool-call steps. To bridge this gap, we propose Agentic Reinforced Policy Optimization (ARPO), novel agentic RL algorithm tailored for training multi-turn L.LSs frequently exhibit increased uncertainty after tool-call steps, as evidenced by higher entropy in the distribution of generated tokens. Motivated by this, ARPO incorporates an entropy-based adaptive rollout mechanism, encouraging the policy model to adaptively branch sampling during high-entropy tool-call rounds, thereby promoting step-level exploration of latent tool-use behaviors. By integrating an advantage attribution estimation, ARPO enables LLMs to internalize advantage differences stepwise tool-use interactions. Experiments across 13 challenging benchmarks demonstrate ARPO's superiority over trajectory-level RL algorithms. Remarkably, ARPO achieves improved performance using only half of the tool-use budget required by existing methods, offering a scalable solution for aligning LLM-based agents with real-time dynamic environments. Our codes are released at https: / thub. cwm/RUC- CUUICCNNIIRAARRAAA PIR/APPOO.
agents. Our preliminary experiments reveal that
INTRODUCTION
Recently, large-scale Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated strong potential in unlcashing the capabilities of frontier large language models (LLMs), showcasing impressive performance across various single-turn reasoning tasks (OpenAI, 2024; DeepSeek-AI et al., 2025; Team et al., 2025b; Qwen et al., 2024; Yang et al., 2025). However, in open-ended reasoning scenarios (Putta et al., 2024; Shridhar et al., 2020; Qin et al., 2024), LLMs should not only cultivate long-horizon planning and adaptive decision-making skills, but also engage in dynamic, multi-turn interactions with external tool environments. To address these challenges, Agentic Reinforcement Learning (Agentic RL) (Singh et al., 2025b; Zhang et al., 2025; Team et al., 2025a) enables LLMs to autonomously interact with external tool environments during RL training, shifting the training paradigm from static task solving to the landscape of dynamic agent-environment interactions (Silver al., 2017; Wen et al., 2024; Qian et al., 2025a).
Current agentic RL algorithms typically perform trajectory-Ievel exploration during the rollout phase (Shao et al., 2024; et al., 2025), sampling complete tool-use trajectories with predefined special tokens and assigning rewards based solely on the final output. To further address tool overuse and sparse reward issues (Qian et al., 2025b), recent studies have proposed more refined reward functions to better align tool-use behavior (Wang et al., 2025a; Bai et al., 2025). Despite some
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2026
Token Entropy Visualization GAIA
Performance Despicarch Benehmarks
Deyliss Brala
Devrt lak damn
20i Devtltie Hie <2P1 mo gwent like (iNi
Efficiency Analysi Stage
Teckcal
Teakkall Sup
Teehcak
Sup
Skgge
How fods
Aool-calling result
dhughhs this
Environnent
Teal
Figure I: Overview of tool-use token entropy exploration and ARPO algorithm performance. Left: High entropy observed in the LLM following tool usage. Right: LLM performance comparison on deep search tasks using only Ik RL samples, along with comparison of training tool-use budgets.
progress, these optimizations often overlook a key aspect of training LLM-based agents: the multi- turn interaction loops between the LLM and the tool environment (Wang al., 2025d; Xue et al., 2025; Jiang et al., 2025; Feng et al., 2025b). Unlike the single-turn reasoning paradigm, the multi-turn tool-use loop offers LLMs informative, real-time feedback in real-time. This highlights the necessity of discovering effective stepwise tool-use behaviors.
To gain an insight into such step-level behaviors, we draw inspiration from series of entropy-basee studies (Wang et al., 2025b;o; Zheng et al., 2025b) and quantify the token entropy distribution of LLM- based agents during deep search tasks. As illustrated in Figure (left), the initial tokens generated after each round of tool-call feedback consistently exhibit high entropy. This indicates that external tool calls significantly increase uncertainty during LLM reasoning, revealing latent tool-use behaviors that remain underexplored (Ruan et al., 2023; Li et al., 2025h; Chen et al., 2025c). Unfortunately, current trajectory-level RL methods overemphasize complete rollout sampling comparisons, neglecting the exploration of fine-gra fine-grained behavior in high-entropy tool-use steps (Xiong et al., 2024; Yu et al., 2024). This oversight limits the diversity and scope necessary for aligning better tool-use behaviors. Consequently, it is essential develop agentic RL algorithm that aligns with agent-environment interaction characteristics to fully realize the potential of LLLM-bseed agents.
To bridge this gap, we propose Agentic Reinforced Policy Optimization (ARPO), agentic RL algorithm tailored for training a multi-turn LLM-based agent. The core principle of ARPO is to encourage the policy model adaptively branch sampling during high-entropy tool-call rounds, thereby efficiently aligning step-level tool-use behaviors. In detail, we propose an entropy-based adaptive rollout mechanism that integrates both global and partial sampling perspectives. In the rollout phase, the LLM initially performs multiple global samplings, recording the initial entropy distribution of each sample. After each tool-calling, we further monitor the real-time token entropy variation, regarding them as branching criteria. If the entropy variation exceeds a predefined threshold, the model triggers additional partial sampling to explore alternative tool-integrated reasoning paths. This design allows ARPO to effectively expand the original sampling space while balancing global and step-level tool-use behavior learning.
To fully exploit the benefits of adaptive sampling, introduce the advantage attribution estimation. Specifically, we explore both hard and soft advantage settings of ARPO, assigning shared advantage values to tokens along the same source reasoning path, while tokens branched paths receive distinct values. This design encourages the model to internalize stepwise differences in tool-use effectiveness.
We conduct comprehensive evaluations across 13 datasets spanning computational reasoning, knowl- edge reasoning. and deep search domains. As shown in Figure 1, ARPO consistently surpasses traditional trajectory-level RI. algorithms in agentic training. Remarkably, ARPO achieves this with
2

---

Published as a conference paper at ICLR 2026
Token Entropy Visualization HotpotQA
directo
W num
Will
Spoovide
S
first
ienntifedd
find
dlaan
Der
no
W
start
date
found
determiin e
matoomation
baseda
search
gathered
Teal milltp
Teal callS
Frequent tokens highest average entropy
Entropy
horn
boxed
DIR
lint
di
Lin
irac
HI
math
Eatorn Tusrhaatn
Poln
aan Enyython
quer
number
scarce
b
suloe
didewiel Me
(b) Frequent tokens with the lowest average entropy
Figure 2: Token entropy variations and token frequency statistics of tool-use agents.
only half the tool-call budget required by other RL methods, demonstrating compelling trade-off between accuracy and efficiency. In summary, the key contributions are follows:
We pioneeringly quantify the token entropy variation of LLM during agentic reasoning, revealing the inherent limitations of trajectory-Ievel RL algorithms for aligning LLM-based agents.
We propose the ARPO algorithm, which integrates an entropy-based adaptive rollout mechanism to balance global sampling while encouraging branch sampling during high-entropy tool-use steps. Furthermore, ARPO employs Advantage Attribution Estimation to assist the LLM in better internalizing advantage differences in stepwise tool-use behaviors.
Beyond heuristic motivation, we also theoretically demonstrate the rationale of applying the ARPO algorithm in LLM-based agent training.
ARPO significantly outperforms mainstream RL algorithms, requiring only half the tool-use training budgets, thereby offering practical insights for exploring agentic RL algorithms.
PRELIMINARY
Before introducing ARPO, we first review preliminary entropy- entrepy-based experiments on LLM reasoning with tools. We also give detailed preliminary of agentic reinforcement learning in Appendix F.I.
Token Entropy Calculation. Following recent catropy-based RL studics (Wang et al., 2025b;;; Cheng et al., 2025; Zheng et al., 2025b), compute the token-level generation entropy step f as:
H. -
Σ Pt.j Pogi Pr.i where p: = Re,, x;; Softmax
Zt
Here, V is the vocabulary size, zt BV is the pre-softmax logits, and is the decoding temperature. Note that this entropy reflects the uncertainty the token generation distribution
2) Pilot Experiment Token Entropy. To gain deeper insights into the reasoning process of LLM- based tool-use agents, we conduct a pilot study with two types of agents: one using a search engine for knowledge-intensiv tasks and another using a Python interpreter for computational tasks. In Figure 2, we measure token entropy variations throughout the reasoning process to assess uncertainty.
Our key observations (2) Entropy tends to increase during early reasoning stages, but remains lower than after receiving tool-call feedback; (3) Search feedback introduces
(1) Entropy rises sharply in the first 10-50 tokens following each tool call;
uncertainty than Python feedback.
We attribute these effects to the distributional shift between external feedback and the model's 's internal reasoning (Oh.1), which introduces uncertainty often exceeding that of the original input (Ob.2).

---

Published as a conference paper at ICLR 2026
Advantage Attribution Estimation
Entropy-based Adaptive Rollout
Rollout Module
Reward
Advantage
Policy Model
Global
Reference Model
Global
Global
Tix
Aes
10l
Group
Computation
Question
Tool Environment
Oey-1
Paying
Reward Model
Ance1
Ang
Figure 3: The overview of the ARPO algorithm.
Furthermore, search engines typically return informative textual content, whereas Python outputs consist of deterministic numbers, resulting in greater entropy fluctuations in the former case (Ob.3).
These findings highlight a limitation of trajectory-level RL methods, which focus on initial reason- ing while overlooking the uncertainty introduced by tool-call feedback. Our proposed ARPO algorithm addresses this by incorporating entropy-based exploration tailored to LLM agent training.
3) Agentic Tool Design. In this work, mainly focus optimizing the training algorithms of LLM-based tool-use agents. After a comprehensive review of agentic RL studies (Dong et al., 2025: Feng al., 2025a; Jin al., 2025a), we identify three representative tools evaluate the effectiveness of ARPO: (1) Search Engine: Retrieves relevant information by executing queries across the web. (2) Web Browser Agent: Accesses and parses relevant web links returned by the search engine, extracting and summarizing key content. (3) Code Interpreter: Automatically executes code generated by the LLM, returning execution results.
AGENTIC REINFORCE POLICY OPTIMIZATION
Overview. In this section, propose the ARPO algorithm, designed guide LLMs in exploring step-wise tool-use behaviors under entropy-based guidance, as illustrated in Figures 3 and 4:
Entropy-based Adaptive Rollout ($3.1): Inspired by the entropy variations observed in pre- liminary experiments ($2), ARPO extends the traditional rollout process by performing not only trajectory-level sampling but also branching at high-entropy tool-use steps. By striking a balance between global and partial sampling, ARPO encourages broader exploration of tool-use behaviors.
Advantage Attribution Estimation ($3.2): To better accommodate the adaptive rollout mech- anism, we propose the advantage attribution estimation, enabling the model to effectively internalize the advantage differences in stepwise tool-use behaviors.
Theoretical Analysis ($3.3): To establish the theoretical foundation, we provide formal analysis demonstrating ARPO's strong adaptability in multi-turn training scenarios for LLM-base agents.
3.1 ENTROPY-BASED ADAPTIVE ROLLOUT
Inspired by preliminary experiments ($2), we incorporate both trajectory-level sampling and entropy- based partial sampling during the rollout phase cover a more comprehensive sampling scope. The design of this mechanism involves the following four core steps:
(1) Rollout Initialization: Given a global rollout size of M, the LLM first generates N trajectories via trajectory- sampling based the input question g, while the remaining M N trajectories budgets are reserved for partial sampling. We then compute the entropy of the first tokens each trajectory using Equation 1, forming the initial entropy matrix denoted Hiniiall [
(2) Entropy Variation Monitoring: After recording the Hinitill 1xlk the model performs agentic reasoning with tools, as defined in Equation 8. To continuously monitor the entropy dynamics following each tool invocation, we allow the model to generate additional tokens after concatenating the tool-call result. For the tool-call step as H, E R xk then quantify the normalized change in entropy relative to the initial state as: A AH,- Normalize H, Hmina). Notably, the normalization means summing all the values of AH
first compute step-level entropy matrix denoted
4

---

Published as a conference paper at ICLR 2026
(a) Eatropy-based Adaptive Beaming
(b) Adaptive Rollout with Advantage Attribution Estimation
Toulsail Sepp
Shared Aévanigg Esinatiin
Individull Advuragee Estinztinn
Ancxez
Hi
Higheentopy Took-call
ARk
A
Question
A
[BracchZ) (Continue, otherwise
Figure 4: Illustration of ARPO's two core components. Left: Entropy- baarep adaptive beaming dynamically expands sampling based on token-level entropy. Right: Advantage attribution assigns shared or distinct values to tokens in inter-group reasoning paths.
and dividing by the vocab size V. A positive AH indicates an increase in uncertainty after the tool-call step k, whereas negative value reflects a reduction in uncertainty.
(3) Entropy-based Adaptive Beaming: encourage adaptive exploration along tool-use paths that exhibit beneficial entropy variations, we define the partial sampling probability at the step as:
Branch( Z),
PEPET. otherwise.
P: = a + 3 AH, Action (P:)
Continue,
Here, a denotes the base sampling probability, and 3 represents the stability entropy. As illustrated in Figure 4(a), the model uses Pt to guide its branching behavior: if P. exceeds a threshold т, it triggers Branch(2) 2) to spawn Z partial reasoning paths from the current node; otherwise, it proceeds along the existing trajectory. This adaptive mechanism directs exploration toward regions of the reasoning space where rising entropy signals greater potential for informative outcomes.
(4) Termination: The process iterates until one of the conditions satisfied: (1) if the total number of forked paths 2 reaches the partial sampling budget M .N, branching stops and sampling continues until a final answer is produced; (2) if all paths terminate before reaching M - N, we supplement additional trajectory-leve samples to satisfy condition (1).
with M - N -
By leveraging this efficient rollout mechanism, ARPO facilitates uncertainty-awant exploration, allowing LLMs to more effectively identify step-level tool-calling behavior. Meanwhile, assuming the global expansion size and the number of tokens per trajectory are n,, ARPO reduces the computational complexity of each rollout from the trajectory-level RL's (nn)) to between O( log n) and O(n?)
3.2 ADVANTAGE ATTRIBUTION ESTIMATION
Our adaptive rollout mechanism naturally produces trajectories containing both shared token segments and distinct beam paths (Figure 4), which motivates us to explore a more principled agentic RL policy update strategy. To this end, consider the following two advantage shaping settings:
(1) Hard Advantage Estimation: As shown in Figure 4(b), a straightforward approach is to explicitly distinguish the shared and individual parts of each trajectory the advantage level. thereby encouraging the model to capture step-level tool-use behaviors. Given d trajectories that share certain tokens while diverging in others, compute the advantage for the individual tokens using
the normalized reward R,: dg ri meam (R For the shared tokens, we assign the average sid( (R.().)
advantage across d trajectories that contain the shared segment: Aghared 1 d
(2) Soft Advantage Estimation: An elegant alternative to hard advantage shaping is to integrate the distinction between shared and individual token segments latently during policy update. Specifically, for the input the Group Relative Policy Optimization (GRPO) (Shao et al., 2024) enables the old
Neglecting the minor overhead from token-level entropy calculations
5

---

Published as a conference paper at ICLR 2026
policy Hol to generate response set (3/1 y2, ........ Ml and optimizes the policy by maximizing:
Iual
(0)
min (rr,(())As...
1=1
clip (T,,)(0), (,)1 3Dx (To Tra))
Notably, the GRPO objective incorporates the distinction between shared and individual tokens through importance sampling ratio (0):
(0) (y x. Me.
(1) (0), (0) # (0,, if yh, # Uj.cc (i.c., individual tokens).
if yi,cr Mj (i.e., shared tokens);
(3i 2; Me,ee))
As indicated by Equation (4), when trajectories yi and By undergo partial rollout at token t, they share the same response prefix tokens, i.e., Vi,<t yj Consequently, the shared prefix tokens in both trajectories are assigned the same importance weight rx.)
In GRPO, the mathematical interpretation is that the policy update is guided by the average advan- tage of tokens within each group, which serves the loss signal.
0.6
Reward Score
Since shared tokens have identical (0), their ad- vantage contributions are effectively aligned and closely approximate Aphured in hard estimation setting. While we retain the original GRPO loss formulation, our novel partial rollout design ex- plicitly distinguishes the update strategies between shared and individual tokens.
0.2
0.0
Qwen2.5 Qwen2.5-78. w. ARPO (Harl)
-0.2
Training
Figure 5: Comparison of different advantage estimation methods: Hard vs. Soft.
In practice, we compare reward dynamics between hard and soft advantage shaping during RL train- ing. As shown in Figure the soft setting consis- tently yields more stable rewards throughout ARPO training. Accordingly, ARPO adopts the soft setting as its default for advantage estimation 2.
Hierarchical Reward Design. The reward function serves the optimization objective, guiding the policy model's 's behavior during training. We follow Tool-Star (Dong et al., 2025), considering both correctness and format rewards, along with multi-tool collaboration reward mechanism. Notably, an additional reward F' is given when the model generates the correct answer, follows the correct tool invocation format, and uses multiple tools (i.e., overall reward R is formally defined as
<search>
and <pythons
during reasoning. The
max(Acc. rM, Ace.)
If Format Good Acc.>0; If Format Good & Acc.=0; ; Otherwise.
0.1 If Ir <search> &
<pythou>
R
(5)
Otherwise.
3.3 THEORETICAL FOUNDATION
Our approach leverages the adaptive partial rollout mechanism, which involves branching at high- entropy tool-use steps. Here, we elucidate the rationale behind this mechanism. In Figure 4, the adaptive partial rollout mechanism dynamically segments the Transformer-baut tokens < OTi, OT2, ..., (ottpptt into K segments. Each segment is defined as macro action, MA, & OT ITi, IT2, ..., [TTippud) > and MS, 22 MS,-1, M A;-I >. This segmentation allows us to derive
policy's output
OT nmmnn >. The corresponding macro states are defined as MS, &
"The theoretical relationship between hard and soft advantage estimation is formally proven in Appendix F.2
"The detailed flowehart for the ARPO algorithm can be found in Algorithm 1.

---

Published as a conference paper at ICLR 2026
the Generalized Policy Gradient (GPG) Theorem applicable to all Transformer-basee policies:
J() = -BE
Vo lolog no M (MA
(6)
where represents the macro step, and (T( denotes the advantage of trajectory т. The GPG Theorem asserts that for any differentiable Transformer-based policy Ty and any objective function J(O), optimization can be effectively conducted using macro actions (i.e., partial rollout segments). This generalization encompasses the traditional Policy Gradient Theorem (Sutton et al., 1999), VaJ(0) = Er-wr {>'1 [Vg log Togma(us (e (T))) which operates on single-token actions (where az is a single output token of the Transformer), as a specific instance of our broader GPG framework. Consequently, ARPO, as advanced implementation of the GPG Theorem, provides robust theoretical foundation. The formal proof of the GPG Theorem is presented in Appendix F.3.
4 EXPERIMENTAL SETUP
Datasets. To comprehensively assess ARPO's effectiveness in training multi-turn tool-using agents, we conduct an evaluation on the following types of long-horizon reasoning tasks: (1) Mathemat- ical Reasoning: including AIME2024, AIME2025*, MATH500 (Lightman et al., 2024), MATH (Hendrycks al., 2021), and GSM8K. (2) Knowledge-Intensive Reasoning: including HotpotQA (Yang et al., 201 8), 2WikiMulchopQAA (Ho et al., 2020), and Musique (Trivedi et al., 2022) and bamboogle (Press et al., 2023). (3) Deep Search: including General AI Assistant (GAIA) (Mialon et al., 2024), WebWalker (Wu et al., 2025a), Humanity's Last Exam (HLE) (Phan et al., 2025), and Seeeeh (Chen et al., 2025a). To ensure consistency with prior work, we adopt the Tool-Star (Dong et al., 2025) test split for math and knowledge reasoning benchmarks, and follow WebThinker and HIRA (Li et al., 2025e; Jin et al., 2025b) for the split in deep search benchmarks.
Baselines. To effectively evaluate the efficacy of ARPO, consider the following three baselines: (1) Direct Reasoning. For 10 reasoning benchmarks, we evaluate the instruction variants of the Qwen2.5 (Qwen al., 2024) and Llama3.1 (Dubey et al., 2024) series. Given the superior math capabilities of the Qwen3 series (Yang et al., 2025), we adopt it as the backbone for testing RL algorithms on the DeepSearch task. We also include strong reasoning baselines such as QwQ (Team, 2024b), DeepSeek-RI (DeepSeek-Al et al., 2025), GPT-4o (Hurst et al., 2024)and ol operviww (Hurst et al., 2024). (2) Trajectory-Level RI. Algorithms. We compare ARPO against trajectory-leve RI. algorithms, including GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), and REINFORCE++ (Hu, 2025). (3) LLM-Based Search Agents. For the DeepSearch benchmark, we further benchmark search agents, including vanilla RAG (Lewis
ARPO against GRPO and set of workflow-drive et al., 2020), Search ol (Li et al., 2025d), WebThinker (Li et al., 2025e), and ReAct (Yao et al., 2022). Detailed descriptions of all baselines provided in Appendix D.
Evaluation Metric For accuracy, Fl scores are reported on four snowledge-intensing QA tasks, while others are judged by Owen2.5-72B-insttie with temperature 0.6 and top-p 0.95. Following Li et al. (2025d), answers are extracted from \box in model outputs. Detailed implementation and training guidelines are provided in Appendix E.
under the LLM-as-Judge setup. We adopt pass @
EXPERIMENTAL RESULTS
5.1 MAIN RESULTS
Results on Mathematical & Knowledge-Intensin Reasoning. Our main results are shown in Table In a fair setting, ARPO consistently outperforms all trajectory-Ieve level RL algorithms, firmly establishing its superiority. Moreover, we highlight the following insights:
(1) Ineffectiveness of Prompting Methods:
Tool-integrated prompting (Li et al., 2025d) fails to elicit effective tool-use behaviors. For both Qwen and Ilama series models, performance gains are marginal or even inferior to direct reasoning, indicating that prompt engineering alone is insufficient and may interfere with inherent reasoning capabilities.
datti on-aime
7

---

Published as a conference paper at ICLR 2026
Table 1: Overall performances on 10 challenging reasoning tasks are presented. The top two outcomes are bolded and underlined. Dataset abbreviations are as follows: WebW (WebWalken) HQA (HotpotQA), 2Wiki. (2wikiMulliHopOA), MuSi. (MuSiQue), and Bamb (Bamboogle)
Mathematical Reasoning
Ghowledge-Intensiig Reasoning
Method
Avg.
AIME24 AIME25 MATH500 GSM8K MATH WebW HQA 2Wiki. MuSiQ. Bamb.
Mamma.1-8B-Instte
3.3
0.0
43.3
81.4
60.6
3.0
24.3
24.6
10.4
40.0
28.8
TIR Prompting
3.3
3.3
39.4
73.8
58.2
15.0
47.5
15.5
58.4
36.3
GRPO
13.3
13.3
62.4
87.4
79.2
26.5
57.8
71.8
31.0
68.2
51.1
+ Reinforce ++
13.3
16.7
61.4
87.0
77.2
27.5
57.1
71.6
29.9
69.1
51.1
DAPPO
16.7
13.3
61.2
87.4
76.4
25.5
56.6
70.3
29.2
67.3
50.4
+ARPO
23.3
16.7
64.6
88.0
80.2
30.5
65.4
75.5
34.8
73.8
55.3
Qwen2.5-7B-Instrucc
10.0
10.0
70.6
90.2
82.0
2.0
12.2
12.6
6.6
24.0
32.0
TIR Prompting
6.7
10.0
68.2
64.6
78.2
15.5
14.8
18.3
9.5
23.6
31.0
GRPO
23.3
26.7
78.0
92.8
87.8
22.0
59.0
76.1
30.6
68.4
56.5
+ Reinforce ++
26.7
23.3
78.0
92.2
88.8
26.0
55.1
68.9
25.2
64.9
54.9
DAPO
20.0
23.3
80.4
91.0
88.8
24.0
57.7
68.4
28.6
65.5
54.8
ARPO
30.0
30.0
78.8
92.2
88.8
26.0
58.8
76.1
31.1
71.5
58.3
(2) Trajectory-Level RL Constraints: Classic trajectory-level RL. algorithms do not effectively harness the potential for tool-integrated reasoning compared to ARPO. While DAPO excels in single-turn tasks, it struggles multi-turn, knowledge-intensive settings, highlighting the difficulty of inducing step-level tool-use behaviors through trajectory-Ievel optimization.
Under identical conditions, ARPO consistently outperforms baseline RL methods across 10 datasets, with an average accuracy gain of 4%. It generalizes well across both Qwen and Llama backbones, demonstrating strong adaptability and efficiency.
(3) Robust Performance of ARPO:
Results on Deep Search Tasks. To further verify the effectiveness of our ARPO in challenging deep search scenarios, we compare the performance of the Qwen3 series models trained with only 1k RL samples, against a series of strong baseline methods. Our observations are as follows:
(1) ARPO's Generalization in Deep Search: In deep search settings, even srate-ofthh-aad LLMs like GPT-4o and PrrpSeekk-R16""
perform poorly on the HLE benchmark (2.0% & 8.6%).. In contrast, ARPO achieves 10.0% and 43.2% pass@) scores on HIE and GAIA using only Qwen3-14B models. Remarkably, ARPO is trained with just 1K samples from an open-source web search dataset, demonstrating strong sample efficiency and tool-use generalization.
(2) Critical Role of Step-Level Tool Use: ARPO consistently outperforms GRPO, with 6% gain on GAIA and WebwalkerQA. This underscores the advantage ARPO's balanced sampling strategy, which integrates global and step-level exploration. This design encourages diverse tool-use behaviors, particularly in high-entropy, multi-step reasoning tasks common in deep search.
5.2 QUANTITATIVE ANALYSIS
Analyzing Sampling Scale.Due the dynamic, multi-turn nature of Deepsearch evaluation, Pass @ alone is insufficient to fully capture model's tool-use potential. To address this, we conducted extended sampling analysis using Pass and Pass@5 metrics, as shown in Figure 6. Both Qwen-8B and Qwen-14B exhibit consistent gains and clear scaling trends in Pass 3 and Pass @5 after ARPO alignment. Notably, Qwen- with ARPO achieves strong P performance 61.2% on GAIA, 24.0% on HLF,, and 59.0 % on xBench-DR. These improvements reflect ARPO's effectiveness in promoting fine-grained tool-use exploration, thereby expanding the sampling space and enhancing both inference efficiency and behavioral diversity."
Call Efficiency Analysis. agentic RL training, excessive tool calls can incur substantial computational and financial costs. Therefore, effective agentic RL algorithm must balance performance with tool-use efficiency. To evaluate ARPO's efficiency during training, we compare
SSince xBench-DR consists entirely of Chinese queries, we Chinese prompts for Pass K evaluation, resulting improved performance compared Table 2.
8

---

Published as a conference paper at ICLR 2026
Table 2: Overall performance on various deep search tasks, with accuracy results for each dataset obtained using Ilm-as-judge The best results are indicated in bold, and the second-best results are underlined. Results from larger or closed-source models are presented in gray for reference.
General Assistant
WehWalkerQA
Humanity's Last Exam XBench
Method
Iv.1
Lx:2
Lv.3
Avg.
Easy
Med.
Hard
Avg.
NS
CE
SF
Avg.
Avg.
Direct Reasoning (>=32B)
Swen3-32B-thinkime
12.1
14.9
6.9
2.9
3.1
14.6
8.4
12.6
14.0
DeepSeck-RI -1-322
21.5
13.6
0.0
14.2
7.5
1.4
4.2
3.8
5.1
6.5
6.4
10.0
QwQ-32B
30.9
6,5
5.2
18.9
7.5
2.1
4.6
4.3
11.5
7.3
5.2
9.6
10.7
GPT-44
23.1
15.4
8.3
17.5
6.7
6.0
4.2
5.5
2.7
1.2
3.2
2.6
18.0
40.5
21.2
5.2
25.2
5.0
11.8
11.3
10.0
8.5
8.1
9.3
8.6
32.7
ol ol-preview
11.9
10.4
7.9
9.9
12.9
8.1
6.6
11.1
Single-Euhaneee Method (Qwen3-8B)
Vanilla RAG
28.2
15.4
16.7
20.4
8.9
10.7
9.9
10.0
5.1
1.6
12.9
5.8
8.0
Search-ol
35.9
15.4
0.0
21.4
6.7
15.5
9.7
11.5
7.6
2.7
5.3
6.4
10.0
WebThinker
43.6
11.5
0.0
22.3
6.7
13.1
16.9
13.0
7.3
4.0
6.3
6.6
13.0
ReAct
35.9
17.3
8.3
23.3
8.9
16.7
18.3
15.5
4.2
4.6
16.0
RL-hased Methad (Qwen3-8B)
Qwen3-8B
28.1
15.4
16.7
20.4
0.0
2.4
2.8
2.0
3.9
2.7
8.4
4.6
9.0
GRRO
48.7
25.0
8.3
32.0
24.4
33.3
26.8
29.0
7.9
4.0
10.5
7.8
20.0
+ +ARPO
53.9
32.7
16.7
26.7
29.6
30.5
7.3
15.8
8.8
Single-Enhanced Method (Qwen3-14B)
Vanilla RAG
38.5
19.2
25.2
17.8
13.1
11.3
13.5
9.4
6.0
15.0
Search-ol
48.7
23.1
0.0
30.1
11.1
21.4
16.9
17.5
6.4
4.0
10.5
6.8
21.0
WebThinker
48.7
26.9
8.3
33.0
13.3
23.8
18.3
19.5
7.0
4.0
9.5
7.0
23.0
ReAct
48.7
25.0
8.3
32.0
11.1
20.2
12.7
15.5
5.8
5.3
10.5
6.6
20.0
RL-based Method (Qwen3-14B)
Qwen3-14BB
33.3
13.5
0.0
19.4
6.7
2.4
4.0
6.7
11.6
6.8
14.0
+ GRPO
51.3
34.6
0.0
36.9
28.9
33.3
26.8
30.0
7.9
6.7
12.6
8.6
27.0
+ ARPO
56.4
40.4
16.7
43.7
31.1
42.9
31.0
36.0
10.3
10.7
13.7
10.0
32.0
Passia1
Pass@i
Passaa5
Overall Performance
70
30
70
60
25
21.8
47.0
50.5
$6.5
$5.0
20
17.2
19.0
41.0
15-
31.5
20-
10
20.0
20
20
10-
10:
Queen3-Ke SB
Qwen3-144
Qwen3 8B
Queen3
Qwen3-8B
Qwen3- 14B
Qwen3 Qwen3-144 14B
(a) GAIA
(b) HLE
WebWalkerQA
xbench
Figure 6: Analysis of Qwen3-8B and Qwen3- using ARPO across Pass
Pass @5 metrics.
with GRPO on Qwen2.5-7B. As shown in Figure 7a, ARPO achieves higher overall accuracy while using only half' as many tool calls as GRPO. This efficiency stems from ARPO's ' entropy-based adaptive rollout strategy, which selectively explores alternative branches only at high-entropy decision points. This targeted exploration significantly broadens the behavioral search space while minimizing tool usage. More ablation and scaling analyses can be found in the Appendix A.2.
Rollout Sampling Diversity Analysis. To demonstrate that ARPO achieves better coverage of the rollout solution space compared to GRPO, we randomly sampled 640 problems from 10 rollout steps, collecting 7.6k trajectories. Using BGEM3 as the semantic embedding model, visualized the sampling distribution of rollouts through PCA dimensionality reduction and DBSCAN clustering.
As shown in Figure 7b, ARPO's sampling trajectories form more distinct and clearer cluster centers (54 clusters for ARPO vs. 48 for GRPO), with greater intra-cluster compactness and larger inter- cluster separation. These findings indicate that ARPO effectively exploits the transformation of high-entropy uncertainty into exploration opportunities, significantly improving rollout diversity and making the distribution of sampled paths more structured.

---

Published as a conference paper at ICLR 2026
500
Qwee22.5 GRPO
Number of Clusters: 40
Number of Clusters 54
Owees2.5-75 7-
4000
350
250
200
Step
(a) Tool- efficiency
(b) Rollout diversity
Figure 7: Comparison of GRPO and ARPO.
6 RELATED WORK
Reinforcement Learning with Verifiable Rewards. Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024; Kaufmann et al., 2025) has become a prominent RLHF paradigm for LLMs, yielding substantial gains in math and code reasoning (Shao et al., 2024; DeepSeek-Al al., 2025; Yang et al., 2025; 2024; Team, 2024b;a; Dong et al., 2024c; Qiao et al., 2024). Following OpenAl ol (OpenAI, 2024), a growing body of work seeks to reproduce and scale RLVR recipes to broader settings (DeepSeck-AL et al., 2025; Team, 2024c; Team et al.. 2025b). Recent studies improve RLVR stability through modular algorithmic and training-recipe analyses (Yu et al., 2025; Zeng et al., 2025; Hu et al., 2025; Yue et al., 2025; Feng al., 2025b; Liu al., 2025; Kool et al., 2019; Ahmadian et al., 2024; Dong et al., 2024a; Hu, 2025), and investigate what RLVR learns via critical-token and entropy-based perspectives (Lin et al., 2024; Gandhi et al., 2025; Li et al., 2025b; Vassoyan et al., 2025; Wang et al., 2025b; Cheng et al., 2025; Wang et al., 2025c). In parallel, segment-level RL objectives have been proposed to encourage exploration and improve credit assignment (Guo al., 2025; Li et al., 2025g: Zheng et al., 2025a). Despite this progress, RLVR tailored to LLM agents remains relatively underexplored; we use entropy as a lens to study RL algorithms that more effectively shape agent behaviors.
Agentic Reinforcement Learning. Reinforcement learning (RL) is increasingly used to train LLM agents for interactive, open-ended environments (Là et al., 2025; Shridhar et al., 2020; Mialon et al., 2024). Beyond classical value-based and self-play successes (Mnih et al., 2015; Silver et al., 2017; Narasimhan al., 2015; Tan al., 2024; Zhai al., 2024; Bai et al., 2024; Wang et al., 2024; Schulman et al., 2017; Peng et al., 2019), recent work optimizes agent trajectories that interleave reasoning and environment interactions (Wang et al., 2025d: Zhou et al., 2024). For tool-augmented agents, many methods rely on rule-based or verifier- with extensions to multi-tool coordination and long- ong-horizon settings (Jin et al., 2025a; Feng et al., 2025a; Song et al., 2025; Chen et al., 2025b; Li et al., 2025f; Sun et al., 2025a; Li et al., 2025e; Singh et al., 2025a; Qian et al., 2025a; Dong et al., 2025; Wang et al., 2025a; Li et al., 2025c). However, trajectory-level RL often provides overly coarse credit assignment for multi-turn tool-use decisions, motivating our step-level ARPO.
rewards to encourage correct tool use,
7 CONCLUSION
In this paper, we present Agentic Reinforced Policy Optimization (ARPO), an innovative reinforce- ment learning algorithm tailored for training multi-turn, LLM-based agents. Our experiments reveal that LLMS exhibit high token entropy after tool usage. ARPO leverages this by incorporating an entropy-based adaptive rollout mechanism, balancing global and step-level sampling to encourage diverse exploration in high-entropy tool-use phases. By integrating Advantage Attribution Estimation, ARPO enables LLMs to internalize advantage differences in stepwise tool-use interactions. Across 13 challenging benchmarks in computational reasoning, knowledge reasoning, and deep search domains, ARPO consistently outperforms traditional trajectory-level RL algorithms. Remarkably, it achieves great performance with only half the tool-use budget of other methods, offering a scalable solution for aligning M.LM-based agents with dynamic environments.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China No.62272467, China Postdoctoral Science Foundation under Grant Number 2025T180444 180440, and Outstanding Innovative Talents Cultivation Funded Programs 2026 of Renmin University of China. The work partially done the Beijing Key Laboratory of Research Large Models and Intelligent Governance.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 12248-12267 Association for Computational Linguistics, 2024. doi: (0.1633N1/224AAALLLLLOCCC https://doi.co http:///oo.or/////// 11/88863312200 wal--onng.66. 662.
URL
Fei Bai, Yingqian Min, Beichen Zhang, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Towards effective code-integrated reasoning. CoRR, abs/2505.24480 2025. doi: 1044550/ARXIV.2505 arXiv. 2505. 24480.
URL https://doi. 0rg/10.48550
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in nn-the-wild device-control agents with autonomous reinforcement learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver BC, Canada, De- cember 10 15, 2024, 2024. URL http: http::/ wwwers.ppppp.ppaaa/eeeeeeee... 10040dddbbaa91159ffeeeeeee5 ort.aaaeeeeertttoao.oeeeeeoeeeee html..
cc/ wwwpc//appppaipaeeeee www.feelleeeee/aeeeeeeaa /2224///222//hhh//
Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, al. xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations. arXiv preprint 7xXiv:2506.3665 2025a.
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang. Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for Ilms via reinforcement learning, 2025b. URL httpps://axii or org/abs/ 2503. 19470.
Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. CoRR, abs/2505.166 16782, 2025c. doi: 10.45500/AXXV.2555.66888 URL hwt.thsss//ii.oooe org/10....... 48550/arXiv. .505. 2575.16782.
Zhipeng Chen, Kun Zhou, Beichen Zhang, Zheng Gong, Xin Zhao, and Ji-Rong Wen. Chatcot: Tool-augmented chain-of-thought reasoning chat-based large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 14777-14790. Association for Computational Linguistics, 2023. doi: 10. 11863////2023.FINDDDD FINDINGS- EMNLP.985. URL https://doi. orgo 18653/v1/2023 findings-emnlp 985.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. CoRR, abs/2506. 14758, 2025. doi: 48550//r5Xii 2506. 14758.
10.48550/ARXII.2506..10 14758. URL httpps://oi.on
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:2110.14168 2021.
Tri Dao. Flashattention-22 2: Faster attention with better parallelism and work partitioning. CoRR, 2023.
11

---

Published as a conference paper at ICLR 2026
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L.. Jin. Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu. Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu. Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
URL wtt.tttps///aiiii http://aaawivv..rrb//s//sse/ss// ......19440 12948.
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving -osluetion-followingg capabilities of large language models. CORR, abs/2406. 13542, 2024a. doi: 10448550/ARXIV:2406.. 13542. URL https://doi, org/10. 48550/arXiv 2406. 13542.
Guanting Dong, Xiaoshuai Song, Yutao Zhu, Runqi Qiao, Zhicheng Dou, and Ji-Rong Wen. Toward general cnrruction-oolowinng alignment for retrieval-augmmened generation. CoRR, abs/2410.09584, 2024b. doi: 1044550/ARXIV241000988. URL https://doi org/10.44550 2410.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 177-198. Association for Computational Linguistics, 2024c. doi: orwgrog////// 10. 18653 /v1553/v1// 2024 long. 12.
URL
https://doi http:/oo..o.ggg///
Guanting Dong, Chenghao Zhang, Mengjie Deng, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. Progressive multimodal reasoning via active retrieval. arXiv preprint arXiv:2412.14883. 2024d.
Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, and Ji-Rong Wen. Understand what LLM needs: Dual preference aligmment for retrieval-augmenter generation. CoRR, abs/2406. 1 18676, 2024e. doi: 10.48550/ARXIV.206..1. 18676.. URL https://doi http:///oooooo//....////// 48550/arXiv. 2406. 18676.
Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. Tool-star: Empowering brained multi-tool reasoner via reinforcement learning. CoRR, abs/2505.16411 2025. doi: 10.48550ARRXV..500..OO 16410. URL hwtph:tp////et org/10 48558/arXiv 2505. 16410.
12

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Jiazhan Feng, Shijue Huang. Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms, 2025a. URL https: /tapp:://axiii org/ ab5/ 11536.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for LLM agent training. CoRR, abs/2505. 10978, 2025b. doi: 10. 1044500ARXXV22555.0007 URL https://doi. orgoo/gg//// 10. 48550 -8550/arXiv... arwiv. 2505. 10978.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01 307, 2025.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang, Minlic Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: http::::////peeeiieee.... ne wwtttee//fffne (etmmmmdddddpdade-e
Yiran Guo, Lijie Xu, Jie Liu. Dan Ye, and Shuang Qiu. Segment policy optimization: Effective segment-level credit assignment in rl for large language models, 2025. URL https://arxiv. org/abs/2505. 23564.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL ht hat.:/aaaaeea..aaaaaaata------- 2021
pro.eeediooceeennsee neurips. cc/paper/ Abstract- html.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Núria Bel, and Chengqing Zong (eds.). Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 6609-6625, Inter- national Committee on Computational Linguisties, 2020. doi: 10. 18653V
580. URL https: orgooogg///// / 118855331 col maalinng-n.....mm 580.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262. 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zerr:: An open source approach to scaling up reinforcement learning the base model. CORR, wtt.s/2253.2222.... 2025, doi: 10.45500ARXXI12553222200 URL //dp:://doi. 435/10.48550//000
2503. 24290.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, et al. Veritool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055: 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training llms to reason and leverage search engines with reinforcement learning. CoRR, 0ts/2503..95166 2025a. doi: 1048550/ARXIV2550.055551 URL https://doi. arg//0.48550 2w..2.255991111
Jajje Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmenter generation research. CoRR, abs/2405. 2024, doi: 13576. URL https:://////t .rgg 1a850//aXiiii 2405. 13576.
13

---

Published as a conference paper at ICLR 2026
Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yang Zhao, Hongjin Qian, and Zhicheng Dou. Decoupled planning and execution: A hierarchical reasoning framework for deep search, 2025b. URL https://arxiv. wwwp:/iwii...ia/oooaa haarrggabs//5500 02652..
Timo Kaufmmann Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey reinforcmment learning from human feedback. Trans. Mach. Learn. Res., 2025, 2025, URL https://openreviiee net/
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for freee! In Deep Reinforcement Learning Meets Structured Prediction, ICLR 2019 Workshop, New Orleans, Louisiana, United States, May 6, 2019. OpenReview.net, 2019. URL https:
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi. and Hannanch Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. CoRR, abs/2411.15124, 2024. doi: URL htttps://oii. org.i..rr//oo 10. 48550/arXiv 241.1...2111...1 1111.11124
0..4850/ARXXI.241..555
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeened generation for knowledge-intensive NLP tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020. URL https: /ttp://:////..oeengg...... neur www.neuriii.... wwwrrpssccccce cccppaper/222.. #b493230205f780e weweebcc294455 1567785551 Abstract.
Chengpeng Li, Guanting Dong, Mingfeng Xue, Ru Peng, Xiang Wang, and Dayiheng Liu. Dotamath: Decomposition of thought with code assistance and self-correction for mathematical reasoning. CoRR, hass/2407.04077 2024a. doi: 10.48500AARXY.2447.00000 URL https://don http:://ooioooooo org/10, 48550/ /at.///rii....m 20072444440 04078.
Chengpeng Li, Zhengyang Tang, Ziniu Li, Mingfeng Xue, Keqin Bao, Tian Ding, Ruoyu Sun, Benyou Wang. Xiang Wang, Junyang Lin, and Dayiheng Liu. Cort: Code-integrated reasoning within thinking. CoRR, abs/2506.008200 2025a. doi: 10.48550/ARXIV225000009 https://doi. http:://doooor////0 48550/ 2506 09820..
URL
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Llms easily learn to reason from demonstrations structure, not content, is what matters! CORR, abs/2502.07374, 2025b. doi: 10.4550/ARXIV.2502.00 07374.
URL https: ptttp:://dot //doi. org/o...///// ///. 40..48//arx...2555
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor: Navigating super-human reasoning for web agent, 2025c. URL https: //wwxva./vvv///a////a//eee/aaaa wwwpii.oorg//aae orgg/bs////// 02592.
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yongkang Wu, Zhonghua Li, Qi Ye, and Zhicheng Dou. Retrollm: Empowering large language models to retrieve fine-grained evidence within generation. CoRR, abs/2412.11919, 2024b. doi: 1048550/ARXIV2412.1000 URL https://doi. www.oo..///00/0
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang. Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. CoRR, abs/2501 05366, 2025d. doi: 1044850//RXXIV2550 105366. URL https://doli http://doooooggg//// org/10.48550 arXiv. 2501. 05366.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776 2025e.
14

---

Published as a conference paper at ICLR 2026
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated RL. CoRR, abs/2503.23383. 2025f. doi: 10.48550AARKIV2250 23383.
URL https: https:://doo..rg/// .rg/ 4805.4855arrr 2503.
Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing. Shuyue Guo, Tinyy Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, Zheng Zhang, Wei Shen, Qian Liu, Chenghua Lin, Jian Yang, Ge Zhang, and Wenhao Huang. Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic rre-based modeling, 2025g. URL https: //arxiv. org/ wwws///// 17445.
Zhicong Li, Hangyu Mao, Jiangjin Yin, Mingzhe Xing, Zhiwei Xu, Yuanxing Zhang, and Yang Xiao. Ngent: Next-generation agents must integrate multi-domain abilities achieve artificial general intelligence. arXiv preprint arXiv:2504.2 1433, 2025h.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang. Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances Ilm's reasoning capability. CoRR, abs/2411.19944 2024. doi: 10.4850/ARRXIV2411.1999 URL https://doi. org/10 /tgg/10.. 100.000.000 48550/ arviv 2411. 19943.
Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haoman Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. CORR, abs/2504.13055 2025. doi: 1044550/ARXXV.2500..0000 URL https::/doi. orgg10. arXiv. 2504. 13055.
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stanczak, Peter Shaw, Christopher J. Pal, and Siva Reddy, Agentrewardhench: Evaluating automatic evaluations of web agent trajectories. CoRR, abs/2504.08942, 2025. doi: (0.48550/ARXXV..500..050 08942. URL https: //doi. org/orgg///oo 48550/ arxiv. 0894.0
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: benchmark for general Al assistants. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openrrvviw. http//oommmidiibbvaiiieeeee
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nat., 518(7540):529-533 2015. doi: 10. 001108//AATRRE12236 URL
https://doi, www.rogg///0 1038/naturel 14236.
Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. In Lluís Märquez, Chris Callison- Jian Su, Daniele Pighin, and Yuval Marton (eds.). Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1-11. The Association for Computational Linguistics, 2015. doi: 10. 10116553/1DD115-0000.
URL htthht/ss//oii org/10.
OpenAl. Learning to reason with 1mms, September 2024. URL https://openai. com/ index/ Learning-to-reesonnw tmml
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- off-policy reinforcement learning. CoRR, abs/1910.00177,, 2019. URL
http://arxiv.cr http:/iww..rgabb///eo//oe http//bb//19 1atos.//9..00000. 00177.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arna Chopra, Adam Khoja, Ryan Kim, Jason Hausenloy, Oliver Zhang,
15

---

Published as a conference paper at ICLR 2026
Mantas Mazeika, Daron Anderson, Tung Nguyen, Mobeen Mahmood, Fiona Feng. Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Jessica P. Wang. Pawan Kumar, Oleksandr Pokutnyi, Robert Gerbicz, Serguei Popov, John-Clark Levin, Mstyslav Kazakov, Johannes Schmitt, Geoff Galgon, Alvaro Sanchez, Yongki Lee, Will Yeadon, Scott Sauers, Marc Roth, Chidozie Agu, Søren Riis, Fabian Giska, Saiteja Utpala, Zachary Giboney, Gashaw M. Goshu, Joan of Are Xavier, Sarah-Jane Crowson, Mohinder Maheshbhai Naiya, Noah Burns, Lennart Finke, Zerui Cheng, Hyunwoo Park, Francesco FournierFaacio, John Wydallis, Mark Nandor, Ankit Singh, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Darling Duclosel, Jungbae Nam, Jennifer Zampese, Ryan G. Hoerr, Aras Bacho, Gautier Abou Loume, Abdallah Galal, Hangrui Cao, Alexis C. Garretson, Damien Sileo, Qiuyu Ren, Doru Cojoc, Pavel Arkhipov, Usman Qazi, Lianghui Li, Sumeet Motwani, Christian Schröder de Witt, Edwin Taylor, Johannes Veith, Eric Singer, Taylor D. Hartman, Paolo Rissone, Jaehyeok Jin, Jack Wei Lun Shi, Chris G. Willcocks, Joshua Robinson, Aleksandar Mikov, Ameya Prabhu, Longke Tang, Xavier Alapont, Justine Leon Uro, Kevin Zhou, Emily de Oliveira Santos, Andrey Pupasov Maksimov, Edward Vendrow, Kengo Zenitani, Julien Guillod, Yuqi Li, Joshua Vendrow, Vladyslav Kuchkin, and Ng Ze-An. Humanity's 's last exam. CoRR, abs/2501.1424 14249,, 2025. doi: (0.48550/ARXIV.2501. 14249. URL https://doi. org org/10. 48550/ www.a.ivvvv........... 14249.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Mcasuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 5687-5711 1. Association for Computational Linguistics, 2023. doi:
F2NN EMNLP.378. URL https:://do.. org/10..///oo 1et///....1663//// 18653/v1/ 2023. findings-emn1 378.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent Q: advanced reasoning and learning for autonomous Al agents. CoRR, abs/2408.07199, 2024. doi: 10448550/ARXIV.2488..777 mrxiv.2408 07199.
URL https://doi.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilck Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504. /395588,04. 2025a.
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. SMART: self-aware agent for tool overuse mitigation. CoRR, abs/2502.11435, 2025b. doi: 10448550ARRXI.2502..10 11435.
URL httpps://oi:. org/ (aarr/g/..485550 http://...055//........
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma Gongque, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Honggang Zhang. We-math: Does your large multimodal model achieve human-like mathematical reasoning? CoRR, abs/2407.01284, 2024. doi: 10.48550/ A.RXIV22407001266 URL https: / //oi org/10 wtrgg/0..o 08w.:/488/858// arxiv. 2407 01224.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview net nett/oormmiiddddng2200000
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2024. URL https://arxx www.ooggg///////oooo/oooooooe/// 2ttbb//241...111.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters. KDD '20, 2020.
16

---

Published as a conference paper at ICLR 2026
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, et al. Tptu: Task planning and tool usage of large language model-based ai agents. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707.06347, 2017. URL http:://axxiv.rr/abbaa. 06347.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: 10.45500ARXX URL https://doi, org/10. 2055444555aaii/ 2ww.....2222..333.0
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint arXiv:2010.03766 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017. URL http://arxet org org//bs/////////// 1712.01815.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for Ilms via reinforcement learning. arXiv preprint arXiv:2505.01441, 2025a.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for iims via reinforcement learning. CoRR, abs/2505.01441, 2025b. doi: 10.48550/ ARXIV.2505.01441. URL https: org/10. 48550/arXiv. 2505. 01441.
Huatong Song. Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in lims via reinforcement learning. CoRR, 0atos//533.0552, 2025. doi: 10.4550/ARKXI22500.5522 URL htctps:/ddoi.org/// 48550/ aa.50//rxiv.. Xiv. 2503.503300 85592.
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of Ilms without searching, 2025a. URL tittps://arxxer http:////aaxxxxxxa//////////////
argg/b//// 04588.
Shuang Sun, Huatong Song, Yuhao Wang, Ruiyang Ren, Jinhao Jiang, Junjie Zhang, Fei Bai, Jia Deng, Wayne Xin Zhao, Zheng Liu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. Simpledeepsearcher: Deep information seeking via web-powered reasoning trajectory synthesis. 2025b. URL https: //gi thub. RAA.AAAAAOoo//mmeeeeeeeeeeeeee
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrum Wang, and Bo An. True knowledge comes from practice: Aligning large language models with embodied environ- ments via reinforcement learning. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreviier eww. met/forum?id-hIL VmJ4Uvu.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint 2rXiv:2507.20534 2025a.
17

---

Published as a conference paper at ICLR 2026
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with Ilms. arXiv preprint arXtv:2501.12599 2025b.
Qwen Team. Qwen2.5: party of foundation models, September 2024a. URL https://wweni.m github. io/blog//////ooo /qwn/. 5/.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face, 2024b.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024c. URL https://qwenl.. gi github. io/blog/que httlbllogggg--/-aoeeoogooggg http///wwwbbbpeeeeeeee 52b-prreview/
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 00:5,9-55,, 2022.
Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the kl penalty! boosting exploration on critical tokens to enhance fine-tuning. arXiv preprint arXiv:2502.06533 2025.
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang. Bowen Jin,
Mengdi Wang, Kam-Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning. arXiv preprint arXiv:2504. 14870, 2025a
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song. Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, abs/2506.01939, 2025b. doi: URL https: //ooi. org/10 48550/
10:48555/ARXIV.25566000
2506. 01939.
Taiyi Wang. Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents. CoRR, hiss22110114803 2024. doi: 10.48550/ARXIV.44100 148.1333.. URL https: //doi. org/10. 48550/ ar.vv. 14803.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng. Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang. Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint urXiv:2504. 20571, 2025c.
Zihan Wang, Kangrui Wang, Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang. Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, 2025d. URL http.httts.///xii.orrr/////// 20073.
Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforcing Ilm agents via policy optimization with action decomposition. Advances Neural Information Processing Systems, 37: 103774-103805. 2024.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. Webwalker: Benchmarking Ilms in web traversal. CoRR, abs/250l.07572, 2025a. doi: 1044850/ARXIV.55010077 //doi.org/1e 48550/ 2501 .75772.
URL https:
Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning Ilms with tools for the deep research. arXiv preprint 250::25002006444,, 2025b.
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li. Wei Peng, and Sujian Li. Watch every step! LLM agent learning via iterative step-level process refinement. In Yaser Al- Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA. November 12-16, 2024, pp. 1556-1577 1572. Association for Computational Linguistics, 2024. doi: 10.18653/V1/ 2024.EMNLP PP MAI 93. URL https: //doii//oiii. org/10. 18653/v1 /22244 emnlp main.
18

---

Published as a conference paper at ICLR 2026
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforeement learning for multi-turn hol-integrated reasoning. arXiv preprint 000K022500.02279, 2025.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao. Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407. 10671, 2024. doi: 10. 40.48550/ARXIV224077 10671. URL https: / //w.i//ooioooooo/ org/ 10. 48550/ ar Xiv. 2407 10671.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujic Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jian Yang. Jiaxi Yang. Jingren Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, https://doi, org/10///// 10. 48555//55iii..../ 250.......555.... 25055.9388.
2025. doi: 10.4550/ARXIV.2505.000 009388. URL
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pp. 2369-2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18- 1255. URL https://aclantthooog. org/
018-1259,
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprini 1rXiv:2210.03629, 2022.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint 25t.::2503.147700 2025.
Yuanqing Yu, Zhefan Wang, Weizhi Ma, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang Steptool: Enhancing multi-step tool usage in lims through step- step-grained reinforcement learning. arXiv preprint
2024.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Cheng- Xiang Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang. Yonghui Wu, and Lin Yan. VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks. CoRR, abs/2504.05118, 2025. doi: 10.44550/ARXX1.2504000 08118. URL https: //doi. .rgg 100g 48550/arXiv 2504. 05118.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. CORR, abs/2503.18892, 2025, doi: 10448500ARXXV.2200.18582 URL https://doi. org/10. 48550/
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Sain- ing Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision- mod- els as decision-making agents via reinforcement learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang
19

---

Published as a conference paper at ICLR 2026
(eds.), Advances in Neural Information Processing Systems 38: Annual Conference Neu- ral Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, Decem- ber 10 15, 2024, 2024. URL -bstractt-Cofeeeenneee c848b7d3adc08f aat.aaado00000o01111111111116120 oooof html.
nips. cc/ /ttpee////appe//22//aaas
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, al. The landscape agentic reinforcement learning for llms: A survey. arXiv preprint arXiv:2509.02555 2025.
Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. First return, entropy-elicitiene explore, 2025a. URL / wttp:/.wwwg..////.ot/oo/ooooss/ 2507 07017.
Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, et al. First return, entropy-eliciting explore. arXiv preprint 1iXx:2507.07011 2025b.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL http: /arxiv. http://axxvv.orggbbb.ss/s http//bbb///..... 2403. 13372.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar, Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402. 19446, 2024.
20

---

Published as a conference paper at ICLR 2026
APPENDIX
Supplement Experimental Results
22
A.1 Ablations of Browser Agents.
22
A.2 Scaling Analysis of ARPO
22
Supplement Related Work
22
B.1
Reinforcement Learning with Verifiable Reward.
22
B.2 Agentic Reinforcement Learning..
23
Datasets
23
C.I Mathematical Reasoning Benchmarks
23
C.2 Knowledge-Intensee reasoning benchmarks
24
C.3 Deep Search Benchmarks
24
D Baselines
25
D.1 Direct Reasoning
25
D.2 Trajectory-level RL Algorithms
25
D.3 LLM-based Search Agent
26
E
Implementation Details
26
E.1 Training Guideline
26
E,2 Supervised Fine-Tuning
27
E.3 Reinforcement Learning
27
E,4 Details of Search
27
E.5 Scaling Experiment Setup
27
F
Theoretical Analysis and Proofs
28
F.1
Preliminary of Agentic Reinforcement Learning
28
F2
Thcoretical Analysis Soft Advantage Estimation
28
F3
Theoretical Proof of GPG Theorem
30
E3.1
Transformer-based Policy
30
F33.2
Derivation of the GPG Theorem
Use LLMs
32
H The Algorithm Workflow of ARPO
33
Case Study
34
21

---

Published as a conference paper at ICLR 2026
A SUPPLEMENT EXPERIMENTAL RESULTS
A. ABLATIONS OF BROWSER AGENTS.
To further assess the impact of the browser agent on the deepsearch task, we design three experimental settings with progressively stronger browsing capabilities: (1) no browser, where only retrieved snippets are available; (2) a browser agent comparable scale to the reasoning model; and (3) larger-scale browser agent with more parameters.
As shown in Table 3, results show that the setting without a browser exhibits the worst performance consistency, indicating that relying solely rule-generated web snippet summaries is insufficient to provide the necessary information support in deep search tasks. This highlights the necessity of web content fetching and browsing. As the capability of the browser agent increases, model performance also improves significantly, demonstrating that a more powerful search agent can more effectively integrate information and extract key details relevant to the question. In summary, the capability of the external browser agent is highly correlated with the accuracy of the Deepsearch task and shows clear upward trend as scale increases.
A.2 SCALING ANALYSIS OF ARPO
To verify the scalability of ARPO and gain deeper insights into its characteristics, we use the Qwen2.5-7B model as the backbone for scaling analysis of three core parameters: en- tropy value, global rollout size, and initial sam- pling size. Our observations are follows:
Table 3: Ablation studies of the backbone model of browser agents in deep search tasks.
Method
GALA HLE Wel Avg.
Qwen3-8B
+ Snippet only
33.0
7.5
29.0
23.2
Entropy Value (AH+): As shown in Figure (left), model performance increases with rising
+ Qwen3-8B Browser
38.8
8.8
30.5
26.0
+ QWQ-32B Browser
38.8
8.2
33.0
26.6
entropy values, peaking at 0.4. This indicates that integrating moderate amount of entropy
Qwen3-14B
+ Snippet only
35.0
8.4
31.0
24.8
as a clue for partial sampling substantially en-
+ Qwen3-14B Browser
43.7
10.0
36.0
29.9
hances the model's ability explore rare tool-
+ QWQ-32B Browser
47.6
32.3
38.4
39.4
use behaviors, thereby improving training out- comes. However, entropy reaches 1.0, perfor-
mance declines, suggesting a trade-off in the weight of entropy in sampling. Over-reliance on entropy may reduce sampling diversity, confirming the necessity of balancing base sampling probabilities with entropy in ARPO.
Initial Sampling Size (N): Figure (middle) illustrates that as the initial sampling size increases, model performance improves, peaking at 8. Notably, with global rollout size of 16, increasing the initial sampling size from 0 to 8 shifts the global-to-partia sampling ratio from 1:15 1:1. This underscores the importance of balancing sampling proportions for improving performance. As anticipated, increasing the size to 16 results in a great performance decline. This because it leads to complete global sampling, which disrupts the dynamic sampling balance.
Global Rollout Size (M): As depicted in the Figure 8 (right), increasing the global rollout size enhances model performance, indicating that the ARPO algorithm is scalable and can improve generalization performance with larger sizes.
B SUPPLEMENT RELATED WORK
B. REINFORCEMENT LEARNING WITH VERIFIABLE REWARD.
Recently, Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024; Kaufmann et al., 2025) has become leading approach in Reinforcement Learning through Human Feedback (RLHF), particularly excelling in enhancing mathematical and programming reasoning (Shao et al., 2024; DeepScek-AL et al., 2025; Yang et al., 2025; 2024: Team, 2024b;a; Dong et al., 2024c; Qiao et al., 2024). OpenAl ol (OpenAI, 2024) first showcased RL's effectiveness in large-scale reasoning tasks. Building this, models like DeepSeek R1 (DeepSeek-Al et al., 2025), QwQ (Team, 2024c),
22

---

Published as a conference paper at ICLR 2026
Result VS. Entropy Weight
Result VS. Initial Sampling Size
Result Global Rollout Size
65
65
61.1
Val. Score
60-
59.8
Val. Score
60-
Vail 5aa.- Soore
60
57.8
58,6
55-
55.1
55
56.2
55.3
55
52.4
50-
50-
50
45
45
0.2
0.4
0.8
1.0
2
4
8
16
2
4
B
16
Entropy Weight
Initial Sampling Size
Global Rollout Size
Figure 8: Scaling analysis of different Flype--aarametess in Qwen2.5-7B with ARPO. The detailed
setting can be found Appendix E.5.
and Kimi k1.5 (Team et al., 2025b) aim to replicate and surpass its performance. To improve RL algorithms' performance and stability, researchers have developed models like DAPO (Yu et al., 2025) and SimpleRL.Zoo (Zeng et al., 2025), exploring algorithm design across various RL. modules(Hu et al., 2025; Yue et al., 2025; Feng et al., 2025b; Liu et al., 2025; Kool et al., 2019; Ahmadian et al., 2024; Dong al., 2024a; Hu, 2025). Lin et al. identified key tokens affecting errors and showed that replacing them can alter model behavior. Studies (Gandhi et al., 2025; Li et al., 2025b) found RLVR primarily learns format Cheng et al., 2025; Wang et al., 2025c) pointed out key tokens to high-entropy tokens to explore RL. learning'ss essence. In recent work, segment-level reinforcement learning approaches have been proposed to broaden the exploration space (Guo et al., 2025; Li et al., 2025g; Zheng et al., 2025a). However, RLVR algorithms specifically for LLM agents remain underexplored. This paper uses entropy as a criterion to investigate reinforcement learning algorithms suited for LLM agent behavior.
content, while several works (Vassoyan et al., 2025; Wang et al., 2025b;
B.2 AGENTIC REINFORCEMENT LEARNING.
Reinforcement learning (RI) is essential for enabling L.LM agents to adapt to dynamic and open environments (Lu et al., 2025; Shridhar et al., 2020; Mialon et al., 2024). Foundational works like DQN (Mnih et al., 2015) and AlphaZero (Silver et al., 2017) demonstrate that self-play-based RL can equip agents with capabilities from natural language understanding to strategic gameplay (Narasimhan et al., 2015). Building on this, value-based RL approaches have been employed to enhance embodied intelligence in hardware control and complex gaming tasks (Tan et al., 2024; Zhai et al., 2024; Bai et al., 2024; Wang et al., 2024; Schulman et al., 2017; Peng et al., 2019). Recent efforts, exemplified by RAGEN (Wang et al., 2025d; Zhou et al., 2024), integrates reasoning states and environmental interactions into turn-level responses using trajectory-Ievel RI. To improve tool-integrated reasoning, studies (Jin et al., 2025a; Feng al., 2025a; Song al., 2025; Jin et al., 2025a; Chen et al., 2025b; Feng et al., 2025a; Li et al., 2025f; Sun et al., 2025a; Li et al., 2025e; Singh et al., 2025a) employ rule-based RL to teach LLMs how to autonomously invoke external tools (e.g. search engines, Python compilers) to boost reasoning accuracy. Further research, including ToolRL (Qian et al., 2025a), Tool- Star (Dong et al., 2025), and OTC (Wang et al., 2025a) explores multi-tool integration and tool-use efficiency. A series of works led by Kimi Deepresearcher and Websailor (Li et al., 2025c) optimize RL algorithms to better adapt to deepsearch's 's long context scenarios. While most works improve tool invocation through reward shaping and rollout mechanisms, simply applying rajectory-lever RL fails to effectively capture the multi-turn, long-horizon characteristics of LLM-based agent behavior. This motivates the proposal of ARPO attempt learning step-level tool-use behavior patterns.
C DATASETS
In this section, we provide a detailed introduction to the datasets used in ARPO's experiments.
C.1 MATHEMATICAL REASONING BENCHMARKS
AIME24 is a dataset in evaluating the mathematical reasoning ability of models. It consists of 30 challenging math problems. All of them are from the American Invitational Mathematics
//t.ossosi./mooottaiii..oo. wig.ihhtttiigittth.bu.ee wt/Kimiiiaiiieeeaaieieeeeii Bwi.iiimi--eeeeathii chee/ 7tttps: co/
23

---

Published as a conference paper at ICLR 2026
The problems in the AIME24 dataset cover wide variety of mathematical fields such as algebraic equations and geometric puzzles. Due to the difficulty characteristics and the richness of question types, it has become popular benchmark for evaluating the reasoning performance of models, and is widely used multiple related research experiments.
Examination.
AIME25 consists of 30 challenging math problems. It directly composed of the real questions from the American Invitational Mathematics Examination (AIME I & II) newly released in February 2025. AIME25's knowledge areas are extremely wide. deeply covers core mathematical sections such as algebra, geometry, number theory, and combinatorial mathematics. This characteristic enables the AIME25 dataset to effectively distinguish the mathematical reasoning abilities of different models.
MATH500 (Lightman et al., 2024) is selected by OpenAI from the MATH evaluation dataset. It contains 500 high-difficulty math problems. These problems cover multiple mathematical fields such as algebra, geometry, calculus, and number theory. The difficulty is close to exceeds the college level. In academic research, MATH500 dataset is often used to evaluate the performance of various reasoning models.
MATH (Hendrycks et al., 2021) is significant academic dataset. It is designed to test and enhance models' mathematical reasoning skills. covers wide range of mathematical fields, including abstract algebra, caleulus, and discrete mathematics. The dataset divides training data into three levels, which helps effectively evaluate model performance at different stages.
GSM8K (Cobbe et al., 2021) is an elementary school math problem dataset released by OpenAL. These problems require 2 to steps to solve, mainly through series of basic calculations to obtain the final answer. This dataset primarily used to test the logical and mathematical abilities of models and has been applied in multiple benchmark tests.
C.2 KNOWLEDGE-INTENSIN REASONING BENCHMARKS
HotPotQA (Yang et al., 2018) is a question-answering dataset for multi-hop. All the documents sourced from Wikipedia, which provides the dataset with rich knowledge base and relatively structured information. It is an important benchmark for evaluating the ability of LLMs to understand complex search tasks.
2WikiMultihopQA (Ho et al., 2020) is a dataset specifically designed for the multi-hop question- answering task. It aims to test and evaluate the ability of natural language processing models to answer questions that require multi-step reasoning and the integration of information from different documents.
Musique (Trivedi al., 2022) is question-ansee answering dataset specifically designed for the multi-hop guestion-answering task. Musique aims to be challenging benchmark for evaluating models' multi-hop question-nsswering ability. It promotes the development of models from simple information retrieval to deeper semantic understanding and logical reasoning.
C.3 DEEP SEARCH BENCHMARKS
GAIA (Mialon et al., 2024) is designed to evaluate the comprchensive capabilities of LLMs in real-world tasks. The dataset contains 466 carefully designed questions to test the performance of Al systems in basic capabilities. It contains tasks such as reasoning, web browsing, and tool use. The proposal of GAIA provides new framework for the evaluation of general artificial intelligence assistants.
HLE (Phan et al., 2025) is an emerging and highly challenging benchmark dataset. It aims to deeply evaluate the performance of LLMs when faced with complex questions requiring deep understanding and complex reasoning. This dataset covers a large number of marginal, interdisciplinary problems that demand highly abstract thinking to solve. Different from traditional benchmarks, HLE aims to simulate an ultimate test of Al intelligence.
Welker (Wu et al., 2025a) is a dataset used evaluate the performance of L.MMs in web traversal tasks. This dataset contains 680 question-answer pairs. It aims to address the limitations of LLMs when dealing with complex information. Additionally, it improves the models' capabilities in molti-hop reasoning and dynamic web page structures.
htttss///heggineeeee....aatttttt ai aime25
24

---

Published as a conference paper at ICLR 2026
Deeach- (Chen et al., 2025a) is evaluation set for assessing the deep search capabilities of Al agents. This dataset takes full consideration of the breadth of the search space and the depth of reasoning. Different from existing knowledge search benchmarks, xbench- Seecch-eeppSeachh is more capable of examining the high-order capabilities of agents.
D BASELINES
In this section, we introduce baselines used in our work.
D.1 DIRECT REASONING
Qwen2.5 Series (Qwen et al., 2024) is a series of LLMs developed by the Alibaba team. It includes the general- purpose language model Qwen2.5, the geggraaming-sppccifee model Qwen2.5-Code and the wwweemmatccssspeciiee model Qwen2.5-Matt The Qwen2.5 series of models have been pretrained on large- datasets. Compared with past Qwen series models, the Qwen2.5 series have richer knowledge reserve. In addition, it has good performance in various tasks such as programming, mathematics, and instruction following.
Llama3.1 Series (Dubey et al., 2024) is series of natural language generation model launched by Meta. It includes three specifications: 8B, 70B, and 405B. These models can handle longer text inputs and generate more coherent long-text outputs. This series of models also performs well in multilingual tasks. The Llama 3.1 series of models have undergone performance tests on more than 150 benchmark datasets. Its large-scale model competitive with leading base models in series of tasks. The smaller 8B and 70B models also perform well in comparisons with closed-source and open-source models with a similar number of parameters.
Qwen 3 Series (Yang et al., 2025) is series of open-source model developed by Alibaba. The Qwen3 series of models includes 2 Mixture-of-Frxe -f-eeeets-s (MoE) models and 6 Dense models, with the number of parameters ranging from 0.6B 235B. Qwen3 natively supports the thinking mode and non-thinking mode. In the thinking mode, the model reasons step by step and is suitable for handling complex problems. The non-thinking mode can provide fast, nearly instant response and is suitable for simple problems. Qwen3 builds a training corpus based on approximately 36 trillion tokens, ensuring the model's powerful capabilities and flexibility.
QwQ (Team, 2024b) is open-source inference model launched by the Alibaba team. It focuses on enhancing Al's capabilities mathematics, programming, and complex logical reasoning. The BwB-32BBRR is dense model with 32 billion parameters. It surpasses most existing models in core tasks such as mathematical reasoning and code- gedeeeenerttion ability. The QwQ-32B achieves breakthroughs through innovative multi-stage reinforcement learning. Its core training approach lies in gradually expanding general capabilities while consolidating specialized advantages.
DeepSeek-RI (DeepSeek-AI al., 2025) is reasoning model developed by DeepSeek-AI. DeepSeek-R is trained using reinforcement learning. The inference process involves a large amount of reflection and verification, and the length of the thought chain can reach tens of thousands of tokens. It performs outstandingly in tasks such mathematics, code, and various complex logical reasoning.
GPT-4o (Hurst al., 2024) is multimodal LLM released by OpenAI. GPT-4o accept any combination of text, audio, and images as input. In addition, it can generate any combination of text, audio, and images as output. GPT-4o has achieved performance comparable that of GPT-4 Turbo in aspects such as text, reasoning, and coding, Moreover, it has also set new highs in performance scores for multilingual, audio, and visual functions.
ol-preview (Hurst et al., 2024) is preview version model in the ol series of LLMs launched by OPenAI. It represents an important breakthrough in the field of reasoning. ol-preview is based on the GPT-4 architecture and trained through reinforcement learning. It aims enhance the reasoning ability for complex tasks and the ability to solve practical problems. It can exhibit powerful abilities in tasks that require in-depth reasoning.
D.2 LRAJECTORY-LEVEL RL ALGORIHTM
GRPO (Shao et al., 2024) is a reinforcement learning algorithm based on policy optimization. aims to address the balance issue among stability, sample efficiency, and theoretical guarantees
25

---

Published as a conference paper at ICLR 2026
in traditional policy optimization methods. By introducing the concept of relative advantage, it simplifies the calculation while maintaining the theoretical guarantee of policy improvement. The GRPO algorithm is applicable to reinforcement learning tasks in both continuous and discrete action spaces.
DAPO (Yu et al., 2025) is a LLM reinforcement learning algorithm developed by ByteDance Labs. It aims to address the key challenges of large-scale RL training. It performs outstandingly in complex tasks such as mathematical reasoning and code generation. The Clip-Higher strategy proposed by DAPO effectively increases the entropy value, facilitating the generation of more diverse samples. In addition, it introduces mechanisms such as dynamic sampling, Token-Level Policy Gradient Loss calculation, and Overlong Reward Shaping to stabilize the training process.
REINFORCE+ (Hu, 2025) is new algorithm for improved versions of the classic REINFORCE algorithm. Its core objective is to address the limitations of the original REINFORCE, and enhance performance by integrating multiple optimization strategies. REINFORCE++ typically incorporates a baseline function to reduce variance by subtracting the baseline. Through the baseline and TD estimation, REINFORCE+ makes the gradient update more stable. It doesn't t need to wait for complete trajectory and supports incremental updates. In addition, it avoids premature policy rigidity through entropy regularization.
D.3 LLM-BASED SEARCH AGENT
RAG (Lewis et al., 2020) (Retrieval-Augmeette) Generation) a technical approach that combines information retrieval with generative model. It aims to improve the accuracy, reliability, and timeliness of the output of generative models. Its core idea before generating an answer, first retrieve information related to the question from external knowledge base, and then let the model generate a response based on the retrieved content. This can solve the problem of internal knowledge deficiency or ballucination within the model some extent.
Search-ol (Li et al., 2025d) is Agentic search-enhanced reasoning model framework. It is mainly designed to address the knowledge deficiency problem existing in the reasoning process. By integrating the Agentic RAG mechanism and the Iesson-i-DDocumen module, it improves the accuracy, coherence, and reliability of model reasoning. Experiments show that Search-ol outperforms native reasoning and traditional RAG methods in complex reasoning tasks.
WehThinker (Li et al., 2025e) is open- source in-depth research framework launched by Renmin University of China. endows LRMs with the ability to autonomously search, deeply explore web pages, and write research reports. WebThinker has developed a training strategy based direct preference optimization. uses training with preference data through iterative synthesis tools to enhance the tool utilization ability of LRMs.
ReAct (Yao et al., 2022) is artificial intelligence method that combines reasoning and acting. aims to enable models to solve complex tasks more effectively through "thinking while doing" mode similar to human thinking. Its core idea is break the limitation of the traditional model, allowing the model to actively generate reasoning steps and call external tools (such as search engines, databases, etc.) during the decision- process, and finally obtain the answer through iterative optimization.
E IMPLEMENTATION DETAILS
E.
TRAINING GUIDELINE
Our study aims to validate the effectiveness of ARPO at the algorithmic level compared to tradi- tional RL in training L.LM agents, rather than merely pursuing performance improvements. To ensure reproducibility. all training frameworks and datasets are sourced from open- resources. Specifically, our experiments adhere to the cold-start SFT with RL paradigm (Song et al., 2025; Dong al., 2025) to mitigate reward collapse during the initial RL training phases.
1. Cold-Start Finetuning Phase: Utilizing the LLaMAFactory (Zheng et al., 2024) framework, leverage Tool-Star's open-source dataset of 54K training samples. To enrich the quality of mathematical reasoning data, we incorporate the STILL dataset (0.8K), drawing inspiration from CORT (Li al., 2025a).
26

---

Published as a conference paper at ICLR 2026
2. RL Phase: To assess ARPO across various scenarios, explore the following domains:
Deep Reasoning Tasks: This includes computational reasoning (e.g., AIME24, MATH500) and multi-hop knowledge-based reasoning (e.g.. HotpotQA, Bamboogle). We utilize Tool-Star's 10K open-source RL training samples for algorithmic comparison.
Deep Search Tasks: These tasks require extensive web exploration and information integration, necessitating longer contexts and frequent tool interactions. We use only 1K mixed hard search samples from SimpleDeepSearcher (Sun et al., 2025b) and WebSailor (Li et al., 2025c) for training.
To expedite the RL training phase, incorporate top-10 snippets from the Bing search engine as search results, employ a Python compiler within sandbox environment, and use token-level F1 scores the correctness signal 9.
E.2 SUPERVISED TUNING
As mentioned in Section E, during the supervised fine-tuning phase, we train the -wwnn..--BB- Instruct model using the Llama Factory framework with learning rate of 7 10-6, We employ DeepSpeed ZeRO-3 (Rasley et al., 2020) and FlashAttention2 (Dao, 2023) for optimization. The batch size to 128, with weight decay of 0.1, and the model is trained for 3 epochs. We use BF16 mixed precision with a maximum input length of 4096 tokens.
E.3 REINFORCEMENT LEARNING
In the ARPO phase, implement the ARPO algorithm based the VERL framework (Sheng et al., 2024). Notably, all tool invocation results are excluded from loss calculation to prevent bias towards tool outputs. The loss computation only considers tokens involved in text reasoning and tool requests. We differentiate settings for Deep Reasoning Tasks and Deep Search Tasks:
1. Deep Reasoning Tasks:
For models with 7B parameters, whether using ARPO other trajectory- level RL methods, our standard setup includes a total training batch size 128, a PPO mini-batch size of 16, global rollout size of 16, and initial sampling size of 8. Each interaction response length is capped at 4096 tokens. For ARPO rollouts, we set the entropy weight to 0.2, the parameter 0.5, and the threshold to 0.5. To stabilize training, the KL divergence coefficient GRPO is set to O. The reinforcement learning phase spans 2 epochs, conducted on 8 NVIDIA H800 GPUs.
Deep Search Tasks: For models with 8B parameters, we maintain the same settings as in the Deep Reasoning Tasks, except that each interaction response length is extended to 8192 tokens. For 14B models, same parameters are used, but experiments are conducted on 16 NVIDIA H800 GPUs. Due to limited dataset of IK samples, the reinforcement learning phase lasts for 5 epochs.
E.4 DETAILS OF SEARCH
During the training and testing phases, we used the Bing Web Search API as the retriever, configured with the US-English (US-EN) locale. Following a series of related works on RAG (Jin et al., 2024; Li al., 2024b; Dong al., 2024e;b;d), we retrieved 10 web pages as supporting documents for each query.
For mathematical and knowledge reasoning, we evaluated using only the top 10 snippets. However, for deep search tasks, we fetched each page with up to 6000 tokens from the URLs and used a model of the same size as the reasoning model browser agent to refine the information.
E.5 SCALING EXPERIMENT SETUP
In our scaling experiments, we align with the aforementioned settings: a total training batch size of 128, PPO mini-batch size of 16, global rollout size of 16, and an initial sampling size of 8. For ARPO rollouts, the entropy weight is 0.2, a is 0.5, and the threshold is 0.5. We vary specific parameters for targeted experiments while keeping others constant.
In this paper, we only the I.I correct grammatical errors.
27

---

Published as a conference paper at ICLR 2026
F THEORETICAL ANALYSIS AND PROOFS
F.1 PRELIMINARY OF AGENTIC REINFORCEMENT LEARNING
In this section, we formulate the agentic RL training objective as:
max D.g.D..yrrof () [To v)] - 3 DRI. [mo(y x;; T) || Ter x; T)), 2
where 7 denotes the set of available tools, нe represents the policy LLM, Tif is the reference LLM, Ts and DxL. denotes the reward function and KL divergence respectively. The input æ sampled from dataset D, and y is the corresponding output, possibly interleaved with tool-call feedback.
Unlike conventional RL methods that rely solely on LLM rollouts, agentic RL incorporates tool-call feedback during the reasoning process (Chen al., 2023; Gou al., 2024; Li et al., 2025f; Wu et al., 2025b; et al., 2024a). The rollout sampling can be decomposed as:
Po 2
2;
II (R, R;
II Polye I Vec, R;
(8)
Agentic Reasoning
Answer Generation
where is the reasoning trajectory of length tR, interleaved with tool-call feedback, and y is the final answer with length fg.. Our ARPO is built upon all-based RL algorithm (e.g. GRPO (Shao et al., 2024), Reinforce++ (Hu, 2025)) designed to optimize LLM-based agents.
F.2 THEORETICAL ANALYSIS OF SOFT ADVANTAGE ESTIMATION
In this section, we conduct a detailed theoretical analysis of Soft Advantage Estimation. First, we present the classic GRPO optimization objective:
lod min (0)
JGRPO = =E Te (11)
(9)
clip (0), -, 1 +)) Aes) - -Daaa(T || TTee)
For each individual problem, we define the optimization objective as:
1 1 loil G
Jy TORPO (
min (rp..)) - -BDKL(T Tee)
(10)
Therefore, the classical GRPO optimization objective can be expressed as:
(8) E D) (11) JuRPo (f)
(11)
Subsequently, focus analyzing JORPO (0). Assume that for the inference of problem q, the partial rollout operation starts from the 1-th token. We define two importance sampling ratios:
Tolyiie x, Vi.ce) marl 2,, Mi,e()
(0)=
(12)
(0) >1 Ta(p x. To(yi, Ir q. Wi.c1)
(13)
Tut( x. 7) (ar( Jr, 9. (b,
where (0)< and (0) represent the importance sampling ratios before and after the I-th token, respectively, q represents the input question, p represents shared tokens, and yi in 13 represents
28

---

Published as a conference paper at ICLR 2026
the sequence from shared tokens to before the t-th token. In addition, we define of as the I-th token of the i-th sequence. Then (8) can be expressed as:
G
(0)
:lipp min (1), clip (r), 1 = c) A..
min (O) (12,10),, 1 =() Ass
- (Teet
lo
min (1)) clip (r)) + v) A..
0
G
loi
+
Σ
min
(12) (r2, +)) ) Av.
E=1
lofl
min
(1) (1 clip (r), + c) A..
G
1=1
lod
[0tal
min (rz) (17.(0)) = c) ()) A..
+
3DD Tret)
|0:|
1=1
lofl
min
(r),(0)=)), clip (1), + +() A..
losl
1=1
10:|
101
+
min (O) (0), + tc) An,
(To || mTet)
[0s|
1o
G
Σ
min
(ri,)())=1, < clip (0), 1 + s) A..
0
1=1
led|
min (0) (r². + =() ) )
G
i=1
t=
led|
10:
min (121 (r2 1() A.. D || Tref)
10:|
01
1=1
10
loi1
10:|
min (ra.(0)s), clip (r.) ALe
0
la:
(12))) (0), ) ALe min (12) clip
G
loul
1
min (O) (1)))) + Ars
DBDra Tee))
G
lotal
1=1
(14)
We make the following definitions:
lo
J
min (0)= clip (r.) + (t)) Are.
(15)
00 1=1
29

---

Published as a conference paper at ICLR 2026
lod
J,dd
min
clip +() Aics
(16)
O1:s
lojl
ld
JeRPO
Σ
min (12) (ce)
(17)
G
where J represents the optimization objective the shared tokens part of the i-th chain, J," represents the optimization objective after partial rollout of the i-th chain, and J represents the optimization objective of directly performing the classical GRPO sampling operation starting from the /-th position. Then, the original optimization objective (f) can be expressed as:
G oo Σ
(c) 1
J
+
J7'
3 (ee
Shared Token Advanlage
(18)
G lo:| [o,1
o
J
[J:4]
+J
(To (eer)
G 1=1
G 2=1
Hard Advaniage Estimation
Regularization Term
In this case, the GRPO optimization objective under Soft Advantage Estimation be expressed as the sum of standard GRPO objective from the partial rollout position, and the weighted difference between the objectives before and after that point. The weight of each difference is closely related to the length of its corresponding reasoning chain.
worth noting that, through our theoretical derivation and decomposition of the objective, we find that under the soft setting, the share token advantage consists of two components: (1) the first term aligns with the Hard Advantage Estimation; (2) the second term corresponds the portion after the partial rollout position 1, which is related to the individual token advantage. We interpret this second component as regularization term. As result, the estimated advantages under the hard and soft settings approximately equivalent, with the key distinction being the presence of this regularization term. This leads us to hypothesize that the improved stability the Soft Ad Advantage Estimation may be attributed this regularization effect.
F.3 THEORETICAL PROOF OF GPG THEOREM
F.3.1 TRANSFORMER-BASE POLICY
The Transformer-base policy no (st). by applying the chain rules, we have the following:
(OTi
ITi, IT2, IT IT (mpput)) x
(OT2
IT1, IT2,..., IT (mmpput) OTi)x
To
IT1, IT2,..., IT (mimpptt) OT,, OTz)
(19)
To (oTtpupt ITi...., Tiinputt OT1, .... OT (otpput)-))
-Te OT2, OT outpul) ITi, IT2, ..., IT impal)
To (MA MS;)
where IT; and OT, are input tokens and output tokens, respectively; MS A< IT1, .T,..., IT Knppat] > and MA & OT,...,... OT output] can be token as the macro state and the macro action, respectively.
In more general form, we can split the complete output OTi, OT2,..., OT joutpptt into K segments, and get the generalized macro states and macro actions, i.e. MS, &C MS,11, M A > and
30

---

Published as a conference paper at ICLR 2026
MA, E< OTm, OT .... OT +m >.
In this case, we have the following:
To JM A
MS1)
m (M A
MS
M A2
no (MMAA
MSI, MA,, M A2, ... -A-11)
mo (M A
MSI)x
(20)
M A2
MS2)x
no To | MSk)
II (M MST)
T=1
where T represents the macro timestep.
F.3.2 DERIVATION OF THE GPG THEOREM
Given the macro states and macro actions defined above, we can get the Generalized Policy Gradient Theorem (for Transformer-bass policies):
VaJ(O) =Ey
Σ T=1
Vo log No (MAr(())MM))
(21)
key advantage of the GPG Theorem is that allows macro-action segmentation of arbitrary length. This flexibility makes the theorem highly practical: for instance, enabling trajectory splitting based on special tokens.
The proof is as follows:
VaJ(0)
(22)
VOlENNA [R(T)]
(23)
V Σ P(T;O)R((T)
(24)
(oP(T:0)((7)
(25)
Σ 0( VpP(T; R(T) P(T:0)
(26)
Σ P(T: log P(r: R(())
(27)
H
Σ Vo II
To (a (ote
(28)
H E V(T:
(a 5z) P(s a.)]
(29)
Σ P(T:0) (log
( (s))ss)))
(30)
Σ Vlr flog
To (M II T=11
(31)
P(T:0)
Vo log no MA [M()
(32)
31

---

Published as a conference paper at ICLR 2026
Σ P(T:0) ():
Ve log (MMT
(33)
T=1
K
B, y
Ve log no (M
(34)
T=1
Vo log To M MM
(35)
T=1
Key steps in the proof presented below:
1. From Equation (29) to Equation (30), this because 8.41 [St, as] for Transformer-based policy, so we have P(s++1 Se, ar) 1.
From Equation (30) to Equation (31), this is because 8141 [8c, ar] for Transformer-based policy, so we have the following derivation:
II (e(a)(s))
(ailss)) x TT(a2)52) x ... TT(aH(sH)
(a(aas))) x Te(a2|51.
x Te(aH ao a1.... a1
(m a2 ... aH (11))))1)))
AMa(MA
MS
(36)
(M
MSI)x
To (2
MSJ. M
AM
MS MA1. MAA...,,AAKII
TO (MAT T=1
MST)
3. From Equation (34) to Equation (35), this is similar to the transition from Policy Gradient Theorem to the general form of Policy Gradient Theorem (i.e., from traditional advantage function to GAE).
G USE OF LLMS
In this paper, the LLM used solely for correcting grammatical and lexical errors.
32

---

Published as a conference paper at ICLR 2026
H THE ALGORITHM WORKFLOW OF ARPO
In this section, we provide a detailed flowchart of the ARPO algorithm in diagram 1.
Algorithm Agentic Reinforce Policy Optimization
Input initial policy model T reward models ræi task prompts D; hyperparameters a, 3, p4, M, N, Z, k
1: policy model Te +- 8mm
2: for iteration 1..... I do
3: reference model f f- Tgg
4: for step 1, .... S do
5:
Sample a batch D, from D
6:
Update the old policy model Te <- Te Sample N reasoning 1-step paths fy Meaaa q) for each question E Dz
7:
8:
Compute initial entropy Himittaal of the first & tokens in each outputs let all rollouts {rollout,} < 1.N tool-call step f +
9:
10:
while Any unfinished y: do
12:
Parse unfinished Yr. execute tools and obtain results d; T(ss) Insert into rollout y; f y4 d,
14:
Generate & additional tokens based inserted yı compute step-level entropy Hip
Compute normalized change entropy AHi, = Normalize(H.,, - Hijinintiol)
for each Y
16:
Compute partial sampling probability P., = a + B AH if Pt,, > then
17:
18:
Branch out Z additional rollouts {ur } and add them to {rollout,)
[y4| M then
20:
Sample {y. until ys produce the final answer
21:
else
22:
Sample (1.) produce reasoning step
23:
t t- t ++1
24:
|i] < .M then
25:
Sample M - |uvl additional independent rollouts and add them to {rollout;)
26:
Compute rewards {rr for each sampled rollouts y! by running T
27:
Compute Å. for the f-th token of through group relative advantage estimation. for GRPO iteration I..... μ do
28:
29:
Update the policy model by maximizing the GRPO objective (Equation (3))
30:
Update r through continuous training using replay mechanism.
Output m0
33

---

Published as a conference paper at ICLR 2026
Table 6: An example from ARPO on GAIA dataset, with special symbols used in think content, search queries, Python codes, returned results and final answer highlighted with purple box blue hox and red box respectively.
green box
Example from GAIA Dataset
Question:
In terms of geographical distance between capital cities, which 2 countries the furthest from cach other within the ASEAN bloc according to wikipedia? Answer using a comma separated list, ordering the countries by alphabetical order.
Labeled Answer:
Indonesia, Myanmar
Model Output by ARPO(14B)
<think> Okay, so I need to figure out which two ASEAN member countries have their capital cities that are the furthest apart geographically. And then present them in alphabctical order, comma comma-separatee separated.. Let me start by recalling what ASEAN stands for-it's the Association of Southcast Asian Nations. The current members include Brunei, Cambodia, Indonesia, Laos, Malaysia, Myanmar (Burma), Philippines, Singapore, Thailand, Vietnam, and maybe others like East Timor Papua New Guinea? Wait, no, actually I think it's those ten countries. Let me confirm that first. (#+ reasoning contentare omitted here**s) First, let me gather the list of capitals and their coordinates. I should search for ÁSEAN member countries and their capitalsio ensure accuracy. Also, maybe check each country's Wikipedia page their capital's coordinates. Alternatiely, look up list that includes both names and coordinates. Let try scarch query first, </thinks List of ASEAN countries with capitals and geographic coordinates <lsearch>
<scarch>>
<result>
The searched web pages provide the list of ASEAN member countries along with their capitals but do include geographic coordinates for the capitals. For example: **Brunc***; Capital is **Bandar Seri Begawan** **Cambodia**, Capital is **Phnom Penh** *Inoonesia **; Capital is **Jakarta**
** **aao***; Capital Vientianee*** **Malaysia*** Capital is *KKala Lumpur ***Manmar*** Capital is " ""Naypyidaw*** **Phllippines*** Capital is **Manla*** *"inggppore*** * Capital is "inggppoe *"Thailand**: Capital is ** Bangkok**. **Vietmam** Capital is "* "*Hanoi**, To proceed with calculating distances between capitals, geographic coordinates (latitude and longitude) for each capital would need to be obtained from an external source.
</results
**eTwo search steps omitted here**s
<thinks] So yes, exactly 10 countries. Therefore, the number of pairs is C(10,2)=45. Computing all 45 isn't too bad for a script."**MMor reasoning contentare omitted here**s) Given all these calculations, I'm confident that the furthest pair is Indonesia and Myanmar. Their capitals Sorting alphabetically, Indonesia comes before Myanmar. Hence the answer should be Indonesia.Myanmen To determine the two ASEAN member countries with the furthest apart capitals, first identified the capitals and their geographic coordinates from reliable sources. Using the Haversine formula, I calculated the distances between all pairs of capitals. After performing these calculations, the maximum distance was found between ** Jakarta (Indonesia)** and **Naypydaw (Myanmar) When sorted alphabetically, the countries are ** Indonesia and **Myanaar*t, </thikk>
Jakarta and Naypyidaw respectively.
Indonesia, M Myanmar
clanswer>
I
CASE STUDY
34

---

Published as a conference paper at ICLR 2026
Table 7: An example from ARPO on HLE dataset, with special symbols used in think content, search queries, Python codes, returned results and final answer highlighted with purple box and red box respectively.
green box blue box
Example #2 from HLE Dataset
Question:
How long was the Second Great War in StarCraft Lore (in years, rounded up)
Labeled Answer:
1
Model Output by ARPO(14B)
<thinks Okay, I need figure out how long the Second Great War lasted in StarCraft lore, right? Let me start by recalling what know about StarCraft's 's timeline. More reasoning contentare omitted **r)***) So if starts in 2500 and ends 2506, that's seven years. But maybe the core conflict is counted until 2505, making it five years. Alternatively, some sources might consider the entire period from the first signs of the Zerg resurgence up the final battles as spanning six years. </think>
<search> Starcraft sttrrab>starcraft second great war duration years
</searchs
cresult>
The Second Great War in "StarCraft* is generally considered to have occurred from *+2504 to 2505** spanning **one year"*. This timeline aligns with the events of "StarCraft II: Wings of Liberty" (2504) and " "Heart of the Swarm* (2505). However, some sources, such as NamuWiki, specify that the war lasted for half a year until April 2505,5uggs the +*core conflict* may have been shorter (six months), though the broader war period is recorded one year. The followed the end of the Brood War (2500) and preceded the *Legacy of the Void campaign.
</results
** search steps omitted here**s
Okay, so the official StarCraft Wiki states it's one year. Even parts of were shorter, the overall duration is considered full year. Therefore, rounding up wouldn't change it since it's already a whole number. But just be thorough, maybe there's different interpretation. Let me see any other sources say Aibberiss.Alteraatiiee
<thinks
perhaps the Second Great War includes the events leading up to the main games. For example, if the war started in late 2503 and ended in mid-2505, that would be almost two years. But according to the wiki, it's 2504-2505. Given that the user specified rounded up.öven if it were, say, 1.5 years, we'd d round to 2. But based on the search results, especially the official wiki, it's one year. So the answer should be year. Wait, but the first search had conflicting info. Let recheck the exact wording from the wiki.Assuming the official timeline clearly states 2504-2505 as the Second Great War, that's exactly one year. Hence, the duration is one year. Rounded up, remains on! Another possibility: the Second Great War might refer to the conflict in the original StarCraft games versus the sequel. No, the Sccond Great War is definitely part of the StarCraft II storyline. Let me confirm once more. Yes, the Sccond Great War is indeed the central conflict of StarCraft II, occurring between 2504 2505, totaling one year. Some sources might mention core battle lasting six months, but the entire war period is year. <//tinkk>
Therefore, the answer is year. But wait, hold
<aus wer>
Since the question asks for the duration in years, rounded up, and it's exactly one year, rounding is needed.
clanswer>
35

---

Published as a conference paper at ICLR 2026
Table An example from ARPO on AIME24 dataset, with special symbols used in think content, search queries, Python codes, returned results and final answer highlighted with purple box box
blue box
and red box
respectively.
Example #1 from AIME24 Dataset
Question:
Let p least prime number which there exists a positive integer n such that v² + is divisible by p². Find the least positive integer such that m + 1 is divisible by p².
Labeled Answer:
110
Model Output by ARPO(8B)
To solve this problem, we need to find the smallest prime such that there exists positive integer n for which n - is divisible by p². Then, we need to find the smallest positive integer m such that m² + +-1 is start by checking small prime numbers one by one to find the smallest p that satisfies the condition. We'll use Python to automate this process. First, let's write function check if n + is divisible by p². We'll iterate over the first few the smallest prime number and move upwards until we find the required Here's the plan: 1. Iterate over the first few prime numbers. 2. For each prime p. check if there exists an n such that n² = (mod p²). 3. Once find the smallest prime that satisfies the condition, we'll find the smallest such that m +
<think>
divisible by p². We
of r find such an n for each prime p. We'll
with
(mod p²). Let's implement this in Python. </think>
<pythons>
from sympy import isprime, primerange
# Function to check if there exists an such that n + is divisible
by p
def find_n_for_p(p):
for n range(1, pa*2 1): if (n**4 + 1) (p**2) == 0:
return
return None
Iterate over the first few prime numbers to find the smallest primes list (primmeeern (2, 100))
# We 'l' start checking from the first few primes for in primes:
find_n_for_ppp)
if is not None:
it Now we need to find the smallest m such that m^4 + is divisible by p^2
for in range (1, p**2 1):
if (m*x4 1) (p**2) 0: result
break
else:
continue
break
print ((p, result))
</pythons
<thinks
The smallest prime for which there exists positive integer n such that n + is divisible by p²
17. The least positive integer such that m² is divisible by 17? is 110. Thus, the final answer
</tinks
wers
110
clunswer>
36

---

Published as a conference paper at ICLR 2026
Table 9: An example from ARPO on HLE dataset, with special symbols used in think content, search queries, Python codes, returned results and final answer highlighted with purple box and red box respectively.
box blue box
Example #1 from HotpotQA Dataset
Question:
Which playwright lived longer life, Edward Albee M. Barrie?
Labeled Answer:
Edward Franklin Albee
Model Output by ARPO(SB)
<think> need to determine which of two playwrights, Edward Albee and M. Barrie, lived a longer life. To do this, I will: 1. Find the birth date of Edward Albee. 2. Find the death date of Edward Albee. 3. Calculate
how long Edward Albee lived. 4. Find the birth date of J. M. Barrie. 5. Find the death date of M. Barrie. 6.
Calculate how long Barric lived. 7. Compare the lifespans. will web search for steps 1-5, -.t.ekiink>>
<search>>
Edward Albee birth death dates [chearch>>
cresult>
Page 1: Sep 16, 2016 Edward Albee was born Edward Harvey in Washington, D.C. At the of two weeks, Mrs. Reed Albee of Larchmont, New York, and renamed Edward Franklin Alhee III. From an early age....
was adopted by
Page 2: Explore Edward Albee's biography, personal life, family and cause death. Discover the real story, facts, and details of Edward Albce.
Page 3: Sep 17, 2016 Edward Albee was a Pulitzer Prize winning American playwright. His popular works include The Zoo Story, and Who's Afraid of Virginia Wolf? Albee was born Edward Harvey March 12, 1928 in Washington, D.C. (#**More search results omitted here
<think From the search results: Edward Alhee was born on March 12, 1928. EEward Albee died on September 2016. Now need find M. Barrie's birth and death dates, [aths,//thinkk> </think>
**oOne search step omitted *ere
<thinks
From the search results:- Barric was born on May 1860. J. M.. Barric died June 1937.
Now have both sets of dates:
Edward Albee: Born: March 12,
Died: September 16, 2016
J. M. Barrie: Born: May 9, 1860 Died: June 19, 1937
Next, calculate how long each playwright lived. For Edward Alhee: 2016 1928 years. For J.
Barrie: 1937 1860 77 years. So, Edward Albee lived 88 years and M. Barrie lived 77 years. Therefore, Edward Albee lived a longer life. </think>
<pythons
Calculate lifespan for Edward Albee hubee_birth_yeee = 1928
albee_death_year 2016
aibee_lifespan aabee_death_yeas (fintw""d Albee lived for (albee_lifespan years.
albbe_birth__ye
# Calculate lifespan for J. M. Barrie
barrie.birth_year 1860
barrie_death_.year 1937
barrie_lifespar
harrie_death_yee
harrie_birth_yee
print(f"J, M. Barrie lived for (barrie_lifespan) years.") # Compare lifespans
if albee_ libessln barrie_lifespan print Albee lived longer life.
else:
print ("J. M. Barrie lived a longer life.")
</python>
cresult>
Edward Albee lived for years. Barrie lived for 77 years. Edward Albee lived longer life.
</result>
canswer>
Edward Albee
<lanswer>
37


Reviews:
[{"review_id": "lsk0Pk24G0", "reviewer": "Reviewer_tzPk", "summary": "This paper proposes Agentic Reinforced Policy Optimization (ARPO), a reinforcement learning algorithm designed for multi-turn, tool-using large language model (LLM) agents. Unlike existing trajectory-level RL methods that sample entire reasoning paths, ARPO focuses on step-level exploration by leveraging the empirical observation that token entropy rises significantly after tool calls, reflecting uncertainty during reasoning. ARPO introduces two core components:\n\n(1) an entropy-based adaptive rollout mechanism that dynamically branches exploration when tool-call steps exhibit high entropy, enabling efficient step-level sampling;\n\n(2) Advantage Attribution Estimation, which assigns shared or distinct advantages to tokens depending on whether they belong to common or branched reasoning paths, allowing finer-grained credit assignment.\n\nA theoretical contribution, the Generalized Policy Gradient (GPG) Theorem, generalizes the policy gradient to macro-action rollouts, providing a solid foundation for partial sampling optimization.\nAcross 13 benchmarks covering mathematical reasoning, knowledge-intensive QA, and deep-search tasks, ARPO achieves 4–6% higher accuracy than baselines such as GRPO, DAPO, and Reinforce++, while requiring only half the tool-call budget, demonstrating both superior performance and efficiency.", "questions": "1.\tDoes adaptive branching ever lead to redundant exploration or inefficiency on simpler reasoning tasks?\n2.\tHow does Advantage Attribution Estimation perform under noisy or sparse rewards?\n3.\tWhat is the computational overhead of real-time entropy computation and partial rollouts compared to GRPO?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tStrong Motivation and Insight – The identification of entropy spikes following tool use provides a concrete empirical basis for adaptive exploration and directly informs ARPO’s design.\n2.\tInnovative Algorithmic Mechanism – The entropy-based adaptive rollout efficiently balances global and partial sampling, encouraging diverse tool-use behaviors without excessive sampling.\n3.\tRefined Credit Assignment – The Advantage Attribution Estimation effectively separates shared versus individual token updates, stabilizing training and improving interpretability.\n4.\tComprehensive Empirical Results – ARPO consistently surpasses trajectory-level baselines across multiple reasoning benchmarks and model backbones (Qwen, Llama), showing broad applicability.\n5.\tEfficiency and Scalability – ARPO achieves comparable or better results using half as many tool calls as GRPO, confirming both cost-effectiveness and computational scalability.\n6.\tSolid Theoretical Foundation – The GPG theorem formalizes ARPO’s macro-action rollouts, grounding its design in a generalizable reinforcement learning framework.", "weaknesses": "1.\tLimited Domain Generalization – All experiments are text-based; no multimodal or embodied environments are tested, restricting claims of general agentic applicability.\n2.\tHyperparameter Sensitivity – The algorithm depends on key parameters (entropy threshold τ, stability factor β), but no sensitivity or ablation study is provided.\n3.\tLack of Runtime Validation – While ARPO claims reduced rollout complexity (O(n log n)), there is no empirical runtime or resource cost analysis to verify this.\n4.\tReward Design Ambiguity – The hierarchical reward function (Eq. 5) combines multiple factors (accuracy, format, multi-tool usage) but lacks a detailed analysis of their relative influence.\n5.\tPotential Evaluation Bias – Reliance on LLM-as-a-judge scoring could introduce evaluation bias due to stylistic similarity between training and evaluation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oeOSeDVIrv", "reviewer": "Reviewer_tZJ2", "summary": "This paper introduces ARPO, a RL algorithm designed to improve LLM agents that use external tools in multi-turn interactions. The key novelty is an entropy-based adaptive rollout mechanism that dynamically performs additional sampling at high-entropy tool-call steps and an advantage attribution estimation method that distinguishes shared vs. branched trajectories. Experiments across 13 benchmarks (math, knowledge reasoning, and deep search tasks) show that ARPO improves accuracy and efficiency, reportedly achieving similar or better performance with half the tool-use budget compared to GRPO and other baselines.", "questions": "1. The paper’s motivation relies on the observation that token entropy tends to increase after tool-call steps, which is claimed to indicate model uncertainty. Could the authors provide quantitative statistics (e.g., average entropy changes, variance, and significance tests) across multiple datasets and trajectories to confirm that this pattern consistently holds?\n2. Since ARPO is designed to mitigate high-entropy uncertainty, it would be important to show entropy evolution before and after training. Do the authors have measurements or visualizations demonstrating that entropy indeed decreases or stabilizes after ARPO training? \n3. Did the authors observe any instability or divergence issues during training due to dynamic branching?\n4. It would be helpful to include a PPO baseline or comparison, given that PPO is a standard RL method.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The evaluation covers a wide range of tasks (reasoning, knowledge retrieval, deep search), showing consistent improvements over multiple baselines (GRPO, DAPO, Reinforce++).\n- ARPO achieves better or comparable results with roughly half the tool-call budget, which is practically valuable given the high cost of tool-based RL.\n- The GPG theorem provides a solid conceptual framework connecting macro-step rollouts to standard policy gradients, offering a theoretical foundation.\n- The paper is well-organized, with clear modular explanations of the rollout mechanism and advantage estimation, supported by informative figures.", "weaknesses": "- The paper’s central motivation that token entropy reliably increases after tool calls is not convincingly demonstrated. Figures 1 and 2 appear anecdotal, with unclear statistical support, and may rely on a single sample.\n- While ARPO’s adaptive rollout is designed to mitigate high-entropy uncertainty, the paper does not show post-training token entropy patterns to demonstrate that the method actually reduces or stabilizes entropy. Without before-and-after comparisons, it’s unclear whether ARPO genuinely controls model uncertainty or simply improves performance indirectly.\n- The experiments only report numerical performance gains, without showing any entropy-related observations or analyses during or after training. This creates a disconnect between the paper’s motivation entropy-guided exploration and its empirical validation, weakening the causal link between the proposed mechanism and the observed improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "38NlsL7w9B", "reviewer": "Reviewer_BZ6R", "summary": "This paper presents a sampling strategy, which generates additional trajectories by forking the trajectory at timesteps where the tool call probabilities indicate high entropy, and use this strategy as part of the rollout stage for multi-turn agentic RL training for LLMs. The presented method (ARPO) provides consistent improvement in a variety of tasks (math, coding, deep search) and highlights the importance of further research in exploration within the area of RLVR for LLMs.", "questions": "1. Could the authors quantify the effects of varying $\\alpha$, $\\beta$, and $\\tau$ during sampling? The entropy figure in the Appendix indicates that if entropy is too high performance degrades. Thus, understanding the relationship between base model probs and entropy during sampling seems crucial.\n2. Are there other actions, besides high entropy tool calls, where branching is effective? The authors seem to only branch at high-entropy tool calling steps, but I wonder if this method is generally applicable to reasoning steps too.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Paper is well motivated and the observation of high entropy tool calling steps is clearly presented. \n- Consistent improvements in evaluated domains (math, coding, deep search).\n- Mathematical formulation is grounded and highlights the efficacy of their sampling approach.", "weaknesses": "- Limited Novelty: The main contribution seems to be the additional sampling introduced by branching from high entropy actions. The learning objective is simply multi-turn GRPO given rollouts generated by their sampling mechanism. \n- Effects of branching sampling ($P_{t}$) is unclear. There are no ablations documenting the effects of relying on the base probability ($\\alpha$) versus relying on the entropy differential ($\\beta$) or the branching cutoff ($\\tau$).\n- Unclear if the effectiveness of high-entropy branching is unique to tool-calling steps or broadly applicable (e.g. to text-based reasoning steps)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)