Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
Authors: Guoqing Wang
Sunhao Dai
Guangze Ye
Zeyu Gan
Wei Yao
Yong Deng
Xiaofeng Wu
Zhenzhe Ying
Venue: ICLR 2026 Poster (2026)
Keywords: Turn-Level Reward
Search Agent
Agentic RL
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose IGPO, a simple and effective reinforcement learning framework with turn-level reward for training multi-turn search agents.

Abstract: Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments through tool use, particularly in search-based settings that require multi-turn reasoning and knowledge acquisition. However, existing approaches typically rely on outcome-based rewards that are only provided exclusively upon generating the final answer. This reward sparsity becomes particularly problematic in multi-turn settings, where long trajectories exacerbate three critical issues: (i) advantage collapse, where all rollouts receive identical rewards and provide no useful learning signals; (ii) lack of fine-grained credit assignment, where the correctness of intermediate turns is obscured, especially in long-horizon tasks; and (iii) poor sample efficiency, where each rollout yields only a single outcome signal, leading to low data utilization. In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training. IGPO models each interaction turn as an incremental process of acquiring information about the ground truth, and defines turn-level rewards as the marginal increase in the policy's probability of producing the correct answer.  Unlike prior process-level reward approaches that depend on external reward models or costly Monte Carlo estimation, IGPO derives intrinsic rewards directly from the model's own belief updates. These intrinsic turn-level rewards are combined with outcome-level supervision to form dense reward signals. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that IGPO consistently outperforms strong baselines in multi-turn scenarios, achieving higher accuracy and improved data efficiency. Our code is available at https://github.com/GuoqingWang1/IGPO.

Published as a conference paper at ICLR 2026
INFORMATION GAIN-BASED POLICY OPTIMIZATION: A SIMPLE AND EFFECTIVE APPROACH FOR MULTI- TURN SEARCH AGENTS
Guoqing Wang" Sunhao Dail 1,, Guangze Ye" Zeyu Gan', Wei Yao?, Yong Deng
Xiaofeng Wu', Zhenzhe Ying!
Venus Team, Ant Group RRnmin University of China 3 Individual Author
(guoqiingwang905, sunhaodai, guangzeye98/ @@mailg.m....m com zyganêruc edu. [dengyong, deng, congyu.wxE, zhenzhe, featteantgouup.coom com
rrr wei wei.yao@ruc. edu, on deng,
ABSTRACT
Large language model (LLMM)based agents are increasingly trained with rein- forcement learning (RL) enhance their ability to interact with external environ- ments through tool use, particularly in search-based settings that require multi-turn reasoning and knowledge acquisition. However, existing approaches typically rely outcome-based rewards that are only provided exclusively upon generating the final answer. This reward sparsity becomes particularly problematic multi-turn settings, where long trajectories exacerbate three critical issues: (i) advantage col- lapse, where all rollouts receive identical rewards and provide no useful learning signals; (ii) lack of fine-grained credit assignment, where the correctness of inter- mediate turns is obscured, especially in long-horizon tasks; and (iii) poor sample efficiency, where each rollout yields only a single outcome signal, leading to low data utilization. In this paper, we propose Information Gain-based Policy Opti- mization (IGPO), simple yet effective RL framework that provides dense and intrinsic supervision for multi-tur agent training. IGPO models each interaction turn as an incremental process of acquiring information about the ground truth, and defines turn-level rewards as the marginal increase in the policy's 's probability of producing the correct answer. Unlike prior process-level reward approaches that depend external reward models costly Monte Carlo estimation, IGPO derives intrinsic rewards directly from the model's own belief updates. These intrinsic turn-level rewards are combined with outcome-level supervision to form dense re- ward signals. Extensive experiments on both in-domain and out-of -ft-o---mmminn bench- marks demonstrate that IGPO consistently outperforms strong baselines in multi- turn scenarios, achieving higher accuracy and improved data efficiency. Our code is available httpps: //itts::///ghhbbbbbb comm//uoccngg////g//////C
INTRODUCTION
Large language model (LLM)-basee agents are increasingly equipped with the ability to interact with external environments through tool use (Zhang et al.) 2025a Huang et al.) 2025 Li et al., 2025c), a capability often regarded as critical step toward building general-purposs autonomous intelligent systems (Gutierrez et al.] 2023 Qu et al. 2025). One of the most prominent application scenarios is agentic search, where an agent invokes tools such web search (Zhang et al. 20256) Qi et al. 2024) to access up-to-date, large-scale knowledge that substantially enhances its ability to solve complex, knowledge-intensinn tasks (Ning al. 2025). Through iterative interaction with the external environment via such tools, search agents can gradually acquire missing information and refine their reasoning trajectories toward solving the target query.
To equip general-pune purpose LLMs with such agentic search capabilities, early efforts primarily re- lied on prompt-basen based workflows (Li et al. 2025b: Wang al. 2024a| Zheng et al.) 2024). which
"qqual contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
allowed tool use without additional training but often suffered from poor generalization. More cent studies have explored supervised ane-tuning (SFT) (Wang et al. 2024b) and reinforeement learning (RL) (Jin et al.) 2025 Song et al. 2025a Zheng et al., 2025b) to explicitly incentivize tool use, achieving markedly better performance. In particular, Group Relative Policy Optimiza- tion (GRPO) (Shao et al.) 2024)- methods have emerged the dominant approach for training agentic LLMs. In this paradigm, a group of rollouts is generated for each query under the current policy, and outcome-based rewards, typically defined by the correctness of the final answer against the ground truth, are used to construct group-relative advantages that drive policy optimization.
Despite their simplicity and effectiivnness in relatively easy tasks, outcome rewards suffer from inherent sparsity (Zhang et al., (2025c), providing supervision exclusively at the final answer. This limitation particularly detrimental in multi-turn agentic settings, where long trajectoooo exacerbate three critical issues. First, outcome-only rewards frequently lead to advantage col- lapse: when intra-group rollouts yield identical answers (e.g. uniformly correct or incorrect), they receive identical rewards, yielding zero group-relative advantages and no gradient signal. As shown in Figure I a substantial portion of training iterations suffer from this issue, especially for smaller models, which struggle more with complex queries. Second, outcome supervision fails to provide fine- grained credit assignment. In multi-turn scenarios, later turns are tightly dependent on earlier ones: the action of the current turn may be correct but rendered useless by prior mistakes, or conversely, early successes may be negated by subsequent errors. Thus, distinguishing the correctness of intermediate turns is essential in this sce- nario, but outcome rewards tend to blur such specific correctness. Third, outcome reward sparsity results in poor sample efficiency. Since the entire trajectory receives only single terminal signal, dense intermediate information is wasted, requiring significantly more samples to learn effective policies.
trajectories
Ratie
0.32
GRRD
GRA0-30
Zero Advantage
0.24
KSPO PP00-2775
0.16
00008
10000
20
40
60
80
100
120 140
Training Steps
Figure 1: Proportion of zero-advantage groups during training-IGPO vs. GRPO on Qween255-7B/BB-Insttu Instruc..
Recent approaches have attempted to mitigate these issues by introducing process rewards. One line of work leverages external oracle knowledge or reward models to judge intermediate steps (Wang jet al.
2025 Feng et al. 2025), but this strategy is costly and risks introducing bias. Another line relies on Monte Carlo simulations to estimate step values (Wang et al.) 2023] Zuo et al.) [025] Zhang et al. 2025c), yet these methods suffer from high variance unless a large number of samples are collected. Overall, both directions face challenges in scalability and fail to provide simple and stable supervision, underscoring the need for an intrinsic and reliable process reward design.
To address these challenges, we propose Information-Gaiinbeee simple yet effective RL framework that provides stable and intrinsic supervision for multi-turn search agent training. The key intuition is to model each agent- enveroenviro interaction turn as an incremental process of acquiring information about the ground truth. Specifically, at every turn, IGPO computes the policy's probability of producing the correct answer and defines the turn-level reward as the marginal increase in this probability compared to the previous state. This information gain reward offers souund-rruth-wwar feedback at every turn, in contrast to outcome rewards that only supervise the final answer. However, because outcome rewards are still necessary to anchor learning to the final objective, IGPO incorporates them alongside the turn-level rewards to construct dense reward signals for each rollout. To further stabilize training, we normalize the information gain rewards and outcome rewards separately within groups and propagate them with discounted accumulation, enabling the computation of turn-level discounted returns that capture long-horizon dependencies. Finally, IGPO optimizes the policy with GRPO-style surrogate objective, replac- ing rollout-level advantages with our turn-level discounted returns. Additionally, we introduce a vectorized implementation to minimize the computational overhead of information gain rewards.
Policy Optimization (IGPO), a
To evaluate the effectiveness of IGPO, we conduct extensive experiments on both in-domain and out- of-domain benchmarks with search-based agents. Results show that IGPO consistently outperforms strong baselines, delivering substantial gains in both answer accuracy and sample efficiency. Our main contributions can be summarized as follows: (1) We analyze the phenomenon of advantage collapse in outcome reward-based optimization, and reveal the inefficiency of existing process- level rewards due to reliance on external knowledge or high-variance estimation. (2) We propose IGPO, simple yet effective policy optimization framework that leverages turn-level information
2

---

Published as a conference paper at ICLR 2026
gain to provide dense, ground-truth-awen supervision with negligible computational overhead. (3) Comprehensive experiments demonstrate that IGPO outperforms strong baselines across multiple benchmarks and significantly improves sample efficiency, especially for smaller models.
PRELIMINARIES
2.1 TASK FORMULATION
Let D = {(g, a)} denote dataset of quesstinnanswer pairs, and let & represent external tool (e.g.. a web search engine). The goal of the agent is to solve question by generating a rollout through iterative interaction with the environment via tool &, where T is the total number of interaction turns. The last turn Ty is the answer lurn that outputs a rationale-then- final answer sequence, while all previous fort < T. each turn is defined as triple consisting of [think], [tool call], and [tool response The [think] step compels the agent to reason explicitly before acting, and each rea- soning process is wrapped in <thi fechinnk</thnnn al. 2025). The [tool call] step invokes the external tool & by producing a structured re- quest, typically JSON-tormatted and wrapped in dedicated tag (e.g., <tool.call 1>llsss aueryy/tcol.calll
0 (Ti, 72:....... TT)
involve reasoning and tool interaction. Specifically,
tag following the DeepSeek- R1 setting (Guo
The [tool response) step then returns structured outputs from &, such webpage snippets with titles, URLs, and text when using web search engine tool, enclosed in <tool response> wetssereetriveee after a [think] step, the agent generates its answer within the <answer>< //as//// tag, and this content is extracted as the trajectory's ' final prediction à, which is expected to correctly address the input query q. This agvnt-environment interaction is illustrated at the bottom of Figure 2
documents</too
response> tags. In the final turn,
2.2 AGENTIC REINFORCEMENT LEARNING PIPELINE
Policy Optimization. Agentic RL typically adopts policy-gradient methods to optimize the agent policy #o- A common approach is Group Relative Policy Optimization (GRPO) (Shao al. 2024). which removes the need for an explicit critic by normalizing returns within each sampled group of rollouts. Formally, given an actor model Ne, group of G rollouts J is sampled from old policy Te q) for each input (g, a) D. The policy is then optimized by maximizing the clipped surrogate objective with KL regularization:
G
10:
To(oi, 0 0i,ce) A; To
JGRPO (0) = (R(g.))D (o
min
G
0i
(0i,t 9, 0i,tt)
1=1
(I)
Te(or,c g. 0,,0)
clip
1 e, 1+e) A, DKLL(TE Tret)
TBaa (oi.
q. Oi,e))
where Ā; Pa vo) is the normalized group-relative advantage for the i-th rollout and st .ra) Ty is the outcome reward of the i-th rollout. the clipping ratio, and controls the KL penalty relative to the reference model Tef. During optimization, gradients are applied only to decision tokens (reasoning, tool calls, answers), while tool responses from the environment are masked out.
Reward. During training, the agent receives scalar reward for each rollout o, which provides the optimization signal. Prior work typically combines an outcome reward with format penalty:
a1
2|ana| |a|taa
E
[0, 1], if the output is in valid format,
Ximes
otherwise,
where à is the predicted answer, a the ground-truth answer, and FI(2, E [o, 1] denotes the word-level Fl score. If the output violates the required schema (e.g., missing tags malformed JSON), a negative constant Arm < 0 assigned as a penalty. Thus, the outcome reward provides a correctness signal, while the format penalty enforces the structural validity of outputs.
3

---

Published as a conference paper at ICLR 2026
Interaction
Rollowt
Thre
logs?
Information
Ture-Level Discouated
Immediste Reward
R
Policy
Anige
Jalle Oaix ST
(Ananner omitted)
Tarn
Anppar
Groand Trash
tegp!
Anfa Gain GT
Graas
Amrgar Ro
Ulerry
Info
delge To
Gate
Knvironment
0:
Turx
Ouger
Duyar
Angp!
Joss
Rollout
[nink]
jost call]
Jteol reggonse
[mhnk]]
[boat call)
[teol regonsss
[think]]
[answert
with Jonc
Figure 2: The training pipeline of IGPO. (Upper) Turn-level information gain rewards are computed by measuring changes in ground-truit truth probability and combined with the outcome reward derive discounted returns. (Lower) Each rollout contains at most T -1 interaction turns, where each turn includes a reasoning step, a tool call, and the returned tool response, followed by a final answer turn. During optimization, the loss tool response is masked out.
INFORMATION GAIN-BASED POLICY OPTIMIZATION
In this section, we first illustrate our motivation and then provide a detailed introduction to our pro- posed information gain-based policy optimization, whose overall framework is shown in Figure 2
3.1 MOTIVATION
While outcome-based RL is effective in single-turn tasks, extending it multi-turn agentic settings faces three critical limitations. First, standard GRPO leads to advantage collapse. In the standard framework (Eq. [ each rollout о; receives a scalar reward derived solely from the final answer. For complex (or trivial) queries, rollouts often yield identical outcomes (uniformly zero or one), causing group-relative advantages to vanish and providing no valid gradient signal. Second, outcome-only supervision lacks tineggriieed credit assignment. In multi-tun -urn scenarios, later decisions strictly de- pend on earlier ones: a tool call may be conceptually correct but rendered useless by prior errors, or conversely, valid reasoning may be overshadowed by subsequent mistakes. Outcome rewards obscure these dependencies, failing to distinguish productive steps from invalid ones. Third, out- come reward sparsity results in poor sample efficiency. By relying solely single terminal signal, the dense semantic information embedded in intermediate reasoning and tool interactions is wasted, necessitating significantly more samples learn effective policies
To mitigate this issue, we introduce Information-Gaie based Policy Optimization (IGPO). The key idea is to exploit the multi-turn structure of agentic rollouts and treat each turn opportunity to acquire additional evidence toward the ground truth. At every turn, IGPO measures the increase in the policy's confidence of generating the correct answer, which we defined the information gain of this turn and uses this as the turn-level reward. By rewarding turn-level information gain, IGPO supplies denser and more fine- -reeggraieed supervision, especially at early training stages. We further present theoretical analysis in [Appendix A which intuitively explains why IGPO effectively ad- dresses the limitations of sparse outcome rewards in multi-turn scenarios. Since the information gain is defined with respect to the ground-truth answer and computed under teacher forcing, it provides rich and dense supervision, ensuring that every sample contributes to learning even when no rollout produces fully correct final answer.
3.2 INFORMATION GAIN-BASEI TURN-LEVEL REWARD
Information Gain Reward. We view multi-ture turi agent- agent-environe environment interaction process of incrementally acquiring information about the ground truth. To capture this intuition, we propose an intrinsic information gain-based reward. At each turn, we evaluate the policy's probability of
4

---

Published as a conference paper at ICLR 2026
generating the ground-truth answer and define the reward the difference between consecutive states. We call this the information gain reward, as it measures the marginal increase in posterior probability mass assigned to the ground truth induced by the current turn. In practice, to ensure numerical stability, we quantify this gain as the increment in log probability.
Formally, let (a....... a..) denote the ground-truth answer tokens. For the t-th turn in the i-th rollout, the log probability under the current policy ve is computed as
a1)
log (a (i,iss))
log To(aj) 05,ge, acj);
L
where gi denotes the prefix of rollout o; up turn L. Then the immediate reward for turn / is
IG i,t = (G g, or.)) = log ma y, oi log (a (1,s)))));
1SI<T.
(4)
In practice, the ground-truth answer is wrapped in the same schema a predicted answer to ensure consistency with rollout formatting, e.g., <think>>ee to /nswer</chinkxxxassseeeeo
there's s enough information Truth o</answer>.
This tern-level reward offers three key properties: (1) Ground truth awareness: It increases when the turn-action raises the policy's s confidence the correct answer, and decreases otherwise. Crucially, this objective derivation minimizes the potential bias inherent in external reward models or manual process labeling. (2) Dense supervision: provides turn-level dense signals that resolve advantage collapse, enable fine-graines credit assignment, and improve sample efficiency. (3) Computational efficiency: Unlike other process reward designs (especially Monte Carlo estimation), incurs negli- gible overhead, advantage further amplified by the vectorized implementation presented below
Efficient Vectorized Implementation. Although the information gain reward already enjoys inherent effi- ciency over existing process reward methods, we seek further optimize its computation. In the standard implementation, computing the information gain re- ward requires T separate forward passes, with com- plexity scaling as Et--1 T.Z. where I, is the sequence length at turn t. To optimize this, propose vec- torized implementation that appends T copics of the ground-truth answer to the end of the trajectory. By applying a custom attention mask (Figure 3) that re- stricts each copy's ' visibility to its corresponding turn, we compute the log probabilities for all T turns si- multaneously in single forward pass (complexity x 1) while ensuring mathematical equivalence to Eq. Given that the ground-truit length orders of magnitude smaller than the full reasoning trajectory. the overhead introduced by these appended copies is negligible. This implementation yields an asymptotic speedup of z T/3 (e.g., for T 10) and further enhances GPU utilization by reducing synchronization overhead.
Original Atieation
Attontion
Tamn
Tran
117
Figure 3: The custom attenyion mask for vectorized implementation. Shaded cells denote allowed attention. Prompt and an- swer tokens are omitted for clarity.
3.3 POLICY OPTIMIZATION WITH TURN-LEVEL DISCOUNTED RETURN
Turn-Level Discounted Return. Given a group of rollouts (o. where each rollout o, yields
sequence of T information gain rewards {re,t)r=) and an outcome reward 7 (calculated via 2). we follow GRPO (Shao et al. (2224) to stabilize training and capture the relative magnitude of rewards. Specifically, we perform group-wise z-normalization on the information gain rewards and outcome rewards separately.
J
pio diG
1St<T,
(5)
Hi t=
Due its log-prob origin, apply hoo-ggrddeent the information based reward.
5

---

Published as a conference paper at ICLR 2026
where Mag, dig are the mean and standard deviation of all information gain rewards (ri,t) within the group, and μlo do correspond to the outcome rewards {-
While Pi, captures the relative quality of each turn, it only reflects immediate effects and ignores the impact of current decisions on future turns. To incorporate such long- dependencies, we compute a turn-level discounted return R, to reflect the cumulative impact of future rewards:
T R ETm-PPa
(6)
where 7 E (0, 1] the discount factor. During optimization, Rs, is assigned to all tokens produced in turn of rollout o,, This yields dense and future-aware supervision signal for policy learning.
Policy Optimization. With the turn-level discounted return R., defined above, we optimize the agent policy using a clipped surrogate objective with KI. regularization, following the same structure as GRPO but with finer-grained credit assignment. Formally, the IGPO objective is
(o q. 01.e)) R
(O) ((g,))>,, (or Tu (-14)
min
G
0i
Thu (oi, 9, 01,00)
To(0 q, 01,0t)
clip
1 1t)) Ra,, - DKI Tree)
Te (0i,1 q, q,01,<1)
where c is the clipping threshold, 3 controls the KL penalty strength, and t maps token o,, to its originating turn. During optimization, only decision tokens (reasoning, tool calls, and answers) receive gradient updates, while raw tool responses are masked out.
To further substantiate the simplicity and implementability of the proposed IGPO, provide an algorithmic flow comparison between IGPO and GRPO in Appendix G
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets & Metrics. To evaluate the effectiveness of our proposed IGPO, we conduct experiments on both in-domain (ID) and out-of-domain (OOD) QA benchmarks an agentic search setting. Following previous work (Zheng et al. 2025b Deng et al. 22025). the ID setting includes four widely
used datasets: NQ (Kwiatkowski et al. 2019). TQ (Joshi et al. 2017). HotpotQA (Yang ct al. (018)) and 2Wiki (Ho al. 20200)) while the OOD setting includes three datasets: MusiQue (Trivedi al.,
2022]. Bamboogle (Press et al., 2022), and PopQA (Mallen et al.) 2022). We report word-level
as the evaluation metrie, which is computed the harmonic mean of precision and recall between
the predicted and reference answers.
Baselines. To directly verify IGPO's superiority agentic search tasks, we compare it against set of competitive basclines: (1) Prompt-based methods: CoT (Wei et al. 2022). CoT+RAG (Gao et al,, 2023 and Search-ol (Li al. 2025b), which represent the bascline performance of LLMs without Turther training on search tasks. (2) Outcome-rewaad RL-based methods: Search-r -basee/tsttteetttutttt
(Jin et al. (025). RI Rl-searr (Song et al., 2025a), and DeepResearcher (Zheng al.) 2025b), the representative search agents with outcome- based reward RL, yielding marked performance gains. (Wang et al., (025), ReasoningRAG (hang et al. 2025c). and GiGPO (Feng et al., 2025), which are the latest approaches exploring step-reward RL in search-agent settings.
(3) Step-reward RL-based methods: SeepSearch-base/msiiee
To further validate IGPO's efffcctiveness. also compare it against the following commonly used RL algorithms under the same configuration: PPO (Schulman et al. 2017), a widely used actor-critic algorithm that requires an additional value model, and critic-free methods Reinforce++ (Hu 2025)
RLOO (oool et al. 2019 Ahmadian et al., 2024), GRPO (Shao et al.) 2024). and ASPOOZhen. al. 2025a) which perform advantage estimation over trajectory groups batchs.
Implementation Details. We use 1ren2.5-7B-Instruue (Qwen et al. 2025) as our backbone model.
The training is conducted using the verl (Sheng et al. 2025) framework. The discounted factor ? is

---

Published as a conference paper at ICLR 2026
Table I: Main results of IGPO compared with different agentic RL baselines across seven datasets.
In-domain
Out-of-domain
Method
NQ
TQ
HotpoiQA
2Wiki
Musique
Bamboogle
PopQA
Avg.
Prompt-hased
CoT
19.8
45.6
24.4
26.4
8.5
22.1
17.0
23.4
CoT+RAG
42.0
68.9
37.1
24.4
10.0
25.4
46.9
36.4
Search-ol
32.4
58.9
33.0
30.9
14.7
46.6
38.3
36.4
Ouwadde-reward
RL-based
Search-rl-base
45,4
71.9
55.9
44.6
26.7
56.5
43.2
49.2
Search-rl-instruct
33.1
44.7
45.7
43.4
26.5
45.0
43.0
40.2
RI-searcher
35.4
73.1
44.8
59.4
22.8
64.8
42.7
49.0
DeepResearcher
39.6
78.4
52.8
59.7
27.1
71.0
48.5
53.9
Step-reward RL-based
StepSearch-bar
49.3
45.0
32.4
57.3
46.0
StepSearch-instrucl
50.2
43.1
31.2
53.4
44.5
ReasoningRAG
48.9
50.4
20.6
45.5
46.2
42.3
GiGPO IGPO
46.4
64.7
41.6
43.6
18.9
68.9
46.1
47.2
46.4 80.6
59.0
72.1
32.7
77.0
53.8
60.2
set to 1.0 with no future tuning. At each training step, we sample 32 prompts, and sample 16 rollouts for each prompt. The maximum dialogue turms are to 10. For the environment, we use the google search API as our tool. The settings of our experiments are consistent with DeepResearcher (Zheng al. 2025b). For the other baselines [Table 1 we directly copy their reported results. All RI. training methods (including ours and the baselines) use exactly the same hyperparameter config- urations. The training and inference prompt templates are shown Appendix H Please refer to Appendix C for more details.
4.2 OVERALL PERFORMANCE
The overall performance comparison between IGPO and the baseline methods is presented in Table and Table Based these results, we can draw the following key observations:
Training-bes based methods consistently outperform prompt-based baselines. As shown in Table I all RL-based methods, whether outcome- step-reward driven, achieve substantially higher perfor- mance than all prompt-based approaches. This confirms that explicit policy optimization essential for developing effective LLM-based agents, as opposed to relying prompting alone.
Existing step-reward methods yield competitive but unstable improvements compared to outcome-reward RL methods. While step-reward baselines occasionally surpass outcome-reward ones on specific datasets (c.g., StepSearch on Musique), their overall performance still lags behind the strongest outcome-reward methods such DeepResearcher. This suggests that existing step- reward designs, although able to provide intermediate guidance, often suffer from noisy or weak supervision signals that limit their generalizability.
IGPO achieves the best overall performance across both in-domain and out-of-domain datasets. Our IGPO outperforms baselines. with an average score 60.2, a clear margin over the best method (+6.3 over DeepResearcher). This improvement is attributed to IGPO's informa- tion gain-based reward design, which assigns intrinsic, ground-truth-awan credit at every turn while preserving the outcome reward. By providing fine-grained credit assignment and improves sample efficiency, IGPO delivers robust gains across both in-domain and out-of-domain datasets.
IGPO consistently outperforms other RL algorithms. Beyond task-specific baselines, Table 2 shows that IGPO also achieves the highest overall score among standard RL methods, surpassing RLOO, PPO. Reinforcet+, and GSPO. Unlike these methods, which rely solely sparse outcome rewards, IGPO incorporates turn-level discounted returns to provide denser and more stable super- vision, leading stronger generalization and more efficient training.

---

Published as a conference paper at ICLR 2026
Table 2: Main results of IGPO compared with different RL baselines across seven datasets.
In-domain
Out-of-domain
Method
NQ
TQ
HotpoiQA
2Wiki
Musique
Bamboogle
PopQA
Avg.
RLOO
40.7
72.5
49.6
55.0
24.8
62.2
43.1
49.7
PPO
38.7
75.4
48.6
59.7
26.2
63.4
48.7
51.5
GRPO
40.3
77.0
48.9
57.7
25.0
65.1
49.6
51.9
Reinforce++
34.3
67.5
45.9
54.5
23.7
61.2
44.3
47.3
GSPO
41.5
77.7
46.3
60.1
25.4
67.6
45.4
52.0
IGPO
46.4
80.6
59.0
72.1
32.7
77.0
53.8
60.2
4.3
ABLATION STUDY
We further conduct ablation experiments to assess the contribution different reward components. As shown in Table 3
observe:
First, using only information gain (IG) turn-level reward or only outcome reward (F1) yields clearly inferior results compared to the full combination. This highlights the complementary roles of turn-level and outcome-level supervision: the outcome reward enforces alignment with the final task objective but suffers from severe sparsity, whereas the information gain reward offers dense and stable guidance for intermediate steps.
Second, IGPO with only IG achieves performance comparable standard GRPO (i.e., IGPO w/ F1). This demonstrates that IGPO's information gain reward is not subject to reward hacking. Usually, without outcome supervision, unstable reward designs would quickly collapse. In contrast, our IGPO remains robust because its turn-level signals are intrinsically defined and grounded in the ground truth.
even exceeding that of
Table Ablation results IGPO on Dwven..5-BB/BB-
with different reward designs. IGPO (w/ F1) corresponds to using only outcome rewards, reducing to standard GRPO.
In-domain
Out-of-domain
Method
NQ
TQ
HotpotQA
2Wiki
Musique
Bamboogle
PopQA
Avg.
Qwen2.5-3B-Inste
IGPO (w/ FI)
31.0
55.6
27.5
29.4
12.1
35.7
34.9
32.3
IGPO (w/ IG)
29.6
54.1
28.1
37.5
17.6
43.8
31.7
34.6
IGPO (w/ FI+IG)
41.9
69.2
47.8
51.4
24.8
58.4
49.0
48.9
Qwen2.5-7B-Instree
IGPO (w/ FI)
40.3
77.0
48.9
57.7
25.0
65.1
49.6
51.9
IGPO (w/ IG)
37.3
75.2
52.1
63.3
28.9
69.8
47.8
53.5
IGPO (w/ FI+IG)
46.4
80.6
59.0
72.1
32.7
77.0
53.8
60.2
Third, the improvements are particularly pronounced on the smaller 3B model. Compared to standard GRPO, IGPO improves the 3B model by +16.6 points (32.3 48.9) and the 7B model by +8.3 points (51.9 60.2). This larger benefit on the 3B model arises because advantage collapse is more severe for weaker models that struggle directly produce correct answers (Figure 1}, making them especially reliant on dense reward signals. In such cases, the information gain reward helps prune noisy reasoning paths and reinforce rollouts that progressively approach the ground truth.
Finally, IGPO demonstrates consistently faster and more stable learning dynamics. As shown in Figure 4 IGPO steadily outperforms its two ablated variants throughout training across all seven datasets. The curves highlight two advantages: (i) IGPO converges to higher Fl scores, confirming the benefit of combining intrinsic turn-level reward and outcome rewards, and (ii) IGPO maintains stable improvements
steps, indicating robustness against reward sparsity and noisy supervi- sion. These results further validate that IGPO provides dense and reliable training signals, thereby improving both training efficiency and final performance.
In addition to the reward ablation, we compare different information gain bases (probability vs. probabbbbbllyy and normalization strategies (joint separate) in Appendix D.I
8

---

Published as a conference paper at ICLR 2026
0.80
0.00
Score S40
IDPO ner
ICPO oed F1
ICPO ner +
ISND PL+
IOPO
50
25 25
NQ
TQ
(c) HotpotQA
(d) 2Wiki
F2 13.5are
Stare
0.50
0.10
(w77L)
IGTO nel
(w)
Training Staps
90
20
Steps
Training Steps
22
90
(c) Musique
(f) Bamboogic
PopQA
Figure 4: Training curves on Qwen2.5-7BB 3-77--nnstrrctt with different reward designs.
4.4
IN-DEPTH ANALYSIS
IGPO
GRPO
F1 Score
3.0
0.48
E1
1.5
0.400
IGPO
0.0
0.32
GRPO
10
20
30
40
50
10
30
50
60
70
Steps
(Million)
Figure 5: Mean reduction in ground-truth swer entropy from the initial query (Turn 0) to the last non-answer turn (T' 1) during training.
Figure 6: Token Efficiency: average perfor- mance with respect to the number of tokens used for gradient updates.
Ground-truth Entropy Reduction. To better understand how IGPO improves training dynamics, measure the change in ground-trutte answer entropy from the initial query (Tum 0) to the last non-answer turn (T' 1). As shown in Figure IGPO consistently achieves a larger entropy re- duction than GRPO throughout training. This indicates that the information gain reward effectively encourages intermediate steps to move the policy closer to the ground-truth answer distribution. In contrast, outcome-based supervision in GRPO provides guidance for intermediate turns, resulting in weaker entropy reduction. These results highlight that IGPO's turn-level supervision translates into confident and grounded reasoning trajectories.
Token Efficiency. We further compare IGPO and GRPO in terms of token efficiency, i.e., the performance improvement per token used for gradient updates. As shown in Figure 6) performance increases more rapidly under IGPO, and the gap over GRPO widens as training progresses. other words, IGPO achieves stronger performance with fewer tokens, indicating that its turn-level rewards deliver denser and more informative gradients than outcome-only supervision, thereby effectively addressing the issue of poor sample efficiency. This finding is consistent with the training dynamics observed in |Figure 4 where IGPO only converges faster but also maintains stable advantage throughout optimization. Such improvements in token efficiency are particularly valuable in agentic RI., where training data is scarce and expensive to obtain, making efficient use of every gradient update critical factor for scaling.
The additional experimental analysis is provided in Appendix D in particular, the analysis regarding spurious correlations is presented in Appendix D.4 Details regarding IGPO's failure modes are pro- vided in im/ppendixx E Beyond empirical effectiveness, our theoretical analysis in Appeendix Alsoows that maximizing turn-level information gain constrains error accumulation in multi-turn scenarios. Thus, IGPO not only alleviates credit-assee aasitt-sssgnmmnt and advantage collapse problems but also reduces enror accumulation in long-horizon agentic tasks. The study be found in Appendix ]

---

Published as a conference paper at ICLR 2026
4.5 COMPUTATIONAL BUDGET
We present an empirical runtime comparison be- tween IGPO and GRPO. Results (Figure 7) show that the info-gain reward computation in IGPO in- curs overhead of less than 0.4% relative to the standard implementation. For the entire training step. IGPO increases latency by less than 0.02%, demonstrating a training speed nearly identical to GRPO. This underscores the superiority of IGPO: it enhances performance via fine-grained credit as- signment with negligible cost. Beyond the empiri- cal validation, we also provide detailed theoretical analysis of IGPO's time complexity n Appendix F
5.85
+0.02276
(s0.4%)
121.4
+0.02276
(<0.02%)
5.80
Avg. Time(s)
121.2
5.75
121.0
5,70
5.65
120.8
5.60
GRPO
IGPO (Ours)
120.6
GRPO
IGPO (Ours)
Recompute Log Prob
A Training Step
Figure 7: IGPO incurs negligible overhead (0.0227s / step), representing <0.4% in-
crease in info-gain reward computation and <0.02% end-to-end.
RELATED WORK
The recent success of reinforcement learning (RL) methods in large reasoning models (Chen et al, 2025a) such as OpenAl ol (laech et al. 2024) and DeepSeek R1 (Guo et al. 2025) has established RL as a central tool for enhancing large language models (LLMs)-based agents to solve more com- plex tasks. A growing body of work has explored different RL algorithms such PPO Schulman et al. (017). Reinforce++ (Hu 2025). GRPO (Shao et al. 2024). RLOO (Kool et al. 2019] Ahmaa dian et al. 2024). DAPO (Yu et al. 2025), and GSPO (Zheng al., (2025a). These methods have been particularly effective in improving the capabilities of LL.M-based agents eL al. 2025a).
Building these advances, important line of research has focused applying RL to search- based agents (Deng et al.] [0255] Dai et al.) (2055]aa). a). Early efforts such as DeepRetrieval (Jiang al., 2025) demonstrated the feasibility of end-to-end optimization by applying PPO with retrieval- oriented metries (e.g., recall) as rewards. Subsequent works, including Search-R1 (Jin et al. 2025),
DeepResearcher (Zheng al. 2025b), and ReSearch (Chen al., 2025b), extended this paradigm to multi-turn reasoning and search. RI RI-Searcher (Song et al. 2025a and Ra-dRR--Seeeeeee ---Searcher++ (Song et al.) 20256) further introduced two-stage RL strategies, separately strengthening the ability interact with retrieval systems and to utilize retrieved information effectively.
However, in multi-turn scenarios, outcome-only rewards remain sparse and often fail to provide suf- ficient guidance, leading to unstable optimization and inefficient sample utilization. Recent studies have explored step-wise or process-level rewards that assign credit to intermediate actions. Rea- SonRAG (Zhang et al. (2025c) adopted Monte Carlo Tree Search (MCTS) to approximate the value of each step. StepSearch (Wang et al. 2025) leveraged a memory vector of retrieved documents, supervising intermediate steps based their maximum similarity to ground-truth evidence. GiGPO (Feng al. 2025) introduced anchor-based grouping to estimate relative advantages for actions orig- inating from the same anchor state. While these methods provide denser supervision than outcome- only rewards, they either rely on external oracle knowledge or suffer from limited stability and scalability, leaving room for more intrinsic and generalizable process-level reward designs.
CONCLUSION, LIMITATIONS AND FUTURE WORK
In this work, we propose IGPO, simple and effective reinforcement learning framework for train- ing multi-turn LLM-based search agents. By providing intrinsic, ground-rruth-wwae supervision at every turn while preserving alignment with the final objective, IGPO delivers dense and stable train- ing signals. Extensive experiments across in-domain and -ft-of-domain benchmarks demonstrate that IGPO consistently outperforms strong baselines, achieving higher accuracy and better sample efficiency, particularly for smaller models where sparse rewards are most problematic. Moreover, IGPO demonstrates remarkable robustness to reward hacking while introducing only negligible com- putational costs. In doing so, it successfully addresses two critical bottlenecks inherent in existing fine-grained credit assignment RL algorithms: reward hacking and computational inefficiency.
However, our approach still relies on the availability of ground-truth answers, which limits its appli- cability in open-ended settings. In future work, we plan to extend IGPO to broader agentic scenarios beyond search, including tasks without explicit supervision.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Ant Group Research Intern Program. We thank the Venus Team, Ant Group for their resource support and technical guidance.
ETHICS STATEMENT
This work follows the ICLR Code of Ethics. No human animal subjects were involved. All datasets (NQ, TQ. HotpotQA, 2Wiki, MuSiQue, Bamboogle, PopQA) were used according to their respective guidelines, with privacy fully respected. No personally identifiable information was in- cluded, and all procedures avoided potential privacy or security risks. Research was conducted with transparency and integrity.
REPRODUCIBILITY STATEMENT
Every effort has been made to ensure the reproducibility of the results reported in this paper. All code and datasets are publicly available in an anonymous repository to facilitate replication and ver- ification. The experimental Setup-includdng training procedures, model configurations, and hard- ware specifications is detailed in the appendix to support accurate reproduction of the experiments. These measures intended to enable other researchers to reproduce the work and contribute to fur- ther advancements in the field.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in Ilms. arXiv preprint arXiv:2402. 147400 2024.
Kedi Chen, Dezhao Ruan, Yuhao Dan, Yaoting Wang. Siyu Yan, Xuecheng Wu, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Biqing Qi, Linyang Li, Qipeng Guo, Xiaoming Shi, and Wei Zhang. A survey of inductive reasoning for large language models, 2025a. URL httttp:://arxiv.ort arxiv.org..////.
2bbs/2110. 10182
Mingyang Chen, Tianpeng Li. Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang. Jeff Z Pan, Wen Zhang. Huajun Chen, Fan Yang, al. Learning reason with search for Ilms via reinforce- ment learning. arXiv preprint arXiv:2503. 19470, 2025b.
Gheorghe Comanici, Eric Bieber, Mike Schackermann. Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626. 2025.
Yuqin Dai, Guoqing Wang, Yuan Wang, Kairan Dou, Kaichen Zhou, Zhanwei Zhang, Shuo Yang. Fei Tang, Jun Yin, Pengyu Zeng, et al. Evinote-rag: Enhancing rag models via answer-supportive evidence notes, arXiv preprint arXiv:2509, 00877, 2025a.
Yuqin Dai, Shuo Yang, Guoqing Wang, Yong Deng, Zhanwei Zhang, Jun Yin, Pengyu Zeng, Zhen- zhe Ying, Changhua Meng, Can Yi, et al. Careful queries, credible results: Teaching rag models advanced web search tools with reinforcement learning, arXiv preprint arXiv:2508. 07956. 2025b.
Yong Deng, Guoqing Wang, Zhenzhe Ying, Xiaofeng Wu, Jinzhen Lin, Wenwen Xiong, Yuqin Dai, Shuo Yang, Zhanwei Zhang, Qiwen Wang, Yang Qin, Yuan Wang, Quanxing Zha, Sunhao Dai, and Changhua Meng. Atom-searcher: Enhaneing agentic deep research via fine-grained atomic thought reward. arXiv preprint 250iv:2508. 12800, 2025.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in -ooup-in-group group policy optimization for Ilm agent training. arXiv preprint arXiv:2505. 10978, 2025.

---

Published as a conference paper at ICLR 2026
Zeyu Gan, Yun Liao, and Yong Liu. Rethinking external slow-thinking: From snowball errors to probability of correct reasoning. In Forty-second second International Conference on Machine Learning, 2025.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint urXiv:2312.10999 2(1), 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforce- ment learning. Nature, 645(808 (:0-308)))633---11 638, 2025.
Carlos Gutierrez, Anthony Aguirre, Risto Uuk, Claire C Boine, and Matija Franklin. A proposal for a definition of general purpose artificial intelligence systems. Digital Society, 2(3):36, 2023.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Interna- tional Conference on Computational Linguistics, pp. 6609-6625, 2020.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.0326, 2025.
Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang. Songcen Xu, Jianye Hao, et al. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506. 18096, 2025.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint axXiv:2412.167200 2024.
Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. arXiv preprint arXiv:2503.00223, 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning, arXiv preprint trXiv:2503.09516. 2025.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 705.03551, 2017.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! 2019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466 2019.
Long Li, Jiaran Hao, Jason Klein Liu, Zhijian Zhou, Xiaoyu Tan, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, and Yuan Qi. The choice of divergence: A neglected key mitigating diversity collapse in reinforcement learning with verifiable reward. arXiv preprint wxXiv22509.04533 2025a.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366 2025b.
Xuefeng Li. Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrater rl. arXiv preprint arXiv:2503.23383, 2025c.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannanch Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:22 105/1, 7, 2022.
12

---

Published as a conference paper at ICLR 2026
Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, et al. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 6140-6150, 2025.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210003350, 2022.
Zehan Qi. Xiao Liu, lat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training Ilm web agents via self-evolving online curricu- lum reinforcement learning. arXiv preprint arXiv:241 2024.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji- Rong Wen. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198443, 2025.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp. 1279-1297 ACM, March 2025. doi: 10. 001114//6690331366 33696775.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ri-Rong Wen. RI-searcher: Incentivizing the search capability in llms via reinforcement 2025a.
learning. arXiv preprini arXiv:2503.0553
Huatong Song, Jinhao Jiang, Wenging Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher++: Incentivizing the dynamic knowl- edge acquisition of lims via reinforcement learning. arXiv preprint trXi::2505.7700 17005, 2025b.
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2022.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint ar arXiv:2312.08933, 2023.
Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang. Shizheng Li, Qi Qian, et al. Searching for best practices in retreeedl-augmeented generation. arXiv preprint arXiv:2407.01219 2024a.
Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting lims search ability via step-wise proximal policy optimization. arXiv preprint arXiv:2505. 15 2025.
Ziting Wang, Haitao Yuan, Wei Dong, Gao Cong, and Feifei Li. Corag: A cost-constrained trieval optimization system for retrieval- -uggeeted generation. arXiv preprint arX1v:2411.00744, 2024b.
13

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. thhouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24877 2022.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: 809009600, 2018.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Guibin Zhang. Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, al. The landscape of agentic reinforcement learning for Ilms: A survey. arXiv preprint arXiv:2509.0255 02544, 2025a.
Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chieh Huang, Chunyu Miao, al. From web search towards agentic deep research: Incentivizing search with reasoning agents. arXiv preprint arXiv:2506. 18959, 2025b.
Wenlin Zhang, Xiangyang Li, Kuicai Dong. Yichao Wang. Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Derong Xu, Zhaocheng Du, Huifeng Guo, et al. Process vs. outcome reward: Which better for agentic rag reinforcement learning. arXiv preprint arXiv:2505. 14069, 2025c.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint vrXiv:2507.18071 2025a.
Yuxiang Zheng, Shichao Sun, Lin Qiu, Dongyu Ru, Cheng Jiayang, Xuefeng Li, Jifan Lin, Bin- jie Wang, Yun Luo, Renjie Pan, et al. Openresearcher: Unleashing ai for accelerated scientific research, arXiv preprint vrXiv:2408.069/ 2024.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments. arXiv preprint arXiv:2504.03 /60,, 2025b.
Yuxin Zuo Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
14

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS
The theoretical analysis here provides an intuitive support for the efficacy of our proposed method by addressing the limitations of sparse outcome rewards in multi-turn agents. Specifically, the the- ory establishes crucial link: maximizing the process reward (IGPO's objective) equivalent to minimizing the upper bound on the undesirable accumulation of snowball errors during the reason- ing process. This minimization, in turn, systematically lowers the theoretical minimum for the final answer error rate, thus providing fundamental guarantee that IGPO's dense, turn-level signals lead to more confident and successful reasoning trajectories.
Notations. Let Emaal be the event that the agent's generated final answer does not match the ground truth answer. Its probability is denoted by IP (EEaa i.e. the error rate. For each turn t, denote the observed response [think], [tool ca 1l 1] as R,. We also posit that there is an unobservable, abstract thinking step I, that underlies the generation of R,, Let R be the process reward, which is dense reward signal received at each turn of the interaction. It is defined the information gain about the ground truth answer, which is calculated as the increase in the log- probabilltyy of the correct answer from the previous state the current state. Then, the total process reward Rooal Ei= [[pppoesss] is the cumulative sum all process rewards over complete trajectory or episode. The expectation is taken over the thinking step and observed response. The training objective of the policy is to maximize this total reward.
Definition A.1 (Snowball Error in Multi-turn Agentic RL). Consistent with Gan et al. 2025 we define the information loss at turn T as the conditional entropy Ent (R.). Consider the non- trivial case where (R,))) is bounded. The cumulative snowball error up to turn T is the sum of these losses:
T-1
Snt=e((I())
(8)
This quantity measures the aggregate uncertainty and ambiguity accumulated throughout the rea- soning trajectory before the final answer is produced.
Next, we connect the cumulative snowball eтror to the agent's final performance. It indicates the fundamental limitation of multi-turn agentic RL pipeline caused by snowball error.
Lemma A.2 (Lower bound of error rate). The probability of a final answer error, P(Emal)) is lower- bounded by the cumulative snowball error accumulated during the reasoning process:
P(Ejmai) = = (Enle(ZIR))
Ceonst-
(9)
T-
where Ceonst is small positive constant.
Proof Sketch. This result is strongly motivated by Theorem 3.3 from Gan et al. 2025 the generation of the final answer at turn T as the final step of multi-step reasoning process. The quality of this final step is conditioned on the information accumulated over the previous T - turns. The theorem from (Gan et al., 2025) states that the error probability of any step is lower-bounded by the average snowball error accumulated up to that point. Applying this principle to the final step = T) yields the stated result.
We treat
Assumption A.3 (Monotonic Reward-Information Loss Link), The expected process reward at any turn H. E RDpeew)) is monotonically non-increasing with respect to the information loss at that turn, Ent (I, RR.) We assume there exists a bounded and monotonically non-increasing convex function f: RR + R. such that:
R [T, R,] S f (En)(I.(R.)).
(10)
Remark. the information loss Eut ( tends to decreases and asymptotically approaches a relatively small value, which is characterized by the convex nature of function f.
RR,) at turn increases. the expected total information loss
15

---

Published as a conference paper at ICLR 2026
This assumption leads to the following result, demonstrating that optimizing for process rewards implicitly constrains the accumulation of snowball errors. We first formalize the intuition that a clearer reasoning step (lower information loss) is prerequisite for high-quality query, which in turn yields a higher expected process reward.
Theorem A.4 (Process Reward as a Bound on Snowball Error). Under Assumption cumulative snowball error is upper bounded by
A.3 the expected
E eE = 0(1) - Q (Rioat)-
(11)
Theorem A.4 establishes that maximizing the process reward is mathematically coupled with min- imizing an upper bound on the cumulative snowball error. The combination of Theorem A.4 and Lemma A.2 provides a complete, end-to-end theoretical justification for the efficacy of our proposed process reward mechanism. The logical chain is as follows:
Maximizing the process reward (our algorithm's objective) forces the agent to minimize upper bound on the cumulative snowball error (Theorem A.4).
Minimizing the cumulative snowhall error, in turn, lowers the theoretical minimum for the final error rate, thereby systematically increasing the probability of task success (Lemma A.2)
In conclusion, the turn-level process reward is not merely an engineering heuristic; it is theo- retically grounded mechanism that fundamentally addresses the problem of error accumulation in multi-step reasoning. By providing a dense, immediate signal for reasoning clarity, it transforms the intractable problem of sparse-reward, long-horizon exploration into a series manageable, short- horizon sub- proo-rrblems, each aimed at maximizing immediate information gain. This explains the significant gains in training efficiency and final performance observed our experiments.
B PROOF FOR THEORETICAL ANALYSIS
B.1 PROOF OF LEMMA A.2
Proof. We achieve this by applying Theorem 3.3 from Gan et al. 2025) to the final decision-making step of the agent. In partieular,
Ent (Z Ci
P P(Eai
T-1
(12)
log( log[|Anal 1)
where |Anial is the cardinality of the final answer space and C₁ a small positive constant analo- 2-1
Enter R) C
gous to Entz(e;) in Gan et al. (2025). Since log(IAnnall and C₁ are constant,
Ara 1) sim-
plifies to a form that is asymptotically dominated by the variable term. Therefore, the right-hand side of the inequality can expressed in terms of the lower bound symbol as Ω [ Cronsts which completes the proof.
T-1
B.2 PROOF OF THEOREM A.4
Proof. According to the nature of f and the fact that there exist constants Ciax and such that for all non-negative bounded there holds fx) Cmax 3x. Therefore, by taking the expectation over Assumption A.3 and summing across all turns from t 1 to - 1, we have
T-1
T-1
Ravial
Σ
El
R (1)
process] S
Σ
Elʃ (Ent(I,(R.))
<
E E(Cmaxx - 33 (L,
T
TT 1)0max 3
E
Ent (I, RR)]
= (T 1)0max 3 EE|Ent - (Z R)).
Rearranging terms yields the final result.
16

---

Published as a conference paper at ICLR 2026
C MORE IMPLEMENTATION DETAILS
All training experiments conducted on 8 NVIDIA A100-80G GPUs. The detailed hyper- parameter settings are provided on this configuration.
Table 4 Unless otherwise specified, all experiments are based
Table 4: Training hyperparameters.
Training hyperparameters
Value
Training Batch Size
32
Mini-Batch Size
512
Infer Tensor Model Parallel Size
Sequence Parallel Size
4
Max Prompt Length
30767
Max Response Length
2000
Actor Learning Rate
1e-6
Rollout Temperature
1.0
Rollout Group Size
16
Max Turn Call Turns
KL-Divergence loss coefficient
0.001
Table 5: Performance comparison across different IG bases and normalization strategies. We com- pare Prob-based IG (Joint & Separate normalization) against LogProb- based IG (Separate normal- ization). The combination of LogProb and Separate normalization outperforms other settings.
In-domain
Out-of-domain
Method
NQ
TQ HotpotQA
2Wiki
Musique
Bamboogle
PopQA
Avg.
Qwen2.5-3B-Instte
Prob+Joint
40.5
69.4
46.8
48.2
23.1
57.9
47.4
47.6
Prob+Separate
41.2
68.9
47.2
49.5
23.5
57.7
48.3
48.0
Logprob+Separate
41.9
69.2
47.8
51.4
24.8
58.4
49.0
48.9
Dwen2.5-7B-Instruu
Prob+Joint
46.7
80.1
57.2
68.2
31.4
74.9
52.5
58.7
Prob+Separate
46.2
80.3
58.2
71.4
31.8
74.6
53.6
59.4
Logprob+Separate
46.4
80.6
59.0
72.1
32.7
77.0
53.8
60.2
D MORE DISCUSSION AND EXPERIMENTAL ANALYSIS
D.1 COMPARISON OF INFO. GAIN BASIS (PROB. LOGPROB) AND NORMALIZATION STRATEGIES (JOINT vs. SEPARATE)
We investigate the impact of different information gain computation bases (probability vs. log- probability) and normalization strategies: joint (normalizing all rewards collectively) sepa- rate (normalizing IG and outcome independently). Note that we exclude the Logprob+Joint com- bination due the significant scale disparity between log-based IG and bounded outcome re- wards, which renders joint normalization ineffective. As shown in Table 5] the combination of seg-probability-bane information gain and separate normalization ((ogprob+Sepprate) emerges as the optimal strategy. Specifically, switching from joint to separate normalization (Prob+Joint + Prob+Separate) yields clear gain (+0.7 on 7B Avg. +0.7 on 3B Avg), validating the necessity of de- coupling the statistics of intermediate and final rewards. Replacing probability with log-probability (Prob+Separate - Dogprob+Separatt provides an additional boost (+0.8 on 7B Avg. +0.9 on 3B Avg), demonstrating the numerical stability advantages of nog-probability. These improvements consistent across both model scales and domain types, demonstrating the robustness our design choices.
17

---

Published as a conference paper at ICLR 2026
D.2 COMPARISON WITH OTHER PROCESS-REWARD METHODS
In addition to its obvious performance advantages, also conduct deeper analysis of IGPO's superiority terms of algorithmic characteristics compared to other process-reward rewars--rwwrd--bsee agentic RL algorithms. We first introduce other existing process-reward
agentic RL algorithms:
ReasoningRAG. The main contribution of this work is the proposal of step-level data labeling strategy based on MCTS. Subsequently, the DPO algorithm is used to optimize the agent's 's policy the labeled step-level dataset. The main limitations of this method are: (1) the data labeling process relies on MCTS, which is inefficient, and when the number of samples is insufficient, it is difficult to accurately estimate the value of each step; (2) the off- off-collcy optimization based on DPO is less effective than on-policy algorithms.
StepSearch. StepSearch constructs turn-level supervision signals by pre-defining golden search keywords and golden tool responses, and adopts an on-policy optimization approach. Although it shifts from off-policy to on-policy, the annotation process resource-intensive and prone to annotator bias (whether from humans LLMs).
GiGPO. GiGPO introduces a step-level grouping strategy based on anchor states and performs ene-grained advantage estimation within each step-level group. Although this provides novel solution, it essentially still relies on the Monte Carlo assumption. When the number of anchor states is insufficient, it is often difficult accurately estimate their value, which in turn leads to biased advantage estimation.
The proposed IGPO effectively addresses the aforementioned limitations. Starting from the on- policy GRPO setting (where rollout data are used for single parameter update), it employs an Informmttion-gain-aaae
incremental reward construction strategy that requires no annotation and does not rely on Monte Carlo. Moreover, the incorporation of ground-truth awareness substantially reduces bias. Table 6 provides detailed comparison highlighting the advantages of IGPO over other algorithms.
Table 6: Comparison between various process reward methods.
Algorithm
On-Policy Explicit Labeling-Free Free Monte Carlo-Free
Introduces No Bias Sample-size Dependent
ReasoningRAG
No
Yes
No
StepSearch
Yes
No
Yes
No
GiGPO
Yes
Yes
No
Sample-size Dependent
IGPO
Yes
Yes
Yes
Yes
D.3 TIME BREAKDOWN OF EACH STAGE IN IGPO TRAINING (SAME AS GRPO)
Table We have calculated the average time percentage spent on each phase of a training step for IGPO (same GRPO). The majority of the time is spent Sampling (Rollout), with Recompute Log-Prob accounting for less than 5% of the total duration. The time spent on the Return/Advantage Computation phase is much smaller than 1% and can be ignored.
Phase
Sampling
Param Update
Recompute Log-Prob
Return/Advantage Comp.
Time Proportion
82.6%
12.6%
4.8%
<<1% 1%
D.4 SPURIOUS CORRELATIONS ANALYSIS
It is widely acknowledged that LLMs often exploit spurious correlations to solve prob- lems lems-achieving correct answers through unfaithful or erroneous intermediate reasoning-ratiee than learning the genuine underlying reasoning process. This tendency significantly compromises their nut-of-distribution performance. To investigate whether the proposed IGPO mitigates or exac- erbates such spurious correlations, conduct the following experiment:
18

---

Published as a conference paper at ICLR 2026
Experiment Setup. We select the set of test samples correctly answered (F1=1.0) by both IGPO and GRPO agents. We extract the corresponding outputs, remove the final answers, and retaine only the intermediate reasoning traces. Subsequently,
employe powerful teacher LLM (gemini- 2.5-pro (Comanici et al. 2025)) to deduce the final answer based on these reasoning paths. By comparing the accuracy of the answers inferred from the IGPO versus GRPO reasoning traces, we assesse whether IGPO is more prone to yielding 'correct answers with incorrect or unfaithful reasoning" (i.e., spurious correlations). The prompt template used for gemini-2.5-pre illustrated in Figure 8
Result. As shown in Table 8 for both 3B and 7B models, the accuracy of the teacher model in inferring answers from IGPO-agent traces consistently outperforms that from GRPO-agent traces across all seven datasets, including both in-domain and out-of-domain tasks. This indicates that the reasoning traces generated by the IGPO agent are more informative and of higher quality. It demonstrates that IGPO effectively mitigates spurious correlations through ground-truth guided fine- grained credit assignment, further validating its generalization capabilities.
Additional Evidence. Beyond the aforementioned experiment, we provide the following evidence to further support that IGPO mitigates spurious correlations: (i) Superior OOD Performance. According to the results in Table 3) compared to GRPO, IGPO achieves an average performance gain of 12.6% (7B) and 42.8% (3B) on In-domain datasets (NQ, TQ, HotpotQA, and 2Wiki), whereas on Out-of-domain datasets (Musique, Bamboogle, PopQA), the average improvement increases to 13.7% (7B) and 55.2% (3B). The fact that performance gains in OOD settings exceed those in ID settings contradicts the pattern of spurious correlations which typically favors ID performance at the expense of OOD generalization. (ii) Exceptional Multi-hop Capabilities. As indicated in Table 3] IGPO outperforms GRPO with an average improvement 7.4% (7B) and 29.5% (3B) on single- hop tasks (NQ, TQ, PopQA), while multi-hop tasks (HotpotQA, 2Wiki, Musique, Bamboogle), the average improvement reaches 17.8% (7B) and 68. 1% (3B). The performance boost on multi- hop tasks is significantly greater than on single-hop tasks. This is also inconsistent with spurious correlation patterns, which are prone to appearing in multi-hop scenarios and consequently causing greater detriment performance. Therefore, the superior performance in both OOD and multi-hop scenarios serves as further evidence that IGPO effectively mitigates spurious correlations.
Table 8: Results of spurious correlations analysis. We select samples where both IGPO and GRPO agents achieve correct answers (F1=1.0). Using gemini-2.5-pro, we infer answers solely based on reasoning traces from these samples to compare the informativeness the traces gen- erated by each method ("All" denotes the aggregated accuracy across all test samples). The results demonstrate that, compared to GRPO, IGPO yiclds reasoning traces that are more informative and of higher quality, further validating its generalization capabilities.
In-domain
Out-of-domain
Method NQ
TQ
HotpotQA
2Wiki
Musique
Bamboogle
PopQA
All
GRPO
93.1
89.0
90.1
94.7
84.3
89.7
93.5
91.2
IGPO
95.4
92.8
94.2
97.7
90.2
92.3
96.1
94.6
Owen2.5-7BB-Instruet
GRPO
86.5
88.8
85.8
92.0
76.1
91.2
95.1
89.0
IGPO
89.5
91.1
92.6
94.4
83.0
94.1
95.1
92.0
FAILURE ANALYSIS
Despite IGPO's superior performance, to ensure comprehensive analysis, we investigated its failure modes, specifically examining instances where IGPO exhibits performance degradation (i.e., scores lower than GRPO). As shown in Table 9 minor degradation is observed across datasets, with IGPO underperforming GRPO on approximately 3.6% of the test samples overall. While this degradation marginal, it warrants in-depth analysis.
19

---

Published as a conference paper at ICLR 2026
Task
I will provide you with question and reasoning trace regarding that question. Your task is to infer the answer to the question relying "solely* the provided reasoning trace. the answer cannot deduced from the provided reasoning trace, you must reply with "unknown".
Question: (Question)
Reasoning Trace: [Reasoning_ Traces)
#Answer
Please strietly enclose inferred answer (or "unknown") within <answer> and <lanswer> tags.
Examples: Answer</annwer> =unwerrruknwwwctawwee..
Figure Prompt template for gemini-2.5-pro in the spurious correlations analysis.
Algorithmically. IGPO extends GRPO by computing the log probability increment of the ground truth answer between adjacent turns. This constructs a turn-level reward, providing stronger, denser, and teaenddttuth-wwaat supervision for fine-grained guidance. While GRPO relies solely on final outcome correctness, both methods depend on ground truth quality. Consequently, while IGPO amplifies the benefits of high-quality data, it inevitably exacerbates the impact of noise within the ground truth.
Through detailed data inspection, identified representative pattern of ground truth failure: am- biguous questions lacking specific conditions, leading to multiple valid answers. In such scenarios, IGPO is prone degradation. If the model leverages reasoning and tool usage increase the prob- ability of a factually correct-bur non-ground-truth--ann answar. it incurs penalty from the turn-level reward. This erroncously suppresses valid behaviors and impairs the model's reasoning capabili- ties. Figure 9 illustrates a real training instance from HotpotQA. The question, "Who is the author of Childhood?", lacks context (e.g., genre), allowing for multiple valid answers. When the model retrieved information regarding "Nathalie Sarraute" (a correct answer, though not the designated ground truth), was penalized (Info Gain = -0.81). This constitutes a false suppression of correct reasoning and tool usage. While GRPO also struggles with such ambiguity, IGPO amplifies this noise, resulting in a more significant negative impact. We observed frequent occurrences this pattern in the training set and identify it as the primary cause of IGPO's performance degradation.
is important to note, however, that this degradation stems from data defects rather than algorithmic flaws. The ability of IGPO to maintain high performance despite these data imperfections (with degradation rate of only 3.6%) effectively demonstrates its robustness. We look forward to exploring more complex failure modes in future work.
Table 9: We compared the F1 scores of IGPO and GRPO on the test set and analyzed the proportion of samples falling into three categories: IGPO GRPO, IGPO GRPO, and IGPO GRPO. Overall, IGPO exhibits slight performance degradation across datasets, with approximately 3.6% of test samples showing lower performance compared GRPO.
Dataset
IGPO>GRPE
IGPO=GRPO
IGPO<GRPO
2Wiki
35.8%
59.6%
4.6%
Bamboogle
47.2%
49.6%
3.2%
HotpotQA
49.2%
48.4%
2.4%
Musique
71.2%
25.4%
3.4%
NQ
57.4%
40.4%
2.2%
PopQA
42.8%
53.4%
3.8%
TQ
33.6%
61.2%
5.2%
All
48.3%
48.1%
3.6%
20

---

Published as a conference paper at ICLR 2026
Who is the author of Childhood?
Ground Truth: Leo Tolstoy
[think] answer this question, need perform a web search to find out the author of the book "Childhood". [tool call] ("query": ['author of Childhood book']}
Info Gain: -0.81
[tool response]
'title': 'Childhood, Sarraute, Kaplan, Wright- The University of Chicago Press' 'title': 'Childhood: A Memoir, book by Shannon Burns Text Publishing' 'title': Anxious Generation: How the Great Rewiring of Childhood Is...'
[think] From the search results, it seems that there are different books with the title "Childhood." " The book "Childhood, Nathalie Sarraute" is published by the University of Chicago Press and is an autobiographical work written when the
author was tighty-three years old. Therefore, the author of "Childhood" in this context Nathalie Sarraute.
FI Score: 0.0
Missing
1. "Childhood, Nathalie Sarraute' 'Childhood, Leo Tolstoy' 'Childhood, Markus Natten'..
Who is the author of Childhood?
Conditions
Figure 9: The query 'Who is the author of Childhood?" (a question from HotpotQA) is inherently ambiguous because multiple unrelated works share the same title-c.g. Childhood by Leo Tolstoy (fiction, ground truth), Nathalie Sarraute (autobiography, the model's answer), and Markus Natten (poem). Without specifying which literary work is intended, several factually correct answers ex- ist. Consequently, when the model outputs correct but non non-grooud-trutte author, it is penalized as 'wrong,' producing misleading negative rewards. Such mislabeled supervision degrades the effec- tiveness of IGPO by punishing valid reasoning aligned with alternative correct interpretations
F COMPUTATION BUDGET THEORETICAL ANALYSIS
This section will provide a detailed analysis of the additional FLOPs introduced by IGPO. Since the extra computation introduced by IGPO occurs during the recomput .log-probb phase (a single forward propagation), we will analyze the FLOPs a single forward propagation in the Transformer model examine the additional computation introduced by IGPO.
F.1 FLOPS CALCULATION
FLOPs (Float Point operations) represents the number of floating point operations, commonly used measure the computational complexity. We will focus on the FLOPs calculation in matrix multi- plication. For matrices A E R!xa and B E Rax1, computing AB requires multiplication opera- tions and n addition operations, totaling 2n FLOPS. Therefore, the FLOOPs required compute AB for matrices A E Rmxx and B E RExy is 2nmp.
F.2 SYMBOLS
Let batch size, s: sequence length, g: ground-truth answers length, h: hidden state dimension (as- sume the intermediate dimension 4h), headuam: number of attention heads, headaim. dimension
21

---

Published as a conference paper at ICLR 2026
of attention head, I: number of layers, V: vocabulary size, the imput data, Wq: the query matrix, Wx: the key matrix, Wv: the value matrix, W; the output matrix of the attention module, Wap: the up-pr up-projection matrix in MLP module, Wdown: the down- wwwn-proecction matrix in MLP module, yattni the output of attention module, Ymtp:: the output of MLP module, a: the activation function.
F.3 ADDITIONAL FLOPS INTRODUCED BY IGPO COMPARED TO GRPO.
We first analyze the FLOPs of GRPO's recompute log prob phase, which refers to the FLOPs of a single forward propagation. The shape of the input data is [b, s). We first analyze the self- attention module, whose computation process is as follows:
Q =: K == = V=IWv.
(13)
QKT
Yattn softmax(
Vh ). V. W.
(14)
The imput and output shapes of the matrix multiplication in Eq. 13 are: [b, s, h] x [h, h] + [b. s, h]. Based on ref FLOPs, the FLOPs for this process is:
FLOPsatta Satin1 = x 2bsh² 6bsh
(15)
For OKT in Eq. 14 the input and output shapes of the matrix multiplication are: [b, headwam. 8, headdim cess is 2b5"h.
x [b, headuum. headdim. s) + [b, headnam 8, s). the FLOPs for this pro-
in Eq. 14 the input and output shapes of the matrix multiplication are: [b, head s, x [b, head S, headaim] + [b, head 8, headdim], the FLOPs for this process is 2bs²h. h.
For attentionsscore
For the linear mapping operation of multiplying by W. in Eq. 14 the input and output shapes of the matrix multiplication are: [b. s, h] x (h, M] + [b. h). the FLOPs for this process is 2bsh?
Therefore, the total FLOPs in Eq. 14 are:
FLOP Suttm2 2bs h + 2bs2h + 2bsh 4bs2h 2bsh2.
(16)
Next, we analyze the FL.OPs of the MI. module. We assume the intermediate dimension (i.e., the up-projection dimension) of the MLP module is 4h, which is common setting. The computation process of the MLP module can be expressed as follows:
Umip o (Yatea Wap) W + Yattn-
(17)
For the up- up-projection operation in Eq. 17 the input and output shapes of the matrix multiplication
are: [b, s, h] x (h, 4h] + [b, 8, 4h] the FLOPs for this process is 8bsh?, And, for the down-projection
operation, the input and output shapes of the matrix multiplication are: [b, 8, 4h} x [4h, h] - [b, 8, the FLOPs for this process 8bsh2. Therefore, the total FLOPs of the MLP model are:
FLOPsmip 2 8bsh 16bsh
(18)
At this point, we have calculated the FLOPs required for a Transformer module during the for- ward propagation process: F FLOPsattn + FLOPsattaa + FLOPs Smip- Therefore, the FLOPs for the entire Transformer model during the forward propagation process are: I (FLOP + F FLOPs + FLOP smip).
In addition, due to the large size of the vocabulary V. the computation involved mapping the hidden state (dimension ht) to logits (dimension = V) at the end is also significant and cannot be ignored. The input and output shapes of the matrix multiplication are: [6, s, h] [h, VJ. the FLOPs for this process is FLOP Pslggtss 2bshV.
Therefore, the FLOPs of the GRPO's recompute.log.pzzo phase are:
FLOPs = 1 (FLOPs + FLOPsatta + FLFP mtpp) ) + FLOPPttgit Stogis
(19)
= 24/bs/h + 4lbs² + 26sh1>
22

---

Published as a conference paper at ICLR 2026
For the sake of simplicity in the subsequent analysis, we let 4lbh, 3 241bh? + 2bh1, The FLOPs of GRPO can be expressed as:
FLOPSGRPO as² + Bs.
(20)
Since integrated the process of calculating the ground truth answer's log probability into the recompute 1 og prob phase by extending the attention mask matrix 4.5 the FLOOPs for the IGPO's cerccmppue_log.ppre.. phase
FLOPsIGPO a(s g)² 3(s - g),
(21)
= a + 2sg) + B(s 9),
via 9 <<>
(22)
Eq. 21 to Eq. 22 use first-order Taylor expansion, as g (on the order of 10!) ) much smaller than (on the order of 10*) during actual training.
The additional FLOPs introduced by IGPO compared GRPO are:
AFLOPs FLOPsIGrO FLOP
(23)
22xg + ßg
The proportion of additional FLOPs introduced compared to GRPO is:
AFLOPs
2asg Bg
FLOPs
as² + 3s
g(as + B) + 08g
s(as 3)
asg
+
(24)
as(s
g
2
2g
In actual training scenarios, 22 typically on the order of 10 the additional FLOPs introduced by the IGPO's recomput..len -ol--robpb phase are only about one-thousandue of the original, and can be considered negligible. Considering the real time proportion of secompute.log.pree phase in Tablle 7 IGPO introduces only about one len-thousandth of the additional time consumption compared to GRPO per training step.
23

---

Published as a conference paper at ICLR 2026
G COMPARISON BETWEEN GRPO AND IGPO
Algorithm iillustrate the algorithmic flow of IGPO (right) and GRPO (left). The key steps cor- responding each algorithm are highlighted with the same color font visually highlight the dif- ferences: yellow for reward calculation, green for discounted/advanteee estimation, blue for credit assignment, and purple for policy optimization. In terms of reward calculation, IGPO constructs dense turn-level rewards through incremental information gain. For discounted retur/aavvantage es- timation, GRPO performs group-wise normalization, whereas IGPO performs group-wiss normal- ization followed by discounted accumulation. Regarding credit assignment, GRPO directly assigns the outcome- based advantage all tokens of the current output, while IGPO performs turn-level credit assignment. In policy optimization, IGPO achieves efficient and effective optimization by maximizing the turn-level discounted returns.
Algorithm GRPO IGPO
GRPO
IGPO
Require: initial policy Teai task prompts D; Require: initial policy Temn task prompts D; hyperparameters e, B.
max turns H; hyperparameters e. B, 7 p
1: < Tm
1: f- Te...
2: for iteration 1, do
for iteration 1, ... do
3: Tee t- To
3: Tre +-
4: for step 1..... M do
for step 1, I..... M do
5: Sample batch Dj from D
5: Sample batch Dr from D
6: Tha
+ m For each q E Do. sample G' outputs {y ~ ~ q)
Thee f- m For each
7:
7:
Do, sample G outputs
{ 1 ~ TBaa (-) for iteration 1.... T do
8:
Compute outcome reward {r:) from the final answer in each y/i
9:
t then
9: Compute each y:'s advantage {A.)
10:
Compute the infomation gain- based turn-level rewards 1G for each y/ (Eq.
via group normalization of {r:) (Eq.in Sec 2.2
{ra,)
Assign A, all tokens of y/
11:
else
11:
for GRPO iter = L...., : do
12:
Compute the final-turn rewards (rir)) based on the answer in each y/ (Eq.
12:
Update Ty by maximizing the GRPO objective (Eq.
13:
end for
13:
end if
14: end for
14:
end for
15: end for
15:
Normalize the turn rewards within the group to derive (frigesT) (Eq. 5)
16:
Compute the turn-level discounted turns {Ri, (m1
in each yi (Eq.
17:
Assign (RA,SIST) G the tokens in each turn for IGPO iteration ,,,..,,,,.... do
18:
19:
Update by maximizing the IGPO objective (Eq.
20: end for
21: end for
22: end for
24

---

Published as a conference paper at ICLR 2026
H PROMPT TEMPLATE USED IN OUR EXPERIMENTS.
Our prompt follows the style of DeepResearcher Zheng et al. (2025)), and the same template is used for training, validation, and testing. The prompt template shown the Figure 10 where {today) represents the current date to ensure the relevance of the model's 's response. Qi tool name } {f tool. de deescriee sereipiiion }} indicates the available tools, while the Rollouut section con- trols the model's output format. The # Tools section provides the model with the tool invocation method.
Today is (today) You are an Assistant*
The question give you complex question that requires a *deep research" to answer.
will provide you with tools to help you answer the question:
(%-- for tool tools.values() %)
[{ tool.name }): {( tool.description }}
(%-- endfor %)
You don't have to answer the question now, but you should first think about the research plan or what search next. Your output format should be one following two formats:
Rollout
<thinlo>
YOUR THINKING
</think>
<answer>
ENOUGH INFORMATION
</answer>
<think>
YOUR THINKING PROCESS
</think>
<tool_call>
</tool_calls call>
should always follow above formats strictly.
Only output the final (in words, numbers phrase) inside <answer-</answer> tag, without any explanations extra information. this yes-or-no question, you should only answer yes
Tools
You may call
functions assist with the query.
You provided with function signatures within <tools></tools> XML tags:
<tools>
(1%- for tool tools.values() (%)
(('(' "typpe" "function", "function": {{ (('( n>nnm "([ tool.name replace(""", ""))", "description":
"[l deo..description })", "parameters": {{ ")""tye":"""jjee""""""" "properties", ((tool.inputs replace(""" "")]), "example": ((tool.example replace("": ""]],, "uniqueltems" true(( ')} }} (\%- endfor (%) </tools>
For each function call, return json object with function and arguments within <tool_ chool_calb<</too__o XML tags:
<tool call>
(('( : <function-name>, "arguments": <args-json-object(("" Y
</tool_catl call>
Figure 10: Prompt template used in our experiments.
25

---

Published as a conference paper at ICLR 2026
1
CASE STUDY
Which film whose director is younger, College Lovers or The Dixie Flyer?
Ground Truth: College Lovers
[think] To answer this question, need find out the release dates of both films... [tool call] f'query': l'release date College Lovers film'...I
Info Gain: -0.80
[tool response] The release date of College Lovers is October 5, 1930 ...
[think] Next, need to find the birth years of John G. Adolfi and Charles J. Hunt to determine which director is younger... ttoo call| f'query': I'John G. Adolfi birth year'...I3
Info Gain: 0.42
|tool response] John G. Adolfi was born February 19, 1888
[think] John G. Adolfi was born on Charles J. Hunt was born on Since Charles J. Hunt was born earlier, he is younger than John G. Adolfi the time of their respective film's release. Therefore, the director of The Dixie Flyer is younger.
Fl Score: 0.0
Case study showing a scenario where the final answer is incorrect but contains a single correct retrieval turn. IGPO provides process reward for this turn, improving token utilization.
Figure 11:
26

---

Published as a conference paper at ICLR 2026
?
When is Augusta Marie Of Holstein-Gottorp" mother's birthday?
Ground Truth: 22 November 1610
[think] To answer this question, need find out the birthday of Augusta Marie Holstein-Gottorp... tool call] l'query': l'Augusta Marie of Holstein-Gottorp" ...IB
Info Gain: -0.39
[tool response] Augusta Marie of Holstein-Gottorp (6 February 1649 - 25 April 1728) was German noblewoman who became a...
[think] can see that Augusta Marie ... was born on To find her mother's birthday, need look for information about her family, specifically her parents. [tool call| ('query': l'Augusta Marie of Holstein-Gottory parents'...I
Info Gain: 0.45
[tool response] ... she was a daughter of Frederick III, Duke of Holttein-Gottorp and Duchess Marie Elisabeth of Saxony...
[think] Augusta Marie ... mother To find her birthday, I need to look for more details about Duchess Marie Elisabeth of Saxony. [tool call] lqquery': \'Duchess Marie Elisabeth of Saxony birth date'...]
Info Gain: 0.78
ttoo response) Duchess Marie Elisabeth of Saxony (22 November 1610 24 October 1684) was a duchess consort of
[think] The search results consistently provide information that Duchess Marie Elisabeth of Saxony was born on 22 November 1610.
Score: 1.0
Figure 12: Case study illustrating a situation where the first round retrieval failed, but the second and third rounds successfully located the correct evidence and produced the right answer. In this case, IGPO imposes a penalty on the erroneous retrieval in the first round.
27


Reviews:
[{"review_id": "sc8OnKpPzl", "reviewer": "Reviewer_LxTU", "summary": "This paper proposes IGPO, a reinforcement learning framework for training multi-turn LLM agents. IGPO addresses the reward sparsity and advantage collapse issues of outcome-only rewards by introducing a turn-level information gain reward, which quantifies how much each agent turn increases the policy’s probability of generating the correct answer. These dense, ground-truth-aware rewards are integrated with final outcome rewards to form a comprehensive signal for GRPO-style optimization. Experiments on seven search-based QA datasets (NQ, TQ, HotpotQA, 2Wiki, MusiQue, Bamboogle, PopQA) show that IGPO consistently outperforms prior outcome- and step-reward RL baselines, improving both sample efficiency and training stability.", "questions": "1. How does IGPO perform on non-search tasks such as mathematical reasoning, code generation, or web-agent? Would the information-gain reward still be meaningful there?\n2. Could you provide empirical evidence showing whether IGPO indeed mitigates spurious correlations, e.g., by testing on tasks requiring multi-step reasoning consistency rather than document retrieval?\n3. How sensitive is the training to the normalization scheme of turn-level rewards? Have you tried per-turn baselines or PPO?\n4. What is the computational cost compared to baseline algorithms, given the need to compute per-turn log probabilities of the ground truth?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. IGPO’s reward formulation is Simple yet effective, estimating turn-level information gain without requiring additional annotation or external evaluators.\n2. Strong empirical results: Extensive experiments on both in-domain and out-of-domain datasets show consistent improvements over strong baselines such as GiGPO and DeepResearcher.\n3. The paper is well-structured and easy to follow.", "weaknesses": "1. The evaluation focuses exclusively on search-based QA tasks. Such settings naturally align with the proposed information-gain reward, as acquiring relevant retrieved evidence directly increases the probability of producing the correct answer. However, in other domains, such as mathematical reasoning, embodied agents, or web agents, this reward may not lead to performance improvements. Consequently, the proposed method appears to be primarily applicable to information-retrieval tasks like search-based QA.\n\n2. Moreover, this reward could reinforce spurious correlations. LLMs are known to exploit shortcuts when solving problems, relying on superficial cues rather than learning the underlying reasoning process. Instead of acquiring new knowledge through comprehensive reasoning, LLMs may overfit to spurious correlations between final answers and intermediate components. In contrast, many recent works on mathematical reasoning explicitly aim to mitigate such spurious correlations. The design of the IGPO method could, in fact, encourage this undesired behavior, potentially having a negative impact on this research area.\n\n3. Finally, the paper computes normalized advantages using: A = r-mean(R)/std(R), where R aggregates all rewards across all steps and all rollouts within a group. This design is theoretically problematic. If the method assumes step-level rewards, the baseline should be computed per step. That is, by sampling multiple trajectories starting from that turn and averaging their returns. Otherwise, the estimator mixes rewards from different time steps with distinct state-action distributions, introducing bias. If, instead, the method assumes trajectory-level rewards, it should not be framed as step-level GRPO. The authors need to clarify or justify this design choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sTiHW94AZ0", "reviewer": "Reviewer_oHJt", "summary": "This paper presents Information Gain-based Policy Optimization (IGPO), a reinforcement learning framework that enhances multi-turn LLM agents for tasks like web search and multi-hop question answering. IGPO employs dense, turn-level intrinsic rewards derived from the marginal probability increase of generating correct answers at each step. By integrating these per-turn rewards with conventional outcome-based rewards, IGPO addresses reward sparsity and advantage collapse in long-horizon RL, enabling superior credit assignment. Experiments across seven benchmarks demonstrate IGPO's superiority over prompt-based and RL baselines, supported by ablation studies, training analyses, and theoretical foundations.", "questions": "1. Applicability Beyond Ground Truth Rich Environments. IGPO relies on ground-truth answers for teacher-forced reward estimation in every trajectory step. How would the method adapt to settings where ground truths are partial, noisy, or unavailable (e.g., open-ended queries, creative generation, real-world search)? Could unsupervised or self-consistency signals be integrated in lieu of explicit ground truth?\n\n2. Comparison With Omitted Recent Baselines. Please provide direct comparison (either empirical or qualitative) against the most recent 2025 turn-level/process-level RL methods listed above (e.g., Turn-Level Credit Assignment, LeTS, ToolRL). What fundamentally differentiates IGPO in real settings?\n  - Zeng, S., Wei, Q., Brown, W. (2025): \"Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment\"\n  - Wei, Q., Zeng, S., Brown, W. (2025): \"LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization\"\n  - Zeng, S., Wei, Q., Brown, W. (2025): \"ToolRL: Reward is All Tool Learning Needs\"\n\n3. Failure Modes and Robustness. What are the typical failure cases or degenerate behaviors for IGPO—especially when intermediate turns provide little actual information gain, or when there are multiple valid answer paths? Could the dense rewards inadvertently reinforce misleading intermediate confidence?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Principled and Well-Motivated Reward Design: IGPO addresses a pertinent weakness in agentic RL for LLMs—the reward sparsity problem—by introducing information-gain signals that provide stepwise, ground-truth-aware supervision. The approach is simple yet effective and is grounded in a clear theoretical motivation (see Appendix A).\n\n2. Thorough Mathematical Formulation: The paper provides clear derivations of the reward formulation (Equation 4–7), discounted advantage calculation, and the overall surrogate objective for IGPO. This mathematical clarity makes the method reproducible and portable to related settings.\n\n3. Comprehensive Empirical Evaluation: The authors conducted careful experiments across in-domain and out-of-domain benchmarks (NQ, TQ, HotpotQA, 2Wiki, MusiQue, Bamboogle, PopQA), and compared IGPO against strong baselines (both RL-based and prompt-based, see Table 1 and Table 2).\n\n4. Reproducibility: Public release of source code enhances transparency and facilitates replication.", "weaknesses": "1. Novelty is Incremental vs. Contemporary Efforts: While IGPO’s design is sound, many core principles—such as dense, turn-level supervision, teacher-forced signals for policy confidence, and combining process and outcome rewards—are parallel to mechanisms introduced or explored in very recent works from 2025 that are not thoroughly contrasted or ablated against. The differences from GiGPO, ReasoningRAG, StepSearch, and especially the missing related work are not sharply drawn.\n  - Zeng, S., Wei, Q., Brown, W. (2025): \"Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit Assignment\"\n  - Wei, Q., Zeng, S., Brown, W. (2025): \"LeTS: Learning to Think-and-Search via Process-and-Outcome Reward Hybridization\"\n  - Tang, X., Xu, W., Wang, Y. (2025): \"Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning\"\n\n2. Reliance on Ground-Truth for Intrinsic Reward: The information gain reward fundamentally requires access to the ground-truth answer for teacher-forcing in every trajectory step (see Section 3.2,). This is not always feasible for open-ended or real-world deployments, limiting applicability. The issue is acknowledged in the limitations, but its practical significance is not fully explored, nor are mitigations proposed.\n\n3. Absence of Fine-Grained Failure Analysis: While the aggregate numbers and training curves in Tables 1–3 and Figures 4–5 are generally positive, there remains a lack of granular breakdown where IGPO underperforms (if any), or qualitative investigation of error modes and tasks/environments where information gain may be less reliable (e.g., ambiguous or multi-answer questions). The failure cases in Figures 6 and 7 are insightful but could be complemented with quantitative measures of failure types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VeLO5fQFZU", "reviewer": "Reviewer_miYY", "summary": "They propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training. Specifically, they use log-likelihood of the ground truth at each turn to define the intrinsic rewards.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The algorithm is clearly presented. The algorithm design is simple yet effective.\n- The empirical evaluation is solid with significance improvements observed over multiple baseline methods.", "weaknesses": "- Limited Novelty: The paper introduces a new intrinsic reward design, but its novelty is somewhat constrained. Further exploration or differentiation from existing designs would enhance its contribution to the field.\n- Insufficient Theoretical Support: The paper lacks robust theoretical justification for the proposed intrinsic reward design.\n- Lack of Insight on Alternative Designs: The paper would benefit from a discussion on alternative intrinsic reward designs. Exploring how the proposed design could be integrated with other existing designs might provide valuable insights and broaden its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)