TIPS: Turn-level Information-Potential Reward Shaping for Search-Augmented LLMs
Authors: Yutao Xie
Nathaniel Thomas
Nicklas Hansen
Yang Fu
Li Erran Li
Xiaolong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Agentic LLM
Reinforcement Learning
Question Answering
Primary area: reinforcement learning
TL;DR: TIPS introduces turn-level reward shaping for search-augmented LLMs, yielding more stable training and higher QA performance than RLVR baselines.

Abstract: Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimization is often unstable due to sparse rewards and difficult credit assignments across reasoning and tool calls. To address this, we introduce Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model. By leveraging the potential-based reward shaping, TIPS offers fine-grained and policy-invariant guidance that overcomes the limitations of outcome-only optimization. Evaluated on seven QA benchmarks, TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability. For instance, with a Qwen-2.5 7B Instruct model, TIPS improves the average Exact Match score by 11.8% and F1 by 13.6% relative to PPO. Our results demonstrate that turn-level information-potential reward shaping provides an effective and general solution to sparse-reward credit assignment for multi-turn LLM reasoning.

Published as a conference paper at ICLR 2026
TIPS: TURN-LEVEL INFORMATION-POTENTIA, REWARD SHAPING FOR SEARCH-AUGMENTER LLMs
Yutao Xie
Nathaniel Thomas'
Nicklas Hansen'
Yang Fu'
Li Erran Li
Xiaolong Wang*
UC San Diego
AWS AI
ABSTRACT
Search-augmented large language models (LLMs) trained with reinforcement learning (RL) achieve strong results on open-domain question answering (QA), but training remains brittle: rewards are sparse, credit assignment across reasoning and tool calls is difficult, and optimization often collapses on long-horizon tasks. We introduce Turn-Level Information Potential Reward Shaping (TIPS), a seg-
simple RL framework that assigns dense rewards to each reasoning-tooll ment based on how much it increases teacher model's s log-likeliine oogllikelhhood of the cor- rect answer. This potential is computed by a frozen or periodically refreshed copy of the policy, TIPS only requires checkpoints of the model being trained- no practical
separate reward model, verifier, or human process for scaling frontier models. We show that this turn-level information reward is a form of potential-based shaping, preserving the task's 's optimal policy while providing fine-grained guidance beyond outcome-only supervision. On search- augmented QA setting spanning seven in-domain and out-of-domain benchmarks, TIPS consistently outperforms PPO/GRPO baselines and substantially improves training stability; for example, on Qwen-2.5-7B Instruct it improves average Ex- act Match by 11.8% and Fl by 13.6% over PPO. These results suggest that information-potentee shaping is a viable general mechanism for stabilizing long- horizon RL on large, tool-using LLMs. The code base for TIPS is available https: / htt//ggttuui wwwccc/cc/eeeocceceeceeeccoccece waassddww------ llla-aa-- 1mm1mm//aa 1m/tips.
INTRODUCTION
Large language models (LLMs) have recently shown substantial improvements in reasoning when fine-tuned with reinforcement learning (RL) under verifiable rewards (Jaech et al., 2024; Guo al., 2025; Lambert et al., 2024). In tool-using settings, this paradigm is especially appealing: we can define correctness at the end of an interaction (e.g., the final answer matches a reference), while allowing the model to freely interleave reasoning with retrieval, code execution, database queries. Yet this same property exposes a central bottleneck: the learning signal often outcome-only- single terminal reward after long sequence of reasoning and tool calls.
Outcome-only RL is brittle for training tool-use LLMs. Outcome-only rewards induce a severe credit assignment problem: the agent must infer which intermediate decisions caused success or fail- ure, despite receiving feedback only at the end (Ng et al., 1999; Devlin & Kudenko, 2012; Wiewiora, 2011). In multi-turn tool use, credit assignment is harder than in pure chain-of-thoughh reasoning for two structural reasons. First, each tool call is discrete intervention that changes the agent's s information state: a retrieval result, an execution trace, or table slice can fundamentally alter what is knowable in later turns. Thus, the policy does not only choose how to reason, but also what in- formation to acquire and when. Second, many distinct intermediate trajectories are consistent with the same terminal outcome (underspecification)): correct final answer may be reachable through both helpful and redundant tool calls, while an incorrect final answer might arise from a single early mistake that later steps cannot repair. As result, the terminal reward provides little guidance which earlier turns were informative versus distracting, leading to high-variance variance updates and frag- ile learning dynamics (e.g., policy collapse drift) when optimizing long-horizon behaviors with

---

Published as a conference paper at ICLR 2026
Policy model
Conversation
Teacher model
Information
Systen proupt
reward
User question
ooo
ooo
Ooo
Agent turn Reasoning
TIPS
reward
Query Search results
truth
Agent turn 2 Reasoning Query
Env
Search results
Agent turn Reasoning Final answer
Outcome
Answer
reward
verifier
Figure 1. Overview of our training framework. The policy model interacts with the environment by conducting multi-turn conversations: each turn consists of reasoning, issuing a query, and receiv- ing search results, until final answer is produced. Two dense reward signals are then derived: (i) an outcome reward, obtained by verifying whether the final answer matches the ground truth; and (ii) an information reward, provided by teacher model that measures the information gain each turn contributes toward the ground truth. Both rewards combined and optimized with PPO.
sparse feedback (Arjona-Medina et al., 2019). This issue is particularly acute in open-domain ques- tion answering (QA), where binary correct/incorrec signal does not indicate which intermediate retrievals or tool invocations improved evidence quality (Lightman et al., 2023; Gurung et al., 2025; Zeng et al., 2025b), thereby inder-incentiviiing reliable multi-turn reasoning with tools.
A natural response is to densify supervision. Process supervision and process reward models (PRMs) provide token- step-level rewards that better shape intermediate reasoning (Lightman al., 2023; Wang et al., 2024). While effective, these methods typically require high-quality intermediate labels and/or substantial offline training of reward models, and their supervision granularity is not always aligned with tool-use interactions, where the semantically meaningful unit is a turn consisting of reasoning, tool action, and the resulting observation. Another direction assigns turn-level rewards from environment feedback. For example, MT-GRPO uses turn-level signals derived from tool outcomes, but it was designed for single tool call; when extended to multiple calls, the signal becomes less discriminative and can inherit the instability of sparse-reward baselines (Table 1). Overall, prior work reveals a gap: we want turn-level feedback that is (i) discriminative enough to distinguish helpful from unhelpful tool interactions, (ii) lightweight (no token-level labeling or training an additional reward model), and (iii) compatible with standard RL fine-tuning.
We propose TIPS, turn-level credit assignment method based on information gain. We repre- sent an interaction as a sequence of turns, each following the Thought-Action- -biee-tttttinnOoovatiooo pattern common in tool-usine -uiin agents (Yao et al., 2022; Nakano et al., 2021: Schick et al., 2023): segment of reasoning tokens, a tool invocation, and the tool output. Intuitively, a good turn is one that makes the correct answer more predictable given the accumulated context, while distracting turn does not. Concretely, we quantify the contribution of turn by measuring the incremental increase in the agent's log- ligllikel of the gold final answer when that turn appended to the trajectory. Turns that raise the likelihood receive positive credit; turns that do not help (or mislead) receive small or negative credit. This converts an outcome-only signal into dense, turn-level feedback that di- rectly targets the core ambiguity in tool-use credit assignment: which ieformatioonacquiiiiti steps actually moved the agent closer to a correct answer.
We formalize the interaction as a segment-level Markov decision process (MDP), where each action corresponds to choosing a tool invocation (and its associated reasoning) and the environment returns the tool observation. Within this view, our information credit takes the form of a potential difference over segment states, yielding a potential-based reward shaping (PBRS) construction (Ng et al., 1999; Devlin & Kudenko, 2012; Wiewiora, 201 l). Under standard conditions, PBRS preserves the set of optimal policies while improving learning signals, offering a clean interpretation of TIPS as a policy- invariant way to reduce variance and stabilize long- long-horizon optimization. We integrate the shaped reward into PPO (Schulman et al., 2017) for direct optimization in standard RL workflows, and study how some internal design choices affect the training.
2

---

Published as a conference paper at ICLR 2026
We evaluate TIPS across eight in-domain and out-of-domain QA benchmarks with multi-turn tool use. Compared to strong PPO/GRPO baselines that rely on verifiable outcome rewards (Zeng et al,, 2025a; Su et al., 2025), TIPS yields consistent gains and more stable training dynamics. On Qwen- 2.5-7B Instruct, TIPS improves average exact match (EM; Appendix E. and F1 (Appendix E.2) by over 10% relative PPO, with larger margins ance and fewer collapse/drift failures. These results highlight that turn-level information shaping is an effective, lightweight alternative to token-level process supervision for making tool-using agents trainable under sparse terminal rewards.
GRPO, while exhibiting reduced training vari-
First, we introduce TIPS, reinforcement learning framework for multi-turn LLM agents that models trajectories as segment-level MDPs and assigns information-gail rewards each turn. Second, we integrate this turn-level reward shaping into PPO using potential-based reward shaping, which preserves policy invariance and stabilizes long-horizon optimization. Third, we validate TIPS on search-augmented open-domain QA across 8 benchmarks and observe consistent gains over PPO and GRPO, with the strongest improvements on multi-hop and out-of-domain benchmarks. Fourth, we analyze training dynamics and advan- tage distributions, and demonstrate that TIPS delivers more stable learning and allocates credit to effective reasoning and tool use while discouraging degenerate behavior.
Our contributions can be summarized as follows:
2 PRELIMINARIES
Problem formulation. We consider an LLM-based search agent for question answering. The agent may invoke a natural language search tool-embedding-bon -baeid or keyword retrieval-that ingests query string and returns the top-k passages. Given a dataset D = {(x,, A.)} where each question I; has small gold answer set A,, the agent must produce response through iterative reasoning and retrieval. Critically, the response must embed its final answer within <answer> tags, and evaluation extracts this tagged answer to compute exact match (EM) F1 against A;; this is not an open-ended generation task.
Token-level MDP. We model the interaction finite-horizon Markov Decision Process (MDP) at the token level. At step I, the state 8, consists of the prefix of all generated text so far including question, reasoning history and retrieved evidence. The action a is the next token sampled from the policy Te st). To transition to the next state, we either concatenate St+1 = St t dg, or if at triggers retrieval, we append an observation On, defining a boundary index bg The episode terminates when an EOS token is emitted, yielding a final reward Rinal based on answer correctness. All other tokens are assigned reward O.
On-policy RL with a response mask. We focus on PPO and GRPO, the most common on on-policy RL baselines for LLMs. PPO trains my against its snapshot Vold with clipped importance ratios
(+) = To (at (t)
Celip (f) = E, [m min(p:(0)A, clip(p:(0),
+))4)), (1)
Told (a; (s))
where A, G. Vo(St). G. -T= - rk, ra is the per-token reward, and Vo is a learned value baseline. GRPO removes the critic by normalizing sequence-level rewards: given g rollouts with terminal rewards ( (R(0), compute mean μ and stdev a, then set A = (R( - p)/a. /a. A binary mask m. E {0, 1} excludes non-train con-trainable tokens (e.g., prompts or tool outputs).
3 METHOD
Credit assignment is the bottleneck in multi-turn tool use. In tool-augmented QA, the model must decide not only how to generate text, but also when and what information to acquire via tools. The essential credit assignment question is therefore turn-level: among the turns that change the agent's information state (via retrieved evidence or tool outputs), which turns actually made a cor- rect answer more attainable, and which were redundant or misleading. Sparse terminal rewards provide little supervision for this: a model receives no feedback on which intermediate tool calls are beneficial until episode completion, and many distinct turn sequences can lead to the same final outcome. A correct answer may be reached despite unnecessary tool calls, while an incorrect answer may be caused by single early mistake that later turns cannot repair. This many-to-one mapping
3

---

Published as a conference paper at ICLR 2026
from multiple actions and CoTs to single outcome reward yields high- variance learning signals and brittle optimization in long-horizon training (Fig. 3).
TIPS addresses this by using the policy itself as a teacher: frozen snapshot the current policy scores each turn by measuring the in- crease in its own log-proe oog-robbbbility of generating correct answer. Intuitively, turn that re- trieves highly relevant passage will make the teacher much more confident some answer in A, whereas redundant or off-topic query leaves belief nearly unchanged or even shifts probability mass toward incorrect answers. The change in log-probability thus provides sin- gle scalar measure of how much turn & helped move the dialogue toward a correct answer. Be- cause the teacher is lagged copy of the pol- icy, their predictive distributions are kept close, so what elevates the teacher's confidence tends to benefit the policy as well. We periodically refresh the teacher prevent its beliefs from becoming stale and misaligned with the latest policy, Besides, the mechanism requires no ex- ternal judge and only adds a modest computational overhead.
system the search engine
Assistant
Aastatent clese atour the opdncces.
ramber of updeodes
question azpnndex is series gane
Query
query Eplaseks some
for meny spiasdes
betrioved preaerssed seader tee...
netrieved kendee, who worked on atow siath...
Mill
aill
S
Correctness probubility 14S:A)
Turn- level information rowerd AE
Turn-level information reward pipeline. At each turn, retrieved evidence up- dates the answer likelihood, yielding turn-level reward Ag. These rewards are then injected at turn boundaries.
Figure 2.
3.1 TURN-LEVEL INFORMATION REWARDS
In the multi-turn QA task, each turn comprises reasoning block, a tool call, and the evidence returned. To formalize the intuition above, let A = for episode. For any context S, we define the answer potential:
A be the set of valid answers
M
((S) = L(S:A) = log Σ
Prench (A(m)) S),
m=l
the teacher's 's log-pno tog-rbbbability of generating any correct answer. The turn-level reward for turn k is the change in this potential:
A a (asp))-((SS-111 (SS-
where Sy is the context up to and including turn and a > scales the reward. Intuitively, ((S) summarizes, under the teacher, how likely the current dialogue state is yield any correct answer in A. and A2 measures how much turn & moves this likelihood up or down. This quantity is positive when turn & shifts probability mass toward valid answers, and negative otherwise.
informationnthhordee interpretation. As can be interpreted as scaled pointwise mutual- information quantity between the evidence turn k and answer correctness under the teacher's distribution: it measures how much information the new observation contributes about the event that the answer lies in A. This view is especially natural for QA, where each tool call intended to supply incremental evidence toward correct answer.
3.2 SEGMENT-LEVER PBRS AND POLICY INVARIANCE
So far, As has been defined as a dense, turn-level signal that measures how much turn k helps the teacher believe in a correct answer. We now show that this signal can be viewed as a standard potential-based reward shaping term, and therefore does not change the optimal policy under the original outcome reward. We treat each turn as a segment of tokens between turn boundaries, and view a whole turn single action in segment-level MDP (where the action subsumes reasoning,
tool calls, and observations). With the answer potential ((S) from Section 3.1 as the potential function, the shaping reward at turn & is exactly
A a (+(()) ((Sa-1))
4

---

Published as a conference paper at ICLR 2026
which matches the standard form of potential- reward shaping (Ng al., 1999).
Policy invariance guarantee. Under episodic returns (y 1) and shaping confined to segment boundaries, the shaped Monte Carlo return for any token t in turn & satisfies:
G ((+1) G
Σ A G ((S--1),
where we assign ((Sk) = 0. Thus G; ((R+T) differs from G. _! only by a constant that depends on the segment boundary state Sk-1, and is independent of the within-tur action sequence тk- Actions that are relatively better under the outcome reward therefore remain relatively better after shaping. As a result, policy improvement under Monte Carlo estimation- and hence under GAE other policy-gradient estimators that use these returns to a baseline- unaffected.
Advantage Baseline Calibration via TIPS. Since Σj= A, -a ((S=-1), every token inside
turn A receives the same offset determined only by the pre-turn state Sk-1. Hence, TIPS densifies learning signals and reduces ambiguity about which turns were helpful, while preserving the optimal policy under the terminal reward. A full proof is provided in Appendix F.
Teacher re-syneing. With a fixed teacher, our shaping is potential based, so for any state the shaped Monte-Cas Caate return differs from the outcome-only return only by state-dependent constant. When we refresh the teacher, we simply change the potential used for shaping in subsequent rollouts; the PBRS view this is equívalent to changing a (state-dependen baseline and leaves the true advantage function invariant, though with an approximate critic, we do observe small shifts in raw returns before it re-stabilizes in practice.
4 EXPERIMENTS
In our experiments, we seek to answer the following questions:
(Q1) Performance & stability. Can TIPS train multi-turn QA agents that outperform outcome- only PPO/GRPO across in-domain and OOD tasks and remain stable (lower collapse rate, smoother/higher plateaus, reduced across-seed variance)?
(Q2) Learning signal design. Does turn-level information reward improve credit assignment compared with sparse final-answer rewards and rule/rubric baselines? How do token-level ad- vantage distributions differ under TIPS vs. PPO?
(Q3) Analysis & ablations. How do shaping scale a and teacher freshness (frozen vs. periodic refresh) affect stability and final EM/F1? What are the comparative effects of dense-reward choices (rule-based, rubric/LLM-judge,, information gain)?
Experimental Setup. Based on VeRL's multi-turn QA tasks setup, conduct experiments with two model sizes: Q.en-B.5-3B Instruct and Qwen- 2.5-7B Instruct (Jin et al., 2025; Qwen Team,
EM Accuracy
0.4
2025; Sheng et al., 2024). For retrieval, we use
the E5 model (Wang al., 2022) over the 2018 Wikipedia dump (Wikimedia Foundation, 2025),
retrieving passages per turn, and train on the merged NQ and HotpotQA training splits. We evaluate both in-domain and oft-oo-dommin perfor- mance on seven QA benchmarks: NQ. TriviaQA, PopQA, 2WikiMultiHopQA, MuSiQue, HotpotQA, and Bamboogle (Kwiatkowski et al., 2019; Joshi et al., 2017; Mallen et al., 2023; Ho et al., 2020; Trivedi et al., 2022; Yang et al., 2018; 8; Press al., 2023). For all evaluations, we report Exact Match (EM) and F1 scores (Jin et al., 2025; Zhao et al., 2025).
100
200
300
400
500
Step
TIPS (refresh@200)
PPO
GRPO
Figure 3. EM accuracy on the training set. TIPS converges to high accuracy; PPO drifts late; GRPO collapses.
Unless specified, use the following hyperparameters across all methods: PPO with GAE (A=1, y=1), KL penalty weight 3=0.001, clip ratio ==0.2, batch size 256, context length 4096, and maximum retrieval turns 4. Training runs for 500 steps, until performance collapse to 0, on
5

---

Published as a conference paper at ICLR 2026
Table Exact Match (EM) for 7B and 3B models. In-domain: {NQ, HotpoiQA]. Out-of-domain:
(TriviaQA, PopQA, 2Wiki, MuSiQue, Bamboogle). The last column is the average over all tasks.
In-domain
Out-of-domain
Model
Avg
NQ
HotpotQA
TriviaQA
PopQA
2Wiki
MuSiQue
Bamboogle
Qwen-2.5-7B Instruct
PPO
41.95
34.46
63.71
43.55
32.94
8.94
35,40
37.28
GRPO
37.15
26.54
56.39
37.11
19.41
7.20
16.00
28.54
MT-GRPO
37.17
29.28
58.29
38.37
22.62
7.99
19.20
30.42
MT-PPO
42.37
26.45
55.12
41.59
22.81
6.91
11.20
29.49
TIPS
43.38
42.95
64.31
44.52
42.96
17.05
36.80
41.71
Qwen-2.5-3B Instruct
PPO
43.80
27.12
58.28
42.81
23.10
6.37
9.60
30.15
GRPO
37.40
29.62
55.23
37.39
26.85
8.73
20.80
30.86
MT-GRPO
36.18
24.71
51.70
35.64
22.63
5.17
10.40
26.35
MT-PPO
39.65
25.54
56.17
40.47
21.35
6.16
8.00
28.19
TIPS
43.46
31.40
58.80
42.76
29.25
8.73
20.80
33.60
Table 2. F1 for 7B and 3B models. In-domain: (NQ, HotpotQA). Out-of-domain (TriviaQA, PopQA, 2Wiki, MuSiQue, Bamboogle). The last column is the average over tasks.
In-domain
Out-of-domain
Model
Avg
NQ HotpotQA
TriviaQA
PopQA
2Wiki MuSiQue
Bamboogle
Qwen-2.5-7B Instruct
PPO
50.53
44.65
70.88
47.62
38.70
17.79
45.33
45.07
GRPO
47.85
35.29
63.58
42.51
24.15
12.12
22.93
35,49
MT-GRPO
48.10
38.36
66.05
44.11
27.94
14.29
28.25
38.16
MT-PPP
50.60
35.29
62.08
46.45
28.56
12.13
20.87
36.57
TIPS
53.22
54.66
72.17
49.26
50.64
26.58
52.16
51.24
Qwen-2.5-3B Instruct
PPO
51.71
36.47
65.75
46.81
28.73
12.65
19.12
37.32
GRPO
47.40
39.51
63.30
43.00
33.36
14.81
28.45
38.55
MT-GRPO
45.77
33.10
59.51
40.40
28.13
10.68
15.90
33.64
MT-PPO
47.75
34.15
62.85
44.83
26.10
11.05
14.00
34.39
TIPS
51.77
41.47
66.43
47.43
35.10
15.85
29.82
41.12
82 GPUs with FSDP and gradient checkpointing. Full hyperparameters of both training and search engine server setup are described in Appendix E.3.
Baselines. We compare against four related reinforcement learning baselines. PPO (Proximal Pol- icy Optimization): Our base method with outcome-only supervision. Policy and value learning rates are 1 x 10 and x 10 respectively. GRPO (Group Relative Policy Optimization): We follow the public implementation with group size 5 and learning rate 5 10-7 To ensure stability, we apply gradient clipping at 1 x 10-4 which mitigates much of the collapse observed in reproducing Search-RI. MT-GRPO* and MT-PPO: Multi-turn extensions of GRPO and PPO for multi-hop QA. MT-GRPO* propagates stepwise normalization across multiple tool calls, while MT-PPO ap- plies environment feedback at turn-level. Both variants are tested with two rule- based rewards: (i) tool-correctness only, and (ii) correctness plus answer appearance in retrieved passages. In our main tables, we report the stronger variant for each model size: tool-only for 7B (answer-based often collapsed) and answer-aware for 3B. Implementations in Appendix F.5.

---

Published as a conference paper at ICLR 2026
Table 3. Generalization of TIPS across model families and scales. EM/F1 are TIPS scores; per- centages in parentheses indicate relative improvement over the outcome-only PPO baseline. FLOPs overhead is the relative per-step increase due to teacher scoring.
Model
EM
F1
FLOPs overhead (%)
Qwen2.5-3B 3B-Instruct
33.6 (+11.4%)
41.1 (+10.2%)
11.761
Qwen3-4B-Insttrut22500
48.4 (+7.3%)
57.1 (+6.1%)
11.846
Qwen2.5-7B- ----ttrructtt
41.7 (+11.9%)
51.2 (+13.7%)
11,810
Qwen2.5-1453 1-1--I1sBssrc
45.4 (+12.7%)
53.1 (+10.6%)
11.813
Llama3.1-88 1-8B
40.3 (+34.0%)
49.0 (+29.3%)
11.659
4.1 MAIN RESULTS From Tables and we find that TIPS consistently outperforms all baselines, with clear advan- lages over PPO and GRPO across both in-domain and out-of-domain settings. The largest absolute improvements appear on multi-hop out-of- tasks such as 2Wiki, MuSiQue, and Bamboogle, where outcome-only methods struggle. At the 3B scale, improvements are more modest and in some cases PPO performs competitively, whereas at the 7B scale TIPS shows a pronounced advantage.
For the MT' baselines, we tested two rule-based reward variants (tool-correctness vs. answer-aware) and observed instability: on 7B models the answer-aware variant often collapsed, while on 3B models the tool-only variant underperformed. As result, we report the stronger variant for each case in the tables. Overall, MT-GRPO* improves over GRPO by enabling multi-tool credit assignment, but both MT baselines still lag significantly behind TIPS. Combined with the training curves in Fig. 3, these comparisons highlight that TIPS not only achieves higher accuracy but also more stable optimization, avoiding the collapse of GRPO and the stagnation of PPO.
Training dynamics. As illustrated in Fig. 3, TIPS climbs steadily to an EM plateau of approxi- mately 0.55-0.60 with low variance. In contrast, GRPO suffers performance collapse around steps 320-350 and fails to recover, while PPO stagnates after 400 steps.
Generalization across models. Using the same training setup, we apply TIPS to five models from two families and multiple scales: Qwen2.5-3B/7B/1 4B, Qwen3-4B, and Llama3.1-80 As summa- rized in Table 3, TIPS consistently improves over the corresponding outcome-only PPO baseline on all models. with relative EM gains ranging from +7.3% to +34.0% and F1 gains from +6.1% 1 to +22.3%. The largest relative improvements appear on Llama3.1-80 which starts from weaker search capability and benefits most from better credit assignment, while stronger baselines such as Qwen3-4B still see solid gains. At the same time, the compute overhead of TIPS remains essentially constant across architectures: using the FLOPs accounting in Appendix H, teacher scoring adds only 11.7% per-step FLOPs for all five models in Table 3. Taken together, these results support our claim that TIPS is backbone-ggnostic providing consistent improvements across model families at a modest and stable relative compute cost.
4.2 ANALYSIS
Task-wise validation curves. In Fig. 4, we plot EM curves of TIPS and PPO for Qwen2.5-7B across all benchmarks. Benchmarks are grouped into General QA (NQ, TriviaQA, PopQA) and Multi-hop QA (HotpotQA, 2Wiki, MuSiQue, Bamboogle), with the top-right panel showing the average across all seven. Results are computed on a held-out validation set of 6,000 samples with the same benchmark distribution as full evaluation. Across tasks, TIPS rises smoothly and quickly stabilizes, whereas PPO exhibits drift-most most severe on multi-hop QA with mid-training degradation and only partial recovery. On general QA tasks the drift is milder but PPO still converges below TIPS. These dynamics align with Tables 1-2, where the largest margins appear multi-hop/On datasets. Overall, TIPS delivers higher final EM and more reliable optimization by preventing PPO's late-stage collapse.
Study of Advantage Distributions. To further investigate contributors to TIPS' stability, we collect all unmasked token-level advantages from final checkpoints. In Fig. 5, TIPS yields clean bimodal distribution with concentrated positive mass, while PPO shows fat-tailed positives and dense mass near zero, indicating instability and drift into poor policy space, which provides mechanistic ex-
7

---

Published as a conference paper at ICLR 2026
TriviGAA
PopΩA
Tasks
0.65
Average EM Accuracy
EM Accuraoy
EM Acguracy
0.60
EM Aazuracy
0.40
0.40
0.56
General
0.50
0.35
0.46
0.30
0
200
400
200
400
200
400
0
200
400
Training
Training Steps
Training Steps
Training Steps
HotperQA
2NHiMutHHopQA
MuBiQue
Bamboogle
0.175
0.40
0.150
Accuracy
0.35
EM Accuracy
0.3-
EM Accurscy
0.125
EM Accuracy
Must-wo aA
0.30-
0.100
0.25
0.075
0.20
0.050
0.1
200
400
200 Training Steps
400
200
400
0
200
400
Training Steps
Training Steps
Training Steps
PPO
TIPS (refresh@200)
Figure 4. Training dynamics of PPO vs. TIPS. Blue curves denote PPO and orange curves denote TIPS with teacher refresh every 200 steps. Overall, TIPS climbs steadily to higher and more stable plateaus, while PPO often suffers mid-training drift or collapse, especially multi-hop datasets.
Table Ablations on dense credit sources. Results are averaged over all tasks. MT-GRPO details are Appendix E.5; the LLM-as-judge variant in Appendix E.7; MT-PPO uses the same reward design as MT-GRPO; and the history- max information gain variant in Appendix E.
0.000
Probability mass
0.025
0.020
0.015
0.010
Method
EM
F1
0.000
GRPO family
2
3
Token advantage
Outcome-only
28.54
35.49
TIPS
PPO
Rule-based turn-level (MT-GRPO)
30.42
38. 16
Rubric-based tum-level (LLM judge)
28.23
35.54
Figure 5. Distribution of token-level advan- tages. Aggregated advantages at final check- points. TIPS yields clean bimodal distribu- tion with concentrated positive mass, while PPO shows heavy tails and dense near-zero mass.
PPO family
Outcome-only
37.28
45.07
Rule-bascd turn-level (MT-PPO)
29.49
36.57
Turn-level information gain max max info gain
History-
planation for why TIPS suppresses late-stage drift and collapse, and how it stabilizes the training trajectory observed in Fig. 4.
Computational Overhead. We isolate the teacher-scoring cost by measuring per-step wall-clock time in single TIPS run with and without scoring. This gives runtime overheads of 18% for Qwen2.5-3B and 16% for Qwen2.5-7B. In terms of FL.OPs, TIPS adds only about 12% relative to vanilla PPO for both model sizes, since given rollout. For comparison, GRPO requires roughly 3.5x the PPO FLOPs for both model sizes. Raw wall-time differences between separate TIPS and PPO runs heavily influenced by how long the model's 's responses are. A simple linear regression of per-step time on mean response length explains most of this gap, and after controlling for response length TIPS is within a few percent of PPO in wall-clock terms. Full overhead analysis is in Appendix H.
KV caches in the teacher forward passes
4.3 ABLATIONS
Shaping scale a. The coefficient a controls the relative weight between the information reward and the terminal outcome reward. If is too small, the shaping term becomes negligible and TIPS behaves like outcome-only PPO: if it is too large, the shaping term can compete with the terminal reward and increase gradient variance. In practice, we pick a that the average per-turn informa-
8

---

short pilot, estimate the typical Published as a conference paper at ICLR 2026
Table 5. EM gains over PPO for different target information-reward ranges under dynamic The medium band yields stable and consistently positive gains across backbones; very small targets effectively turn shaping off, while very large targets let shaping compete with the terminal reward and can hurt performance.
Base model
Small (0.001-0.05)
Medium (0.05-0.3)
Large (0.3-1.0)
Qwen3-4B Qwen2.5-7B
+1.4% (stable)
+7.3% (stable)
-3.4% (stable)
0% (crashed)
+11.9% (stable)
+3. 1% (stable)
Qwen2.5-3B
0% (stable)
+11.4% (stable)
0% (stable)
Table 6. Ablation on teacher selection. Rows fix the policy backbone and vary the teacher.
Policy
Frozen policy
Qwen3-4B-TIPS
Ilama3.
Qwen2.5-78
41.7 (+11.9%)
30.0 (-19.5%)
29.0 (-22.2%)
Qwen3-4B
48.4 (+7.3%)
45.88 (+1.7%)
43.0 (-4.7%%
tion reward clearly smaller than the terminal reward:
magnitude of Ax). and choose a fixed such that [[ R 0.2.
Across all backbones this places in medium band E [0.05, 0.3), within which TIPS is sta- ble and consistently improves over outcome- only PPO. We also tried dynamic-a scheme that keeps the mean information reward in target band; as detailed in Appendix D, the medium band (0.05, 0.3)) again yields sta- ble and consistently positive gains, while very small or very large targets hurt performance.
0.42
Average Score
Average EM Score
0.40
0.48
0.22
0.54
0.32
0:38
200
450
200
400
Training Steps
Training Steps
Different dense reward choices. Table 4 com- pares different dense reward signals. Outcome- only rewards perform better under PPO than GRPO, reflecting the advantage of value-based credit assignment. Within GRPO, rule-based turn-level shaping brings minor gains, while rubric-based supervision from an auxiliary LLM judge adds little-likely due to noisy sig- nals from heuristic or surface-form matching, or prompt drift. This mirrors our MT-setting findings, where the answer-aware variant often destabilized 7B training. For PPO, our turn- level information gain consistently outperforms outcome-only supervision, while history- gating weakens results, likely because it dis- cards informative negative deltas. Overall, ward shaping TIPS shows the most effective- ness across different reward settings.
TIPS
TIPS (refresh@200)
(a) Qwen2.5-7B Instruct model. Fixed vs. updated teacher. Updating every 200 steps improves stability.
Qwen
Qwen3-48
46
Avg EM (1)
44
42
40
Teacher selection. TIPS uses the teacher only through its log-likelihoods over valid answers, which define the potential (S)). In all main ex- periments we set the teacher to periodically refreshed frozen copy of the policy, so their distributions stay closely aligned. To test this choice, we fix the policy and vary the teacher among: (i) the frozen policy, (ii) a TIPS-trained Qwen3-4B, and (iii) Llama3. 1-8B, keeping en- vironment, data, and RL hyperparameters iden- tical. Across both policies, the frozen policy is clearly best: using different backbone (even stronger TTPS-traing one) degrades the 7B model and brings only small gains for 4B.
38
Interval (steps)
(b) Effect of teacher refresh interval final average for Qwen2.5-7B freshes (N=500) degrade performance, while a broad optimum emerges around N -0200 all TIPS runs with N E [100, 500) outperform PPO.
Qwen3-4B. Very rare re-
Figure 6. Effect of the teacher refresh interval on EM for TIPS.

---

Published as a conference paper at ICLR 2026
This suggests that behavioural alignment, rather than raw teacher strength, is crucial for TIPS, and supports using a lagged copy of the policy as the default teacher when scaling to other backbones.
Refresh interval. In TIPS the teacher is a frozen copy of the policy, refreshed every N updates. Very small N makes the potential & change rapidly (noisy shaping), while very large N makes & stale and misaligned with the improved policy, so treat N as a simple hyperparameter. Within each rollout the teacher is fixed and no gradients flow through Q so PBRS invariance holds at the trajectory level; refreshing only switches new potential between batches (App. F). We ablate N E {1, 100, 200, 250, } on Qwen2.5-7B and Qwen3-4B (Fig. 6), where 5-500 keeps the teacher fixed for the whole run.
Both models perform worse with a fixed teacher (N=500), confirming that a stale teacher harms the shaping signal. A broad optimum appears around N with -=100 and -=250 close behind. Across N E [100, 500). all TIPS runs still outperform outcome-only PPO, so the refresh window mainly controls how much improvement shaping yields rather than whether it helps at all.
RELATED WORK
RL for LLM reasoning and credit assignment. Outcome-supervised RLHF and PPO-style meth- ods (e.g., GRPO) are widely used for post-training LLMs on verifiable domains such as math and code (Ouyang et al., 2022; Schulman et al., 2017; Shao et al., 2024). However, outcome-only wards yield a severe credit-assignment bottleneck on long-horizon reasoning: the agent receives little guidance on which intermediate segments caused success failure (Arjona-Medina et al,, 2019). Classical potential-hased shaping provides dense guidance while preserving optimal policies (Ng et al., 1999; Devlin & Kudenko, 2012), and counterfactual baselines such difference rewards reduce variance by attributing marginal contributions (Wolpert & Tumer, 1999; Foerster et al., 201 7). In the LLM setting, prior work densifies supervision using process-level labels or PRM-style reward Ikeeihood-impprveeeedd
models (Lightman et al., 2023; Wang et al., 2024), or likelihood improvement objectives (Gurung al., 2025). Our method instead computes segment-level. leave-one-ous increments in a reference model's 's gold-answer bog-likelhhood, yielding dense counterfactual credit without manual step labels.
RL with tools for QA reasoning. Tools such as retrieval, search, and code execution improve QA by injecting evidence and exact computation (Gao et al., 2022; Schick et al., 2023; Yao et al., 2022; Nakano et al., 2021). Recent RL-with-tools methods increasingly target step-level credit assignment for tool calls (Zeng et al., 2025b). Our shaping generalizes to multiple tool segments and attributes reward proportional to each segment's 's marginal contribution to the gold answer likelihood.
LLM-as-a-judge and verifiers. LLM-based judges and verifier-style approaches provide scalable feedback for open-ended tasks (Liu et al., 2023; Zheng et al., 2023; Lee et al., 2023; Zhang et al., 2024). In contrast to subjective judge scores, credit signal is verifiable and model-based: frozen reference LM supplies seliihodd-base. segment-conditional increments toward the gold answer, enabling counterfactual attribution without training reward model.
CONCLUSIONS
We addressed brittle optimization in search-augmented RL for QA with TIPS, turn-level informa- tion shaping method grounded in a segment-level MDP. The potential is the teacher log-likelihood of acceptable answers, and shaping at turn boundaries provides dense credit while preserving the objective under PBRS. We integrate this into token-level PPO with response masking and KL con- trol. On seven benchmarks and two model sizes, TIPS improves EM and F1 over PPO and GRPO and trains more stably, with largest gains on multi-hop and oft-o--domain tasks. TIPS has two lim- its: the modest, but real computational overhead, and that it is currently tied to PPO. Future work will test quicker refreshes, explore using the policy as teacher, and study transfer to reasoning- heavy domains such programming and math. If successful, TIPS could become general mechanism for lorg-horizon credit assignment in LLM agents beyond web search.
Acknowledgements. The authors would like to thank Darrien McKenzie for helpful discussions. This project was supported, in part, by NSF CAREER Award IIS- IIS-2440144 NSF CCF-21 12665 (TILOS), and gifts from Amazon, Meta, and Qualcomm.
10

---

Published as a conference paper at ICLR 2026
STATEMENTS
Details of LLM use. We used large language models for three purposes: (i) to aid and polish writing, and (ii) to support retrieval and discovery of related work, and (iii) for code generation. For writing assistance, the authors drafted the text and used model suggestions to improve clarity, grammar, and flow. All text was reviewed and edited by the authors; the models did not originate technical ideas, methods, results, or claims. For retrieval and discovery, we used LLM-assisted search and summarization surface potentially relevant literature; inclusion decisions and all cita- tions were verified by the authors through manual reading. We used the Cursor IDE, which supports L.L.-based auto-complete and code generation during development. No experimental design or data analysis was produced by LLMs.
statement on reproducibility. Reproducibility is a priority. We' ve released code base for TIPS training and evaluation in https: //git thub. com/ cwm/ccsom picsowwang-les larnnglaabbaee mmmmippps We pro- vide exact training/evaluation configs for reproducing experiments; all materials are consolidated in the appendices. Specifically, implementation and hyperparameters for PPO/GRPO/MT variants are detailed in App. E.3 and App. E.5, with additional variants in App. E.7 and App. E.6. We also report training/ablation curves Figs. 3 and Together, these artifacts enable exact reruns and independent verification of all numbers in Tables 1-2.
Ethics Statement. This work does not involve human subjects, user studies, or the collection of personally identifiable information. All experiments are conducted publicly available QA benchmarks (e.g., NQ, HotpotQA, TriviaQA, PopQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) and a static 2018 Wikipedia dump for retrieval; follow the corresponding dataset and model licenses and terms of use. Because retrieval can surface biased or harmful content and can amplify incorrect information,
restrict evaluation standard, non-adversarial benchmark queries and report performance using established automatic metrics (EM/F1). We do not release any query logs, retrieved passages, or other artifacts that could contain sensitive or copyrighted text beyond what already available in the original public sources, Finally, while improved tool-using QA agents may be misused (e-g., to support misinformation at scale), the proposed method focuses on stabilizing training rather than enabling new tool capabilities; we encourage responsible use and recommend applying standard safety filtering and deployment safeguards in downstream applications.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. In Advances in Neural Information Processing Systems URL htttps://papers pttpss///pppersnne.ip..... wttppppee/219//111e...
(NeurIPS), 2019. 16105fbbcc61444229e11eo hww.ll/aao0oobde1-aaaaa11-
Zijian Chen, Xueguang Ma, Shengyao Zhuang. Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymogaaddam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang. Luyu Gao, Wenhu Chen, and Jimmy Lin. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent, 2025. URL https: / http//aaivv///////////////..a../ www.org//aaoeoooo. /as///b/////
/at////ss//...../ /2008.508000 06600.
Sam M. Devlin and Daniel Kudenko. Dynamic potential-basee reward shaping. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp. 433 443, 2012. URL https: //w::///wwwwww/ faamas.................a. org/ www.oooooceeiggg////ooooooooeeee a0mp papers/2C_5 pdf.
Jakob N. Foerster, Gregory Farqubar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1705.0890 08926, 2017. URL https: //rpxii///xxvvv///// org/abs /bs//705 08926. AAAI 2018 version available.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program- aided language models. arXiv preprint arXiv:2211. 2022.
URL pectps://arxia org/ abs /211..2111.11 10435.
Dong Guo et al. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv preprint arXiv:2501.12448 2025. URL https: /ttp///aaavvvv//aa///a//e....aaa org/ /twpb///251.... 12948.
Ayush Gurung, Jiaqi Huang. Shuyuan Wu, Kun Qian, Sam Thomson, Mirella Lapata. Marianna Apidianaki, and Chris Callison-Burch Learning to reason for long-form story generation. arXiv preprint arXiv:2503.2 22828, 2025. URL https: aariivivii. org/ abs /20050.... 22828. Intro- duces Verifiable Rewards via Completion Likelihood Improvement (VR-CLI).
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-6625, Barcelona, Spain (On- line), 2020. doi:
1tt.33//1/2222.oooogg..o. 2020. col ing in. 5.00
URL https: https////aaaaaahlloaao/cooooo org
Aaron Jaech et al. Openai ol system card, 2024. URL htt.::t:://axwvv..rggg///// 22111///....0///b//211.2. 2412. 16720.
Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O. Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-seane search interleaved Ilm agents, 2025. URL https:// arxiv. wtwpgg/b//////////// 15117.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1601-1611, Vancouver, Canada, 2017. doi: 10. 10.18653/v1/P17-116 URL https: https::/aclaathhlloee oxg/P17-1147/
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-War Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466 466, 2019. doi: 16.116//tacl 00276. URL hatps:///clantaii ogy. wtw.thhooooooo///111-100 9026/.
12

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tülu 3: Pushing frontiers in open language model post-training, 2024. URL https: //arxiv. orw. aaaaab///2444 15124. v5, revised 2025-04-14.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedhack with ai feedback. arXiv preprint arXiv:2309.00267, 2023. URL https: http:://wxwvv.../..aa//eo
00267.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050. 2023. URL https: / /ttp//aix...../////e//a/a/ 230.:ab///23.00000
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In EMNLP, 2023. URL https:
/ /aclanthologyyyo..o.ooo wwwpogg..or...oooooooo org/2023. emnlp- connp-mminn...
pdf.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 2802-9822, Toronto, Canada, 2023. doi: 10. 10.18653/v1/ a023 URL https: http://aaaaattltotooooo wtg//0233332222
546/
Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Floundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assiised question-answertig with human feedback. arXiv preprint 12X1v:21 2021. URL https: //tp///aaivv.v///a//aa//aaaaaaaa www.o/////o.o.ooo abs /2122.21122 09332.
Andrew Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward trans- formations: Theory and application to reward shaping. In Proceedings of the 16th In- ternational Conference on Machine Learning (ICML), pp. 278-287. Morgan Kaufmann, 1999. URL sttpss//people..o.o wewkeeee......ee..eeeeeeeeeeeeee edu/ www.eeedweeeeeeeeeeeeeee/ wpwppppeecss/7---eeee fa09/ Baadunngg//haaaadddddssll---pppo
pdf.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow in- structions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL LLEEGe3DeeE2TT3914580000000
neurips. cc/ppppp////////2 /202p////022hashh www.straatt-oofeeeneee...e html.. html.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Mea- suring and narrowing the compositionality gap in language models. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pp. 5687-5711, Singapore, 2023. doi: 10. 118653/v1/2223.3 1111:6551122223mmmnnnnnmmmmmmpm findiing--ennnl.3788 URL https: https/////aaaaaeoooooooog///////
findings-emn 3.78//
Qwen Team. Qwen2.5 technical report, 2025. URL httpp::/aaaadv..ogg//ab////////
15115. arXiv v2 (2025-01-03).
Timo Sehick, Jane Dwivedi-Yu. Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476.. 2023. URL htRp://s://ant http//.xivvv///////////////..av. org/ abs/2300 04761.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017. URL https: //arxiv. org/abs/1700 06347.
13

---

Published as a conference paper at ICLR 2026
Zheng Shao, Peiyi Wang, Qinkai Zhu, Ruipu Xu, Junteng Song. Ming Zhang, Yao Li, Yi Li, Yuxin Wu, and Dong Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open lan- guage models. arXiv preprint arXiv:2402.03300, 2024. URL 24022.0300.
org/ abs/
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409. 2024.
Yi Su, Dian Yu, Linfeng Song. Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23822 2025. URL http:: /arxiv org/ abs/2503 23829.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, (0:439-554, 2022. doi: 10. 1 URL https: http///aaaaaannoaoo.oaooo
org/2022. tacl-aallaa 1..1/.
volcengine and verl contributors. verl: Volcano engine reinforcement learning for llms. //github. ctt/ub.cco/// www/volooooooooo.oo.oooooooooo.. www.voonnnnnneeoooooooooooe /verl, 2025. Version 0.5.0.
https:
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly contrastive pre-training, 2022.
URL hwtps:tpp///... /wwa/aaavvoorg.. abs/ 2212. 03533.
Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhi- fang Sui. Math-shepherd: Verify and reinforce LLMs step-b without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Long Papers), pp. 9426-9439, Bangkok, Thailand, August 2024. Association for Computational Lin- URL httt:::: / httaa//aaaaaaalaa//a/a/a org/2024.
guistics. doi: 10. 10..0.66581 1633/////222242llllogg acl-long 510/.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung. Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents, 2025, 12516.
Eric Wiewiora. Potential-based shaping and value initialization are equivalent, arXiv preprint orXiv://06.522 2011. URL http:: / //tp///aavvv.//aa/e///ee.eeaaava www./a////////o//////..//..e/eee /100s.//1...000 5267. Original JAIR ver- sion 2003.
Wikimedia Foundation. English wikipedia dump (enwiki): pages-articles-muliise https: //dumps. wi k imeeia http://wwwk///222500/s/ 2025. Version 20250901.
David H. Wolpert and Kagan Tumer. An introduction to collective intelligence. arXiv preprint cs/9908014, 1999. URL httph:t///// /ttp:i///axivvv//ia wwt./br/g///eeeee////eeeeeeeeeee 9908014.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop ques- tion answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 2369-2380, Brussels, Belgium, 2018. doi: 10.18653/11/D18-111 URL attps::/aclantee
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint 0tti::2000006222 2022. URL https: /tt///aaiiiv/..///////eea./a.aaa
org/ abs/////////
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Reinforcing multi-turn reasoning in Ilm agents via turn-level credit assignment, 2025a. URL.
https:// /ttp///aavvvv/.a///a//eee/eawav. org/ wttgg/bs/////ooooe/oogo///eeaere 2200/2505... 11821.
Siliang Zeng, Quan Wei. William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Re- inforcing multi-tum reasoning in Ilm agents via turn-level credit assignment. arXiv preprint arXiv:2505. 11821, 2025b. URL https: /ttp://aaiii.////aa/eaa wwtp:/i.wwo//a//////./ 11821.
14

---

Published as a conference paper at ICLR 2026
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint 11X15:2408.15240. 2024. URL httt::///aaxt...rrrt abs
Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, and Limin Liu. R-search: Empowering Ilm reasoning with search via multi-reward reinforcement learning, 2025, URL
org/abs/ 2506. 04185.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, et Judging Ilm- as-a-judge with mt-bench and chatbot arena. In NeurIPS Datasets and Benchmarks, 2023. URL https://arat wwg//ogg/s...ooooooo...ee......e atgg/bb//330 2306. 05685. Introduces MT-Beech and analyzes judge bi-
15

---

Published as a conference paper at ICLR 2026
A DATASETS
A.1 GENERAL QUESTION ANSWERING
Natural Questions (NQ) Natural Questions (NQ) is benchmark introduced by Google using real, anonymized Google Search queries. For each query, an annotator is shown one Wikipedia page selected from the top five search results and labels long answer (typically paragraph) and, when possible, a short answer (one or more spans) or a boolean yes/no. If no valid answer is found, the example is marked NULL. This setup is intended to reflect the natural distribution of user information needs.
The public release includes 307,000 training examples, 8,000 development examples, and 8,000 test examples. Annotators identify long answer in about 49% of the examples, and short span or yes/no in about 36%. Each instance provides the question, the full Wikipedia page HTML, list of long-and answer candidate regions (HTML bounding boxes) with indices, and the gold annotations. NQ thus suitable both for variants (e.g. NQ-Open) that search the whole of Wikipedia.
models given the page, and for retrieval- retrieval-augmented
TriviaQA TriviaQA was released by the University of Washington as a large-scale QA dataset built from trivia and quiz websites. The diversity of question phrasing ensures the dataset challenges models in linguistic variation and evidence reasoning. It is widely used for benchmarking open- domain QA systems.
The dataset includes over 95,000 manually authored question-answer pairs and more than 650,000 censtion-answer-eviiee
triples. For each question, multiple evidence documents (around six on average) are provided; these come from two domains: the Web (retrieved pages) and Wikipedia. Each instance gives the question, gold answer(s), and associated evidence text, enabling evaluation of both retrieval and answer extraction performance.
PopQA PopQA was proposed to evaluate QA systems across both popular and long-tail factual knowledge. It is entity-centric and built from Wikidata triples, with questions generated via relation- specific templates. includes popularity metadata (monthly Wikipedia page views) to allow evalu- ation across popularity bands.
PopQA has approximately 14,000 English QA pairs. Each instance is created from sub- ject-relation-object triple, templated into natural question, and includes the gold answer plus fine-grained metadata: subject/entity IDs, relation type, and Wikipedia page-view counts. This setup supports controlled studies retrieval bias and factual memorization versus retrieval.
A.2 MULTI-HOP QUESTION ANSWERING
HotpotQA HotpotQA introduced to assess explainable multi-hop reasoning over text. Un- like single-hop QA, it requires combining evidence across multiple Wikipedia articles and provides sentence-level supporting fact annotations, encouraging models to justify answers via explicit evi- dence.
HotpotQA includes about 113,000 question-answer pairs, including a subset of "comparison" ques- tions (e.g. "Which person was born earlier?"). is offered in two settings: (i) distractor, where each example comes with a fixed candidate of Wikipedia articles (typically 10: 2 gold + distractors), and (ii) fullwiki, where systems must search over the entire Wikipedia. Each sample includes the question, the gold answer, and sentence-level supporting- fact annotations; in the distractor setting, the candidate documents are provided with sentences and titles.
2WikiMultiHoPQA 2WikiMultHHopQA a large-scale multihop QA dataset combining unstruc- tured text (Wikipedia) and structured knowledge (Wikidata). In addition to sentence-level supporting facts, it provides an explicit reasoning path via Wikidata triples, enabling evaluation of intermediate reasoning steps and explanations. The dataset uses relation-awan templates and logical rules to ensure genuine multihop questions across various reasoning types (comparison, inference, compo- sitional, bridge-comparison)
16

---

Published as a conference paper at ICLR 2026
The dataset contains about 192.606 question-answer pairs (commonly split 167K 12.7K / 12.7K for train/dew/test). Each instance follows a HotpotQA-style format (question, candidate contexts, supporting facts) and additionally includes field evidencess a set of Wikidata triples (subject, relation, object) forming the gold reasoning chain, as well as ent ity ids linking to Wikidata entities. The official evaluation reports metries answer accuracy, supporting fact identification, evidence triple prediction, and joint EM/FI for end-to-end reasoning.
MuSiQue MuSiQue (Multihop Questions via Single-hop Question Composition) was introduced to reduce shortcut learning in multihop QA by constructing questions via linking independent single- hop questions. The answer to one hop becomes necessary as input the next hop, thus enforeing compositional reasoning. The dataset provides the intermediate single-hop questions, their answers, and supporting paragraphs to allow analysis of decomposition.
The MuSiQue-Ans variant contains about 25,000 questions over 2-4 hops, with gold answers and candidate contexts (including distractors). A complementary variant, MuSiQue-Fai fulle adds contrast- ing unanswerable questions paired with the original ones, resulting in set. Each instance includes the multihop question, the gold answer, candidate passages, and the decomposition into single-hop steps. The dataset is designed to prohe compositional reasoning and resist shortcut strategies.
stringent evaluation
Bamboogle Bamboogle small but challenging dataset of hand-crafted two-hop questions. The authors curated questions for which popular search engines (e.g. Google) fail to return correct answers in top-rank featured snippets, while ensuring that both supporting facts can be found on Wikipedia. Its goal is to stress-test compositional reasoning without exploitable artifacts.
The dataset comprises 125 two-hop questions. Each question requires integrating two supporting facts from Wikipedia to arrive at the answer. In its public release, Bamboogle provides the question text and gold answer (but does not include supporting passages or fact annotations). serves as compact but difficult benchmark for multi-hop QA.
17

---

Published as a conference paper at ICLR 2026
PPO
4B-TIP5
GRPO
TIPS
8
4B-PPO
Accuracy (%)
Search-r1 32B-PPO
7B-TIPS
4
7B-GRPO
4B-GRPO
3B-TIPS
7B-PPO
3B- B--PP//GRPO
3B
4B
7B
32B
Model size (B)
model size. We plot PPO (blue circles), GRPO (orange squares), and TIPS (green triangles) for Qwen2.5-3B/7B and Qwen3-4B, together with the 32B SEARCH-RI PPO agent. TIPS consistently outperforms outcome-only PPO/GRPO at each scale, and 4B TIPS agent even surpasses the 32B SEARCH- R baseline.
Figure 7. BrowseComp- performance
B BROWSECOMP-PPLUU
BrowseComp (Wei et al., 2025) is a deep-research benchmark that evalu- ates M.LMrch agents using live, black-box web search API. While this setting is realistic, the underly- ing search backend is dynamic and opaque, which makes fair compari- son and controlled analysis difficult. BrowseComp-Plus (Chen et al., 2025) derived from BrowseComp and replaces the live web with fixed, carefully curated corpus and shared
Table 7. BrowseComp-Plus accuracy (%). TIPS consis- tently improves over PPO/GRPO, and our Qwen3-4B TIPS agent surpasses 32B SEARCH-R baseline.
Model
PPO
GRPO
TIPS
Qwcn2.5-3B-hll -nntstrructt
1.20
1.20
1.57
Qwen2.5-7B. 5-B---trrtttt
1.24
3.61
4.10
Qwen3-4B-Instee
6.75
2.65
9.40
Scarch-R1-32B
4.11
retriever over human-verified supporting documents and mined hard negatives. This design enables fair, reproducible comparison of deep-research agents and disentangled evaluation of the retriever and LLM components. To test whether TIPS also helps in such modern deep-research settings, follow the BrowseComp-Piin models in the LLM component, keep the retriever (BM25) and corpus fixed, and report the of- ficial accuracy metric (exact match against the gold answer). We compare our best models against the open-source SEARCH-RI ----R1132BBB agent reported on the BrowseComp-Pie Plus leaderboard under the same BM25-based BM25 setting.
agent-evaluation protocol. We plug our PPO/GRPO/TIPS-traine
C ANALYSIS OF TEACHER SELECTION
TIPS uses the teacher model only through its log-liikessoods over valid answers, which define the potential @(S). In all main experiments, choose the teacher to be a frozen copy of the policy (periodically refreshed), so that the two distributions stay closely aligned and the information reward reflects what is actually useful for the current policy. To test how sensitive TIPS is to this choice,
18

---

Published as a conference paper at ICLR 2026
ablation where we fix the policy and vary the teacher among three options: (a) the frozen policy, (b) a TIPS-trained Qwen3-4B 4B model (the strongest model we obtained), and (c) Llama3.1 1- 8B. The environment, data, and all RL hyperparameters scores on the multi-turn QA suite, together with relative gains over the PPO baseline parentheses. Two patterns emerge. First, for both Qwen2.5-7B and Qwen3-4BB the frozen policy configuration is clearly strongest. Replacing the teacher with a cross-family model (Llama3.1-8P) or with a different Qwen checkpoint (Qwen3-4B-TIPS) substantially hurts the 7B policy, and only yields small gain for the 4B policy. Second, the degradation is not explained by teacher capability: both alternative teachers are least as strong as the policy in absolute EM, yet their potentials induce worse shaping.
kept identical. Table 6 reports EM
This suggests that what matters for TIPS is not raw teacher strength but behavioral alignment be- tween teacher and policy. When the teacher distribution drifts too far from the policy, increases in the teacher's 8 answer likelihood no longer reliably indicate actions that are good for the current pol- icy, and the shaping term becomes a noisy or even adversarial signal. In contrast, a lagged, frozen copy of the policy remains close in behavior by construction, so its potential provides a stable, low- variance credit-assignment signal. These results support using the frozen policy configuration as the default choice when scaling TIPS to other backbones.
D ANALYSIS OF a SELECTION
The shaping scale a controls the relative weight between the information reward and the terminal outcome reward. Recall that the turn-level shaping term is
A = a ({{( $ (S,-
so for any token in turn k the shaped Monte Carlo return satisfies
G (( - a ((S-11),
(
(3)
where (() is a constant shared by all tokens in turn l and does not depend on the within-turn action sequence тk- Scaling therefore does not change which actions preferred within turn; it only rescales the returns (and hence the variance of the advantages) used by PPO/GAE. If a is small, the information term As becomes negligible compared to the terminal reward and TIPS behaves almost like outcome-only PPO, losing the benefit of improved credit assignment. If a is too large, the shaping term can dominate the terminal signal and the advantages are driven mainly by the teacher's 's potential, which may slow convergence and increase gradient variance. In practice, we therefore choose a so that the information reward remains clearly smaller than the terminal reward (1.0).
Fixed a: rule-of-thumb. For a fixed-a configuration, we run short pilot and use the first few training steps estimate the typical magnitude of A under the current teacher (with provisional o). We then choose a fixed a so that the average turn-level information reward is capped around 0.2, i.e. [[[[==]]]] 0.2, keeping the shaping term well below the terminal reward. Across all backbones we consider, this procedure places in a medium band a E [0.05, 0.3), within which TIPS is stable and consistently improves over outcome-only PPO.
Dynamic a: target information-rewart range. To further reduce sensitivity to single fixed value, we also explore a dynamic-a scheme.
Instead of fixing a, we maintain target range for the mean information reward and adjust online so that the running mean of |a stays in that band. Concretely, we define three target bands for the (normalized) information reward:
small: [0.001, 0.05], medium: [0.05, 0.3], large: [0.3, 1.0],
and adapt a during training to keep the signal within the chosen band. Overall, both the fixed-ax rule- th-uh and the dynamic-a ablation point broad medium regime where the information reward is bounded to be substantially smaller than the terminal reward (on the order of x 0.2 per turn), and within which TIPS is robust and reliably improves outcome-only PPO without delicate tuning of a.
19

---

Published as a conference paper at ICLR 2026
E IMPLEMENTATION DETAILS
E. EM COMPUTATION
The EM, or exact match metric, is if the submitted answer is exactly equal any acceptable answer, and 0 otherwise.
E.2 F1 COMPUTATION
The Fl metric between predicted answer and a gold answer is computed as
2
Appred 0 a'gold
Fl (Cipreds (goldd)
(4)
Opred
(lgold
If there are multiple acceptable answers, we log the maximum Fl score among them (Zhao et al., 2025).
E.3 HYPER-PARAMETERS
Table 8. Retrieval server configuration
Parameter
Value
topk (-topk) faiss gpu (-faiss gpu) perievall_ethor litievaalppooinggmeee method
True
e5 (dense)
mean
retrieval Sirievvllluury..eee _elelluuee_mm__l suueer_max_eennll
256
retrieval use fp16
Truc
retrieval batch size (-batch size)
512
max content tokens (-max content tokens)
500
server.host
0.0.0.0
server,port
8000
20

---

Published as a conference paper at ICLR 2026
Table 9. Hyperparameter settings for GRPO-famiLy algorithms
Parameter
Value
dgooritmm.ae estimator
grpo
algorithm.gamen
1.0
algorithm.Ian
1.0
algorithm. use rewarrd www.rooolreffaaeeeeeo
False
loss
True
actor actor_rollow rollout ref.actor.ki loss type actor actor_rollout rollout ref.actor.kl loss loss coef actor actor_rollout rollout ref.actor actoor.grad graddcliigpipp clip policy_loss, Loryyolssssossss loss.mode mode
low var low_var kl
0.001
le-4
vanilla
clip ratio
0.2
clip ratio.
3.0
loss_agg_man mode
token-
actor_rollout_st Lee aetor.ctppppp . batch
256
actor_rolloue aecor_rolluut_fff.ccoooo..p wwtp://faccttopopooc..... pairoomicaaatleee batch_size.per.guu
ppo_epochs
1
shuffle
True
train batch size
256
data. bat.cal.bchhiiiii
256
www.maxxprropt.eec
4096
data.max aitaamax_rcaa www.rreeepeeeneeeeeee
4096
trta.trunn
eтоo
trainer. total totalle
trainer.critic warmup
actor wwwlooutree..aaeeeeeee pelllouuteff.ctooootii...tt
5e-7
actor www.rroooluutreeeeooooooooo ref.roolout..na name
sglang
actor rollout www.rrooooo....oolllooooo ref.rollout.ma. model.
15000
actor rollout ref.rollout.tense. model parallel size
refooroollouutfffoolloloooooo memom iinemooyttillization _utilization
0.7
actor wttprrooooouuutfrolloooooooooo
5
http:roolouurrffrolllllleeooo patuuttulti__uuut mmulti_tuun.mat turn.max..mmmmmmmmmm _assistant_turms
5
actor rollout ref. ref.model.use remove padding
True
actor rollout ref.model.enabll gradient checkpointing
True
actor rollout ref.rollout.enable chunked prefill
True
'LLM.USE.VI (env)
0
www.rrolooutteffcctooooooo cenddpccooig.gooe wow.iio-paaaaammaacmmmmmm param param-offload offlload
True
actor_rollout .te.ttrfff.o....ee.eee aww....soopp........... wetrrrsddo.ooooeeeee.eee optimizze offload
True
cctor_rolloutrrff.o wwnp...oni......omomoooooooooooe param.orfo oofflodd
True
actor ciforrrolloutrre ref.nncll_innnnn
600
trainer.n_gpus Saaeeennnpp___ee__ee pouu_per_node node
8
CUDA DEVICE MAX CONNECTIONS, (env)
1
NCCL DEBUG (env)
21

---

Published as a conference paper at ICLR 2026
Table 10. Hyperparameter settings for PPO-family algorithms
Parameter
Value
algorithm.ade estimator
gae
algorithm.us use kl reward
False
data.train batch size
256
data. val_bachh_size
256
data. aata.max.pronne waax.ppooppt.eeneee
4096
data. max_ wewx_resppose_eeee length
4096
data. wwwpfffltrrovveee oveerong prompts trtncctunnation
True
error
data.return raw chat
True
actor wooooroolouut... www.uuteeffeeeeee ref.actor.optim..t
le-6
actor_rolle www.ooot.uu.... wwwloot..ef..cttttt. ref.actor. ceotorggrdd.cci grad
1.0
actor actor_rollout www.loolloteeeeee actor mini biteh_size
256
www.loot.eff.ctorrrrrrr....o.opp wwwpooooooo.ieeeeeeeeeee mmicoo.bacch.siaee batch.sizeeperrgpuu
8
actorrrollout.ref.ccoo
kl_loss
True
heto_rollouurref.atttoooo _l__oss_coef
0.001
actor_ne actor_rollowe wowloot..ef..ctor..... wtt.rr.attorr.rll.sss....
low_var_k k
actor rollout ref.actor.entropp coeff
0
actor rollout ref.model.use remove padding
True
howprrolooot.ef.mmooe...o wnw.oeee.eealleeeeeeeeeeeeeeeeee enable-gradient. www.ggrrdddeeeeeeeee wtt.gggien.ceeeecontiiiiiei
checkpointing
True
ctttoorrooloutrrf..aooo www.:rrsssdspp.ppaaar.e. caatiigparamoofieee offlload
True
actor actor_rollowe actor cend__cofig..o config.optimin
petimizer_offfoann
True
actor_rollout_rff.rr wtw.:r..fff..ooooooooo... par--prooomrrrro...e
size_perrgpp
8
actor www.tt..f..ffffffffoofoooooo
paramggppaam...m param_offload offload
True
actor actor_rollout www.orollout... rof.roolooutnnaae name
sglang
actor cictorrrollouut._. www.rroottee..oolllleeeot www.re.ooolou..mm... waloouttmmaammmee maax_model_kee Jeen
15000
actor coloorroolouurreeeo ref.rollout.eee wensortteenor....oo tensor_model. paralllllllllllaaaeae size
1
actor rollout ref.rollout.gpp memory utilization
0.7
actor rollout ref.rollout.n
1
actor_rollout.rg
boo_probmmicro.... www.mmmmcattttt batch_size
128
actor actor_rollout_ret hwcllouutteff.rolllllll www..rollou..maaiiooooo multi turn.tuaasss aasiistat_tuunnss
5
critic.ppo_miee haapn_mmrrooohhhhhhhaeeeeea www.///ahhhhhaaeeeeeeeeeeeeeeeee wethhsssssee_eeppppp
8
reward reward kwargs.score source
em
trainer.critic warmup
trainer.n gpus per node
8
trainer.nnodes
1
trainer.save.
50
trainer.test_fie
1000
trainer.log trainer.ogg.val.emeeeree val_erncsssss
50
trainer.toaal.cpe .epochs
1
VLLM_USE_V (env)
0
HYDRA FULL ERROR (env)
1
CUDA DEVICE MAX CONNECTIONS (env)
NCCL DEBUG (env)
info
22

---

Published as a conference paper at ICLR 2026
E.4 MULTI-TURN RULE-BASED REWARD DESIGN
Per-segment rule rewards. For each tool-use segment 2 </tool_response>, we define rule-based segment reward
delimited by
rule Cexec T[E Cans ((Ai,,);
where Cexec; Caus > 0 are fixed coefficients, and the events are
E,, = (<tool.cal>> E v.) (segment non-empty) ^ (-segment starts with "Error:"),
A = (3a E Aace : a segment (lowereased)).
with Asce the set of acceptable answers from ground truth, At most one presence credit awarded segment even if multiple matches occur.
From segments to tokens. Let s(t) ! -, denote the segment token t, boundaries. We map rewards r to tokens via
obtained by scanning for
Triple 0,
t= maxfu: s(u) = 8, m. = 1};
rule i,t
otherwise,
where my E [0, is the response mask. Segments with E 0 receive no reward.
Segment boundaries are detected by regex over s/toolrrepponse. robust to cross-token splits. (ii) Answer extraction supports list/string fields and lowercases both sides before matching. (iii) We provide two mapping modes (last_toked suit different credit-shaping preferences; experiments default to last-token placement. (iv) We set fixed magnitudes for the two rule components: the tool-call correctness reward is 0.1 and the answer presence reward is 0.15 per segment before scaling (k) and mixing (w). These values are intentionally small so that the (standardized) final outcome reward remains the dominant learning signal.
Implementation notes.
/ distributed
E.5 MT-GRPO
Single tool call MT-GRPO Denote prompt (input state) and its sampled group of trajectories (responses) )m Let R; be the outcome (final) reward of trajectory and r be verifiable tur-level reward at turn t. MT-GRPO assigns turn-level advantages by combining normalized turn rewards and normalized outcome rewards. For two-turn agent, let
P.())
R,
0,0) +
On +
and introduce hyperparameter 3 E [0, 1]. Then define
A 3 f(()) (1 3) R.
A R.,
These scalar advantages are broadcast to the token-level each turn, and the final loss is
(NT-CGRO(() -yy
Σ
min(p clip(a,,1e,,1111
Aetum
min(pi ()) 1 1=
(5)
Thus MT-GRPO refines credit assignment: the first turn's policy update is influenced by both ver- ifiable intermediate reward and the final outcome, while the second turn directly relies on outcome reward. This finer attribution helps stabilize training of multi-turn tool-using agents.
23

---

Published as a conference paper at ICLR 2026
Multiple Tool Call MT-GRPOS. For the general case with S 2 1 tool segments, extend the same framework by assigning each tool-call segment its own normalized credit. Specifically. for response i and segment 2 0, compute the mean reward Fi,s over its tokens and apply group- wise standardization across only those responses that include 8, yielding Fí,s- The final (outcome) reward is standardized as before, giving R.
The token-level advantage then becomes
8-1
Σ Miss, (Aaii (s,s) + Ainal Ri: ==0
where Mi...t indicates whether token belongs segment Tokens in the final answer segment (s -1) only receive the outcome reward.
The policy is updated with the same clipped surrogate objective in the single-call case. This formulation naturally scales to multiple tool calls, with each tool segment contributing its own credit while the outcome reward provides shared global signal.
E.6 TIPS WITH HISTORY MAX
Motivation and design choices In the original TIPS formulation, the turn-level shaping reward is simply
A ((S) ((Sx),
i.e. the marginal increase of the teacher's set-marginal log- log-likelihood over acceptable answers. How- ever, A& may become negative when an intermediate tool invocation temporarily misleads the teacher's belief, which could discourage exploration. In contrast, our history-max variant defines
A hmax max max (()) max (S))),
J
jck
so we only reward turns that elevate the maximum teacher belief seen so far, never penalizing non- improving but potentially useful steps.
We inject J = A hmax at turn boundaries (with a >> 0) via potential-based reward shaping (PBRS). Under standard episodic assumptions (y and when using Monte Carlo returns (A 1), the shaped return from any token in segment becomes:
K
G (R++)) = G
-
- j=k
( + +a(FK Fk-1),
where F2 maxjsı (S)). Because the extra term a(Fx Fk-1) is independent of within- segment actions (it depends only on prefix Fk-1 and terminal Fx), it does not affect relative ordering between policies. Hence, the shaping preserves policy optimality.
E.7
LLM AS JUDGE
Motivation and design choice. Beyond measuring tool correctness and answer presence in tool response, we also wish to evaluate each segment from the perspective of process qualit--iccludig query formulation, retrieval focus, and local answer clarity-which are crucial for MT-GRPO credit assignment. To this end, we introduce a rubric-guided LLM-as-judge that provides dense, segment- level process scores complementary
signals. To avoid preference mismatch and calibration drift, we instantiate the judge with the same buse model family and size as the policy (frozen): this aligns inductive biases and tokenization. improves score calibra- tion the policy's 's own outputs, reduces domain-shift across updates, and is compute-efficient The judge only consumes the (question, <tool.call>, <tool.essoonee>) context; no gradients flow through it.
outcome-basee
and information-shappe
We augment turn-level credit assignment with a subric-guided LLM-as -ujdgge-udge that scores each tool- segment and injects a process credit into the MT-GRPO update. For given prompt group sample i, and token positions t I,..., T. let segments be indexed by s E (0....,S 1} for tool calls and -1 for the final non-tool segment. Denote the binary mask Mf
24

---

Published as a conference paper at ICLR 2026
(f 5} and the set of responses in the group by G(g). For each tool segment that exists sample i, we present to a judging LLM the user question and the paired block (<tool call> </tool call>, <tool response> ... </tool response>), and quest rubric scores over D criteria (e.g., factual correctness, search efficiency, clarity), yielding
q E [0, 1]:
a (q.
We collapse to scalar per-segment process score via nonnegative weight vector E RR..
W1,s w qi
U1 [0, 1].
In practice, the judging prompt uses a fixed rubric and forces a structured, per-pair rating extraction, and the weights w are chosen balance factuality and search efficiency. Groupwise normaliza- tion stabilizes scale across prompts and different numbers of tool calls, while masking ensures no auxiliary credit leaks the final (s 1) non-tool segment.
Rubries for judge
You are on ovaluation acsistant. Your goal is to acsees hov well Large language model, ueing search tools, answered userfa Factual for.
Evaluate each stool_ccall </tooo.../ooooo. cells and its tollowing stoo__eespone .tt.sse................... <//ooo__rerponsoo pair. For EACH paix, scoze the rollowing three dimensloaz or 0(2 szale (integers):
factual
0: incorrect er mislezding
1: partially correct or incomplete
2: tully correct sell-supported
ettrcheytti
0: inaffactive oz irrelevant seazch effective
sedundant
highly estective
answes_ctarity:
contusing tails underatandabie aeeds clarity structure 2: cicor, woll-orgenized, concise
soquiraneats:
provide reascning por pair structured choin-ot-thought, ziting evidence.
Then output satinga in the exact cemplace:
seasoning...
Pair 2 zeasoning...
</thinks
<answer>
halinge_hy_paiee oos, el Aoolal, pos, r2 clas))...
</answez>
25

---

Published as a conference paper at ICLR 2026
F SEGMENT-LEVEL PBRS: DEFINITIONS AND POLICY INVARIANCE
F. SEGMENT-LEVEL MDP FORMALISM
While shaping is naturally defined at the turn-level, optimization operates on tokens. We formalize the mapping for completeness. Let token-level states evolve as {st with actions Tol. (t). When retrieval completes, the environment appends an observation and defines boundary index baethe the last token index of turn k. The kth segment is the token sequence in (bx-1. bk). with segment state S == Sb.. The induced segment policy
[TT (k-1)
II t=bg-1+1
To (az St).
Thus the token process partitions into K turns, each serving unit for reward assignment.
F.2 TOKEN-LEVEL MDP AND SEGMENTIZATION
Consider Enite-horizon episodic MDP
Miok PS,
(6)
and specialize to the undiscounted case = 1. A (stochastic) token-level policy T[a s) induces trajectories {(s,, at, r=)]{=u with a, .-- 5t) and r, R(St, at, St+1).
The environment declares segment (turn) boundaries
0= ho bb ... < bK = =T.
The k-th segment (turn) is the action block
Tk (bb Ub. 1+1: -..... Obe -1) A A
(8)
and the boundary states are Within turn, the sequence probability induced by the token policy is
:= Sb, E S.
(9)
II( (
II
maa st).
(10)
tmm 6y
F.3 VALUES AND RETURNS (UNDISCOUNTED)
Let RG = R((s, Oe, 8141). The undiscounted token return from time / is
G!
(R)
Σ R.
(11)
u=t
The state-value and action-value functions under π are
(*(() = Ex (
=
(12)
(s, E, G R)
St 8, 01
(13)
An optimal policy maximizes the start-state value:
π E arg max E Esgopo [*()))
(14)
26

---

Published as a conference paper at ICLR 2026
F.4 TEACHER LIKELIHOOD POTENTIAL AND BOUNDARY SHAPING
Let Ance = (A A(M) denote the set of acceptable answers, and let
Lsc; =- log Σ
Prench (A(m) ())
(15)
m=1
Define the potential
((5) == L(5; Aace).
(16)
We apply shaping only at segment boundaries. For = 1,..., K, add
:= a (5x) - ((x-1i)
(17)
the reward on the unique transition that lands in Sk. i.e., time 1 by- 1. Thus the shaped per-step reward is
Rr + Ik:
if by 1 for some k,
Re =
(18)
Ra,
otherwise.
The corresponding shaped return is
c
((+)))
R
Σ
R
+
Σ b
(19)
u=t
1:2
We assume zero terminal potential
((kk) = 0,
(20)
which can always be enforced by subtracting a constant from 4
F.5 TURN-CONSTANT SHIFT OF RETURNS (KEY LEMMA)
Lemma. Fix any t with by-1 S < by. Under equation 17 and equation 20,
( =
Σ1 G( + a ($(Sk) (( G
(($ (21)
/=k
which is a constant with respect to the within turn token sequence Tk-
Proof. Because shaping occurs only boundaries,
T-1
K
G
(R++))
R +
Σ 1, G(R)
+ Σ Uy.
(22)
8=t
j:by-12t
j=k
By equation 17, the sum telescopes:
Σ1=a
(((S)) +(5 a = ( ((S+-)))
(23)
j=k
j=k
Using equation 20 yields equation 21. For fixed Sk-1, the additive shift does not depend on
F.6 POLICY INVARIANCE (UNDISCOUNTED EPISODIC CASE)
Theorem. Under equation 17 and equation 20, any and any taken at a time E [ba-1, bk),
QR+ (R(s,a) a 11)
(24)
Consequently, for all
arg max (s,) arg max (s)
(25)
and the of optimal policies is preserved.
Proof. Taking E-I- | s, = s, a = a] of equation 21 yields equation 24. The additive shift in hence the argmax in equation 25 is unchanged. Therefore any
equation 24 is detiooninndeeendee policy improvement step based on action comparisons (e.g., greedy, advantage-basee or policy- gradient with baselines) is unaffected, preserving optimal policies.
27

---

Published as a conference paper at ICLR 2026
Implementation note. When estimating G. Monte Carlo within turn, subtracting the constant a (S (or simply ignoring it) leaves all within turn action comparisons unchanged, so learning dynamics based on advantages or relative Q-values are unaffected by the shaping.
28

---

breRd Published as a conference paper at ICLR 2026
G SAMPLE ROLLOUTS
G. ADVANTAGE HEATMAPS
We provide few rollouts on samples from checkpoints of TIPS and PPO, along with heatmaps which display token-level advantages computed by both critics. Both actors and crities are from checkpoint at step 450. Advantages are z-score normalized before being mapped to the blue-red spectrum. Blue is negative and red is positive.
Sample 16644
Corsect (2)) Jeel Ecpezzon
Actoz Daseline critic
Actoz Info critis
<lin stazcis ayatan
<lis starci ayatea
You are a helpfel and barneesz sesimcant
You ace selptol and harmiszz essletant.
Tools
Scols
aay
fonctions
Bay
funations
cooe qoaey
usec quaey
You are provided with Eanction aigisturos within
You are provided with Sxiction aignstioos within tools 2e/ tools
<Sools be/ cools
Type
type
descciption snarck dE the wob for relocast inforeation based or the gives gesty (N, " "peresctero t "y -a jeo ", a pproeettioo by (* (eigery Miat ". cypp "sreay" 5, tiess it 1" type aacring "
description 20arch or She wob for rolovatt Information based on the giver query iT, " "paronetera Tyee obyeet 5 a propertieo to (" query Jist "y "seeyy fy ttes 1s type "asring" "Iy List
list
quer
reuoles quacy 1I, requized "t (2 query Lise 333
coazy 11,, requized "I [" query List 3133
</ tools>
</ tools
Eor each Sunction call, raturn 1808 objecs with
For each Sunction call, ratero 130z object vita
argunects
celst_cell>
<bp01_ call>
Copol 2all>
1" 1a0e " Csectioon eaaee by "arganease f i Caree -jooa
1 Jase 4 Conction waaee by "argunasto f Carge -joor-
object 33
objact 53
</t00l <lis a2di>
</tepl_ call> <118 aodi
and 5/ think firot every resessing you get now Lor. Affer reaconl 19 e if you find you
ceaerning you get
and 57 think firat every now ieforeat lon. After re30001e) / If you find you lack come krosindge you CaS call coarch eagine by
Inside
Jack ccon koovledge, you san call rearch sagine by <eto and it will rsturn the
<alol_call>
calo
</te Rnd St Hill retuen the
top
your wast : If you find turther external bnowlecge
your vact If you Eind Eurcher extRcial knowledge
reeded, you oer directly grovide the antver irside
needed you oor directly provide the anDwer fronde < Canaae > and szavez 3, vithout decadied sllestratiora Ter example
<anaver > and of 43ANNE 3, without datasled :llastratiera For exeapie,
Betzing
Belsing
played Quastion Wbat Asstralian actor /2 sbar agens
Questics What Austcallan
apeSS steminng fror his relatiosssip with the Winter UsE 4 B0d1>
sterming from his relsticamhp with the Minter H133 uang ciix addi>
<isn stars 1> ansistasz
<110 stars anslatasz
BIving KELOY (SONCTONe
sonsldorsey grestion
vostive
specifig
inforsation about
directly
Bocle4:
1 anaves B
Crow
the Austzelian actos V10
tesell
Cro4
<) aRSPEt
1he Auaczel5s4 astor VSS
Fita 131x
the vole Xon
05
The
of
fEction
and
the
348
Secton
42d
The Bectcon of StEder chacpes
The Bention of SEESKE 2N42992
Storeing Frox
Mincer H113
Eashes steeing
Vinter HEE3
she
characteeR 34 playad
actoe #isticnal charasteeR
2:ayed
annaees
4380013 Crow
</ BRINOr
anEveR barend
CroP
47 Enevez
The
the
endi>
Indocmstion related
Figure& 8. Sample failed rollout generated with PPO with advantage heatmap
29

---

Published as a conference paper at ICLR 2026
die durc sposs
Ios 1a11 tee sae fanntiees to saatey 4ith SARE qury.
In sy sell ar ar sae fasities aealey MASN LM WAY TeIy.
TV si tancise sistatane RUSH COOLEE bJ tonte 36 1931
No RUNAE with tasstics siceatores RLEME COOCEE SO tOOLE DO (933
"lactioo Prass's
iesevaal aiimiie bas
retenaat Recennic bust the tines Cae
lst Lably
Leate)
for fasmia you sesset aith faseries siee and
tor 1pm fesrii mll,, neers the ceseet ait taserion SIM and
dire oederr 1I
fiS by Jac6
linetit the mith
wisk foom fill bay dhagua
Eany
STOEREKER trare
wineer
Toople
Toeple
ane
and
TRRU THIS CNERRR Crese
Inforeation eRoDUgt LEEeE cappeted
reality
puat
Lhe yeany
yeany
slze
The
Iastne (RAM BBB
Iesenee larene Dlannsa
B (508 lasten Daaa
REVOORASS 5000 2444 BB Mas
Figure 9. Sample failed rollout generated with TIPS with advantage heatmap
G.2 TURN-LEVEL REWARDS
30

---

Published as a conference paper at ICLR 2026
Deoespoy Jess stssale stus suppasee odeas Hoke aib 5. HN 20 sskardy sliges por yay win let proiizt Lissllas stel,, Labe did
Info Actor fereline tritts
Actor Inde cestle
tea salyfal and Samblass aEst
to salptel ane samness aeststane
Teale
Taiis
relessst infommni bered esrry
eeecrver relesss inforntiin bise th
perpariiss qary sppe
passsii tagjant ppoppriia qany las sppe
toi sill tan mach rasitt for 1820
fomee
tall will setars mar cesitt for 1925
parp
miar Godd Lott cresni 31 use
case freesio Carperasee Lares for rl
ft Faaesiss securesss cares for ontern
oote gastins saa contert rassiing intite
qusriee TH mrs coterr messise insiee
2ea sall munk
line
Gnd
Ltd 10s
rand doed
por
moute bestimgs smaze. au
Baslley (Camm famiss
d
sertial ertist ais
LYES Bay Res aucuat sile
LES Aay Boy surciad statam criniral seeed thiee vicer Is ageased
stllre crisinal west shite viter is
king
DRER eXPrTS) EREEEE To eyette BEE Istersatir
STOERESTE
Staxs
martial site
1ite:
ana
atsereed testuling 14 Legens parcageed hastinise
RIVE POTORE
Figure 10. Sample successful rollout generated with TIPS with advantage heatmap
31

---

Published as a conference paper at ICLR 2026
Sample 15630
Correct (2): Tuen
Actoz Raseline critic
Actoz Info cxitis
<lia Isarci ayacea
<lis staastarciiaaeo apasea
You are a heipfei and harneesa sesiacant.
You are seipfol and haraiaza 4EsLesant.
Tools
Soole
any
sore fonctioss to Jestar with the
way
more fonetions to
ceos quaxy.
uses queey
You are provided with Fanction olgestures within
You are provided with fetction olgertires with
<ooll be/ 100l6
toolc tools.
typee"
newe
noxe desor iptiaa zoaret or the web for relovitt
type
deses iption soarch or the wob for relourt intorration based or the gives geaey (N, " "paresstera" t eyppe seeeee 7, a "prspertieo (1 (figery Jist iias "; (2 type "arring" "
Information based on the giver, quacy (5, "parenttars"" sype objeett 5. * propectieo to ( gugry Jlstt fr "xxxay"
syppa
Lies 1t type "asring"
ceecles The teell will return search rescits for exch
queriea- The cool return search results
coacy 11, requized "I [" query List 31333 </ tebls> >
quacy 2 1I, cequized "t [2 quary List *1 </ tcols
For each Sunction call, raturn jaea with
For each Suncticn call, ratera Jaoz abject vita azqunesta
acqunacta vitsin call>
<essi
</tosi_eall> Sage:
S/tesi_calls <spel_ call>
<5201_ call>
noin 41 Frrection -ooee by "argeeett ft cargs +joos
1 naee 4 foooti conttionnnoo more by Targmesto ft Carge -jsor
object 53
abjact 53
</t00l_ Clis
</tool_
resonelag
reaoeelig
cmside
and 5/ think first every cime you get
inside <hitk> and < think first every time you get
rev Incornation... After reasoring v 1f you find you
nev Information After reasonseg v If you find you
lack scee krowledee
San call coarch oagine by s/tool PEL
Trok cane: krow wwwlegee you 031 6311 noarch cagine by
<epol call>
will ratuen the
<Cpo2 call>
exll> R55 it MI22 ratzEn the
your wast.. I2 you find DO Surthar extarnal tnokingge reeded you Dar directly groride the anaver iraidn < <anavac > and </ 433N02 3, without dacasled
your VasE If you Find turthar extRctal knokindoe needed 700 car directly provide The aNSMEr Inaide < CanaMa > and of RESVEE vithout decaiied exampis.
:1:ustratieca For examp:e
2e13ing
Delsing
endi>
side_endi>>
<iia stazci> azsiatant
<lim starci> ansiatast
Ehios
Upoa caading the qu06Eio6 realsee the
Upon FORtEnN 19005101
the
psovided abous M pahe
aed the
cio proviced 10035
R00 ha 59hc
464 C5c
Ector
allat a7510g
and
confosies
Abost
find 156
26k abont
oimilar
De
actoz 1:
63582
n6 prominact
58t
clear
13ere
36
proniment
Sactial
mectil
asaociated
S4:1ed
5055
thESIsh typice:
sead
theosgh typicel
dacabasn
tezzches
758
Sifferenc
Bifferens
the
cead
siichh night
the
Ph:ch
ALghS
2413
4202bez
1ike
5954
U598
28
4002342
1ike
0558
2404 1: 0ESSA
300
and
Shores
the
Figure 11. Sample successful rollout generated with PPO with advantage heatmap
32

---

Published as a conference paper at ICLR 2026
Systes
helafal with function Lion sigatuures within <cullse/tusls RtL stoulss ("ypp"; "furclion" "function" ("ram" "searh"" "descristion" "Searches the wib for relevant information based on the gaven query." ", "paraneters": P"ypp": "object", "properties" ("euery, lisi", ("type"; "array", "descriptine ipsirn""""" List fully=/ursed seuntic quaries. sull33, "regutrest: argunents within steot_ collos/tool__om calls tags: PrtlLcatte ("hame:: stanction-neee cetto
Taols my call
fonction
will retern search resulis euery. 1ton carge-j ton-oojecto)
"strict": faise))
User
Ansver the gives evestion, You nust conduct reasening inside <thinks and </think- first every time you get now information. After reasoning, you you stoo krovledge, you can call search engine Lcall query </too Leall searched
retur external krowledge neeses, can directly provide the arower inside vanswers and </answers, without detailed stlustrations, For example, <ansver> Beijing v/arsvers, question: ferthew ceorce Frower, is an American Canadian actor, singer, voice artist and coredian, he is known partraying the villain Noloch which 2885 fewrican superbere film directed by Zack Snyder, besed on the 1986- Limited
Assistant
Reward: 4.90999311864144447
<teol_call> ("nase"; "search", "arqunents": ("query, l'Maithow George Frewer portray Noloch superterre
by
Zack
bave Glibans")>> w/too
User
cteol responses ( ("result*; "Bec (Title: Vhattcheen (ft (filnlt"rvnlattt (film) Watchmen 2089 American supertero file directed Zack Snyder, based on the 1988-87 BC Conics lirited sertes of the same same by Alan Soore and Dave G1 Gibbors. It stars as ensesale rast Akermat, Billy Crndur, Matther Gonde, Darla Gugtnn, Sackie Parle ey, Jeffrey dystopoan
history Rongan, Patrick kilson.
between the bed States and the Soviet getre,
Cold
mustly redd kerrican superhernes invstigete yngeee (fiilee sWattceen (fital "TMWES wwards seven wards at the 3617 Sature Awards, and 13 awares at the 2009 Screas awars, The fila was also pre-nonirated for the Acadery Rorrd for Best Visual Effects, although did not rake the firal stortlist, katcheen (1i0a) Notcheen is series
inerican superhera
Styder, lkk
contes
name isley,
crusup, Potthen Dean Mongan, and Patrick kilson. 4 dark satarical. and dystopian take on theinin0ss 3 (Title: \*Prodiction of Natcheen 1 inProcuction of detcmaen astemmen 15 2069 tiin based on the bee Lve-1stue graphic novel serses the come rase creased by writer Alan Moore, areist graphic novel's different
enserble
Bave Gibzons, and colorist Jonn litogins, publisshdd
rights were acquired by screanvrite iters being aachee the project through iventy years. In 2086, Zuck Snyder, who at the Lime was filainn "anather comic book. adap popiatioon, was sared by Marner bros. to bein """"atchemn""" "V" Filaing star sed in") </ =/tool Lesponies
between
helted
Reward: 1. 333325507748802224
Notthev George Frever is knowt for portraying the reciined vallain Moto in the 2009 American suporteero file ewatchmene, directed by Zock Snyder and based on the 1986-87 DC Comses Liasted series by Alan Hoore and dave 61b0ons. <answer> watchmen </answer>
Ground Truth Watchmen
Figure 12. A failure case for TIPS' reward. We see that the policy executed good, detailed tool call, whose results returned the correct answer Watchmen. However, the tool call was assigned a negative A, indicating the answer was less likely after seeing the tool results, compared to after only seeing the prompt. We do note that this may just be noise, as the absolute scale of the reward is rather small.
33

---

Published as a conference paper at ICLR 2026
H OVERHEAD ANALYSIS
H.1 INCREASE IN FLOPS
We analyze the computational overhead introduced by teacher scoring in TIPS relative to vanilla PPO. The information-theorree reward computation proceeds as follows:
def mean__ogy (context, answers):
return mean (sum (teacher (context + a) [-len (a) :] for in answers) initial_prompe
phis - [mean_ (s, answer_set).
for segment in segments:
S += segment phis.append (mean_logme (S, answer_set))
deltas = [philit1] - phi [i] for i in range (len (phis) -1)
Crucially, all values of 9 share prefixes with previous values, enabling reuse of KV cache compu- tations across forward passes. This prefix caching yields substantial FLOP savings at scale. We calculate the teacher scoring FLOPs assuming perfect prefix caching using the following formula adapted from the verl codebase (volcengine & verl contributors, 2025). The parameters are: B (batch size), L, (length of prefix i), Lmuax (maximum prefix length), S (number of prefixes per sam- ple), A (average number of candidate answers), La (average answer length), Ndense (dense parameter constant), (head dimension). H (number of attention heads), and L (number of transformer layers).
qsire Hd, kiszee Havd,
Usize
Hydd
Ndense L 3hI (ggzze Risie Usize + Hd)) + 2Vh.
The total FLOPs for teacher scoring decompose into prefix processing and answer scoring compo- nents:
T T.ma max T.j
F Ppetixx = 2N 2.Neese BLmax 4BL
2 max
dHL
8
Fans 2.eense BSAL. + 4BA (Labe + ba(la-1)) dHL
Protal Forefix + Fans
Using verl's baseline computation functions, we find the relative FLOP increase of TIPS over vanilla PPO be approximately 11% for both Qwen 2.5 3B and 7B models, detailed in Table 11.
Table 11. Comparison of PPO step FLOPs and teacher-scoring FLOPs per model.
Model
PPO (TFL.OP/stee))
Teacher scoring (TFL.OP//tepp)
Relative increase (%)
Qwen2.5 3B
64661.474
7604.648
11.761
Qwen2.5 7B
136219.934
16136.034
11.846
Qwen2.5 14B
271071.084
32013.051
11.810
Llama 3 8B
147038. 119
17369.666
11.813
Qwen3 4B
90932.211
10602.213
11.659
34

---

Published as a conference paper at ICLR 2026
Table /2. Model-specific inputs used in the teacher-scoring FL.OP equations.
Model
qsize
ksize
Usize
N Nense
Qwen2.5 3B
2048
256
256
3.397
10
Qwen2.5 7B
3584
512
512
7.615
10
Qwen2.5 14B
5120
1024
1024
1,477 x 10t0
Llama 8B
4096
1024
1024
8.030 10"
Qwen3 4B
4096
1024
1024
4.411 x 10
Table 13. Shared constants for the teacher-scoring FLOP setup.
Parameter
Value
B
256
S
5
Lmax
3676.8
Le
10.0
A
2.0
{T.)
(400.0, 1219.2, 2038,,,,,00,,,,,8,,,,,,
H.2 COMPARISON OF WALL-CLOCK TIMES
While FLOPs provide theoretical complexity measure, we also empirically evaluate the wall-time overhead of teacher scoring during TIPS training. Table reports the per-step training time with With Reward
and without the reward computation. The overhead is computed Without Reward 18%.
1, yielding 16-
Model
Without Reward (s/step)
With Reward (s/step)
Overhead (%)
Qwen 2.5 3B
30.56
35.95
17.64
Qwen 2.5 7B
93.00
108.22
16.37
Table 14. Wall-time overhead of teacher scoring during TIPS training.
For completeness, Table 15 provides direct per-step time comparison between TIPS and vanilla PPO. We emphasize that this metric is primarily determined by the average response length gen- crated by the policy rather than teacher scoring overhead, and included only for comprehensive evaluation.
H.2.1 WALL-CLOCK TIME AFTER RESPONSE-LENGTTH NORMALISATION
Raw wall-clock times scale with the length of the decoded response. To isolate the fixed overhead, therefore regress the logged per-step time the mean response length for every Weights & Biases run. An affine model
t == or + 3
fitted with intercept B8, coefficient of determination R² and the mean response length F re-estimated directly from the W&B logs, and the two fitted lines are then compared at common response lengths.
to every history row that contains both metrics. The slope a,
After this adjustment, response length alone explains 92.6 % of the Qwen 2.5-3 PPO PIPS gap and 70.6 % of the B gap. The residual differences are -2.3s (favouring TIPS) and +13.2 (penal- ising TIPS), respectively. Evaluated the PPO mean length, TIPS is only -2.1 % (3 B) and +7.7 % (7 B) away from PPO; evaluated at the TIPS mean length the gaps become - % and +13.9 %. These normalised figures match the wall-clock comparisons reported in the main text.
35

---

Published as a conference paper at ICLR 2026
Model
PPO (s/step)
TIPS (s/step)
Qwen 2.5 3B
67.62
35.95
Qwen 2.5 7B
63.39
108.22
Table 15. Raw per-step training time comparison between TIPS and PPO.
Model
Variant
a (s/tok)
(s)
R²
F (tok)
3
PPO
0.035988
18.427
0.8948
1367
3
TIPS
0.037113
15.463
0.0869
552
7B
PPO
0.039683
31.235
0.1401
810
7B
TIPS
0.050097
27.694
0.6539
1607
Table 16. Per-step regression coefficients and response-length statistics.
36


Reviews:
[{"review_id": "khNAqnKkLY", "reviewer": "Reviewer_EXdd", "summary": "The paper introduces Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of this paper is sound: it aims to introduce denser reward signals to improve RL training.\n2. The experimental results show notable improvements.", "weaknesses": "1. The writing quality needs improvement: Section 3 is hard to follow, and I could not find any mention of what exact model is as the teacher model (if I did not miss anything).\n\n2. The experimental setup seems outdated. Why not evaluate on GAIA or BrowseComp for search LLM? Likewise, why stick with the Qwen2.5 series, which is barely capable of search, instead of building upon the latest Qwen3 or other up-to-date models?\n\n3. The proposed method is heavily tied to a teacher model. A fairer baseline would be direct distillation from that same teacher.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sw5H21H2ll", "reviewer": "Reviewer_vPWX", "summary": "This paper presents TIPS (Turn-level Information-Potential Shaping), a reinforcement learning framework for optimizing search-augmented large language models in open-domain question answering. The authors identify that existing reinforcement learning methods such as PPO and GRPO depend heavily on sparse, outcome-only rewards that are only provided at the end of an episode. This design makes it difficult to assign credit accurately across multiple reasoning and retrieval steps, leading to unstable learning dynamics.\n\nTIPS addresses this limitation by introducing a turn-level dense reward mechanism. Instead of waiting for a final correctness signal, the framework measures how much each reasoning-and-retrieval step increases the teacher model’s confidence in the correct answer. This reward, based on the change in the teacher’s confidence, follows the Potential-Based Reward Shaping (PBRS) principle, which guarantees that the underlying task objective remains unchanged while providing richer learning feedback. The system therefore offers fine-grained credit assignment across reasoning turns without compromising theoretical soundness.\n\nThe authors evaluate TIPS on seven open-domain QA benchmarks (including NQ, TriviaQA, 2Wiki, MuSiQue, and HotpotQA) using Qwen-2.5 models of 3B and 7B parameters. Across tasks, TIPS consistently surpasses PPO and GRPO in both accuracy and stability, with particularly strong improvements on multi-hop reasoning datasets.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Built on potential-based reward shaping, TIPS ensures that policy invariance is maintained while providing denser feedback signals, addressing a fundamental limitation in sparse-reward reinforcement learning for language models.\n- The paper presents a well-structured pipeline—multi-turn reasoning, retrieval, teacher evaluation, and potential-based shaping—supported by consistent mathematical logic and implementation clarity.\n- Evaluations across seven QA benchmarks and two model scales demonstrate reproducible and significant improvements in EM/F1 metrics, convergence speed, and training stability.\n- The inclusion of advantage-distribution analysis and ablation studies on shaping coefficients and teacher refresh strategies provides concrete evidence of behavioral differences and system robustness.\n- The framework is directly applicable to retrieval-augmented and reasoning-intensive LLMs, making it relevant for real-world agentic settings that require iterative tool use.", "weaknesses": "- The reward signal is fully determined by the teacher model’s likelihood estimates. If the teacher is miscalibrated or biased, the shaping signal may misrepresent information gain. No calibration analysis or correction mechanism is discussed.\n- All experiments use the same teacher model (Qwen-2.5), differing only in whether it is fixed or periodically refreshed. The paper does not evaluate how the reward behaves with different teachers, leaving the robustness of TIPS to teacher variation untested.\n- The need to query the teacher model for every reasoning turn increases training cost compared to standard PPO/GRPO. The paper does not quantify the additional compute or discuss trade-offs between stability and efficiency.\n- All results are reported using EM/F1 metrics. There is no qualitative or human study to determine whether TIPS improves reasoning coherence, interpretability, or factual correctness in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4l3TcOFDQ", "reviewer": "Reviewer_VPap", "summary": "This paper introduces TIPS (Turn-Level Information-Potential Reward Shaping), a novel method to address the sparse reward and credit assignment problem in training multi-turn, search-augmented Large Language Models (LLMs). The core idea is to provide dense, turn-level rewards by measuring the increase in the likelihood of a correct answer, as evaluated by a \"teacher\" model, after each reasoning-and-retrieval turn. This information-gain signal is framed as a Potential-Based Reward Shape (PBRS), which theoretically preserves policy invariance. The authors integrate TIPS into the PPO framework and evaluate it on seven open-domain QA benchmarks. Results show that TIPS consistently outperforms strong baselines like PPO and GRPO, achieving significant improvements in Exact Match and F1 scores (e.g., +11.8% EM and +13.6% F1 for a 7B model) while demonstrating superior training stability and more effective credit assignment.", "questions": "1. Computational Cost: Could the authors provide an estimate of the computational overhead introduced by the teacher model's forward passes for calculating $I_k$? How does the wall-clock time or FLOPs for a TIPS training run compare to a standard PPO run?\n\n2. Teacher Model Selection: The paper uses the agent's initial checkpoint as the teacher. Did you experiment with using a larger, more powerful frozen model as the teacher? If so, what were the results? Conversely, what happens if the teacher is significantly weaker than the agent being trained?\n\n3. Theoretical Guarantees and Teacher Refresh: The policy invariance proof holds for a fixed potential function $\\Phi$. Your refresh@200 strategy, while empirically beneficial, technically breaks this strict invariance between refresh windows. Can you comment on the theoretical implications of this? Did you observe any instability or \"jumps\" in the return values immediately after a refresh, and how does this relate to the \"constant shift\" mentioned in the text?\n\n4. Failure Modes: The advantage distribution (Fig. 5) for TIPS is much cleaner than PPO's. However, are there specific types of tasks or reasoning steps where the information-gain reward might provide misleading guidance (e.g., when new evidence is subtly distracting but doesn't immediately lower the teacher's likelihood)? Can you provide an example of a failure case for TIPS?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The core idea of using a teacher model's likelihood of the correct answer to compute information-gain rewards is highly original. It provides a principled and automated way to generate dense supervision, distinct from heuristic rules or learned reward models.\n\nQuality: The work is of very high quality. The combination of a solid theoretical grounding (PBRS) with extensive and carefully designed empirical validation is commendable. The ablations and analysis sections are particularly thorough.\n\nClarity: The paper is a model of clarity. The writing is precise, the concepts are well-motivated, and the flow from problem statement to solution to validation is logical and easy to follow.\n\nSignificance: This work addresses a fundamental bottleneck in RLHF/RLVR for complex LLM tasks. The demonstrated improvements in performance and, crucially, in training stability, represent a significant advance. The method has the potential to become a standard component in the toolkit for training reasoning agents.", "weaknesses": "The weaknesses are minor and do not detract from the overall excellent contribution.\n\n1. Computational Overhead: While not explicitly quantified, using a teacher model (especially a 7B model) to compute log-likelihoods for every turn during training introduces non-trivial computational overhead compared to outcome-only rewards. A brief discussion of this cost (e.g., estimated % increase in training time or FLOPs) would be helpful for practitioners.\n\n2. Teacher-Student Capacity: The method assumes the teacher model is sufficiently powerful to make reliable judgments about information gain. The paper uses the agent's own initial checkpoint as the teacher. It is unclear how the method would perform if the agent significantly diverges from the teacher's knowledge distribution, or if a less capable teacher were used. An ablation with teachers of different capacities could further strengthen the work.\n\n3. Comparison to PRMs: The introduction contrasts TIPS with Process Reward Models (PRMs) by citing their need for \"high-quality supervisory labels.\" However, recent work has explored training PRMs on synthetic or LLM-generated step-level feedback. A more direct comparison or discussion on how TIPS's \"information gain\" reward compares in quality and efficiency to a learned PRM would be insightful.\n\nIt is worth noting that I am not an expert in this field, and I will adjust my score based on the scores of other reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CdDdh4nPiQ", "reviewer": "Reviewer_ajMR", "summary": "This paper introduces Turn-Level Information-Potential Reward Shaping (TIPS), a novel reinforcement learning framework designed to address the challenge of sparse rewards and difficult credit assignment in multi-turn, search-augmented LLMs. The proposed approach exploits the notion of \"information gain\" as an increase of the teacher model's log-likelihood of the correct answer. The gain in the form of log-likelihood ratio serves as Potential-Based Reward Shape (PBRS), which maintains policy invariance. The proposed approach follows a standard PPO training procedure. Experimental results on various QA tasks show that the proposed approach outperforms the comparison approach based on accuracy and training stability.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tModeling the \"information gain\" as a turn-level reward is intuitive, well-designed. It effectively addresses the critical problem of sparse credit assignment in long-horizon reasoning tasks.\n2.\tThe authors provide detailed proof of policy invariance in Appendix C, which lends strong theoretical credentials to the statement that the approach used by them has not affected the original policy.\n3.\tExtensive experiments on several QA benchmarks demonstrate the effectiveness of the proposed method.", "weaknesses": "1.\tLack of efficiency analysis. TIPS requires two forward passes of the teacher model per turn (to compute the likelihood before and after the observation) for every rollout. This introduces a non-trivial computational cost compared to outcome-only methods.\n2.\tHave you considered using a distilled, smaller model as the teacher to minimize the computational overhead?  Besides, could the \"information gain\" signal be approximated in a more computationally efficient manner?\n3.\tThe reward shaping heavily relies on a set of acceptable answers A to calculate the teacher's likelihood L(S;A). The performance might be sensitive to the completeness and quality of this answer set, especially for questions with many valid paraphrases. The robustness of TIPS on imperfect answer set hasn’t been investigated in the paper.\n4.\tThe teacher model is the key component of TIPS, but its selection and properties are not deeply explored. The paper uses the same LLM as the backbones of  both the policy model and teacher model. It is unclear how the method could perform with a teacher of different capabilities (e.g., larger, smaller, or from a different family) or if the policy and teacher drift too far apart even with periodic refreshes.\n5.\tThe authors only compare TIPS against naive RL baselines (PPO, GRPO) and their multi-turn extensions. Several key and highly relevant baselines [1-10] are missing from the main results table (Table 1), which makes it difficult to assess the true contribution of TIPS.\n\n[1] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning.\n[2] ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning.\n[3] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs.\n[4] Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL.\n[5] Stop summation: Min-form credit assignment is all process reward model needs for reasoning.\n[6] DeepRAG: Thinking to Retrieve Step by Step for Large Language Models.\n[7] Deepdive: Advancing deep search agents with knowledge graphs and multi-turn rl.\n[8] O-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering.\n[9] Search-o1: Agentic Search-Enhanced Large Reasoning Models.\n[10] R1-searcher: Incentivizing the search capability in llms via reinforcement learning.\n[11] From <Answer> to <Think>: Multidimensional Supervision of Reasoning Process for LLM Optimization.\n[12] Openprm: Building open-domain process-based reward models with preference trees.\n\n6.\tExperiments are restricted to the Qwen2.5 family. The performance and stability of TIPS haven’t been unverified on other model architectures (e.g., Llama, Deepseek) and a wider range of model scales, raising concerns about its general applicability.\n7.\tThe author claims that they used the same experiment setting as used in Search-R1. However, the reported results in Table 1 are inconsistent with those in the cited Search-R1 paper. \n8.\tMarginal Gains vs. High Cost: The performance gain is rather marginal, particularly in the case of the 3B model where it occasionally performs worse than baselines. With the added complexity and computational cost of the teacher model so high, the cost-benefit ratio does not seem favorable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JiKjJmW5YE", "reviewer": "Reviewer_jPXU", "summary": "This paper introduces TIPS, a framework designed to address the significant challenges of training search-augmented Large Language Models (LLMs) with reinforcement learning (RL). The authors identify that optimization is often unstable due to sparse rewards and difficult credit assignment across multiple reasoning steps and tool calls.TIPS mitigates this by assigning dense, turn-level rewards to each \"reasoning + tool-call segment\". This reward is calculated based on the increased likelihood of the correct answer, as computed by a teacher model, following the addition of that turn's information. By leveraging potential-based reward shaping (PBRS), the TIPS framework provides fine-grained guidance that is policy-invariant, overcoming the limitations of standard outcome-only optimization. The method was evaluated on seven open-domain question answering (QA) benchmarks. The results show that TIPS consistently outperforms PPO and GRPO baselines and substantially improves training stability.", "questions": "none", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles one of the most significant obstacles in training LLM agents: the sparse-reward and credit-assignment problem. Providing a dense, principled reward signal for intermediate reasoning steps (i.e., tool calls) is a crucial area of research.\n\n2. The core contribution—defining the reward as the change in log-likelihood of the gold answer under a teacher model—is an elegant and well-motivated approach. It directly measures the \"information potential\"  of each tool-use turn.\n\n\n3. The method is not merely a heuristic. It is formally grounded in potential-based reward shaping (PBRS) , which guarantees that the optimal policy remains unchanged (policy invariance) under standard PBRS conditions.\n\n4. TIPS consistently outperforms all PPO and GRPO-based baselines across seven diverse QA benchmarks, using both 3B and 7B models. The performance gains are particularly pronounced on difficult multi-hop and out-of-domain (OOD) tasks, which is precisely where one would expect a better credit assignment method to excel.", "weaknesses": "1. The primary weakness, acknowledged by the authors, is the computational cost.\n\n2. The method introduces a new critical hyperparameter, $\\alpha$ (the shaping scale). The ablation in Figure 6 shows that the method is highly sensitive to this value. \n\n3. The entire framework's success hinges on the availability and quality of a \"teacher model\". The paper shows that this teacher must be \"refreshed\" periodically to prevent the potential function from becoming stale, which itself causes instability. This introduces new complexities, such as determining the optimal refresh rate.\n\n4. As noted by the authors, the method is currently demonstrated only within the PPO framework. It is unclear how this segment-level PBRS approach would be integrated with other, potentially more sample-efficient, RL algorithms (e.g., off-policy methods).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)