Agentic Reinforcement Learning with Implicit Step Rewards
Authors: Xiaoqian Liu
Ke Wang
Yuchuan Wu
Fei Huang
Yongbin Li
Jianbin Jiao
Junge Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
large language model
agent
process reward
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a general credit-assignment strategy for LLM agent reinforcement learning in interactive environments with implicit step rewards.

Abstract: Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL) that reason and act in interactive environments.
However, sparse and sometimes unverifiable rewards make it extremely challenging to assign credit when training LLM agents that serve as a policy.
Recent work attempts to integrate process supervision into RL but suffers from biased annotation, reward hacking, high-variance from overly fine-grained rewards or failures when state overlap is rare.
We therefore introduce implicit step rewards for agentic RL (**iStar**), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms without relying on additional rollouts or explicit step labels.
Particularly, we alternatively optimize an implicit process reward model (PRM) with the policy model to generate step rewards for each action via a multi-turn DPO objective. Theoretical analysis shows that this learning objective produces a step-wise reward function learned from trajectory preferences.
Then the implicit step rewards are used to compute step-level advantages, which are combined with trajectory (or episode)-level advantages for policy updates, creating a self-reinforcing training loop.
We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA. 
Crucially, our method shows superior performance over frontier LLMs and strong RL baselines across domains, achieving state-of-the-art results with higher sample-efficiency and training stability.
Further analysis also demonstrates efficient exploration by **iStar** with increased rewards in both step- and episode-level while maintaining fewer steps to achieve task success.

Published as a conference paper at ICLR 2026
AGENTIC REINFORCEMENT LEARNING WITH IM- PLICIT STEP REWARDS
Xiaoqian Liu 123,, Ke Wang", Yuchuan Wu², Fei Huang', Yongbin Li², Jianbin Jiao' and Junge Zhang>l
'University of Chinese Academy of Sciences. liuxiaoqiann2emaills htt.ooiaannmmmmll..cc...e.ee. ac. "Toggyi Lab. (wk258730, shengxiu. wyc, shuide wyw wwwp///laaabbaaiinc..... com The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences. jgzhangenIpr ia. ac. cn
ABSTRACT
Large language models (LLMs) increasingly developed as autonomous agents using reinforcement learning (agentic RL) that reason and act in interactive envi- ronments. However, sparse and sometimes unverifiable rewards make extremely challenging to assign credit when training LLM agents that serve as a policy. Re- cent work attempts to integrate process supervision into RL but suffers from bi- ased annotation, reward hacking, high-variance from overly fine- fine-grained rewards or failures when state overlap is rare. We therefore introduce implicit step re- wards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms without relying on additional rollouts or explicit step labels. Particularly, we alternatively optimize implicit process re- ward model (PRM) with the policy model to generate step rewards for each action via multi-turo DPO objective. Theoretical analysis shows that this learning ob- jective produces a step-wise reward function learned from trajectory preferences. Then the implicit step rewards used compute step-level advantages, which are combined with trajectory (or episode)-level advantages for policy updates, cre- ating a reiffreinnor training loop. We evaluate our method on three challeng- ing agent benchmarks, including WebShop and VisualSokoban, as well as open- ended social interactions with unverifiable rewards in SOTOPIA. Crucially, our method shows superior performance over frontier LLMs and strong RL baselines across domains, achieving sate-oo-the-aas art results with higher sample-efficiency and training stability. Further analysis also demonstrates efficient exploration by iStar with increased rewards both step- and episode-level while maintaining fewer steps achieve task success.
INTRODUCTION
LLMs are rapidly evolving from passive generators into autonomous agents that can reason, act, and adapt strategies over long horizons, including search agents (Jin et al. 2025 OpenAl 2025a), mobile and web navigators (Furuta et al. 2024; Bai et al. 2024), software engineering assistants (Yang
let al. 2025 Wei et al.) 2025a), and social embodied intelligence (Liu et al. 2025 Lu et al.
2025
Unlike conventional RL for LLM post-training in static, single-tum tasks (Ouyang et al,
2022
Shao et al. 2024 Wang et al.) 2026), training LLM agents in interactive environments faces particular challenges: (1) rewards are typically sparse and delayed, complicating credit assignment
intermediate actions; (2) trajectories are long and Mar-Miaovi in token level, with each step consisting of a
(CoT) (Wei et al. 2022) and an executable action, inflating variance when credit is pushed to individual tokens; and (3) environments and counterparts are non-stationary, open-ended and often come with unverifiable rewards (e.g, dialogues). Consequently. trajectory- level optimization with a single outcome reward (Wang et al.) (2025) Wei et al., 20256) suffers from credit assignment, yielding high-variance policy learning, brittle exploration, and limited gains on agent tasks.
"Corresponding author
Work done during internship Tongyi Lab.

---

Published as a conference paper at ICLR 2026
Recent work has attempted solve these problems particularly through process supervision in RL. For example, Zeng et al. (2025): Zou et al. (2025); Zhang et al. (2025b) provide denser feedback at intermediate steps but require handcrafted step labels that are costly, biased, and vulnerable to
reward hacking. Generative reward models (GRMs) (e.g, ILLM-as-judge (Liu et al. 2025 /Zha ee al., (2225) that predict criticality or correctness for each step reduce annotation overhead but can be noisy and inconsistent across domains. Implicit PRMs (Yuan et al. [2025] Cui et al., 20255) help in single-turn tasks, but the token-level process rewards tend to be overly fine-grained in agent training, amplifying variance and destabilizing training as trajectories grow. Other approaches (Feng et al., 2025 Choudhury 2025) compute step-level advantages by grouping identical states, an assump- tion that fails in open-ended language environments where state overlap is rare. Together, these limitations raise a core question for agentic RL: How can we design credit-assignmett strat- egy that is label-efficient and stable, scales to multi-turn interactions, and remains robust and generalizable to (un)verifiable rewards in open-ended environments?
To address this, we propose implicit step rewards for agentic RL (iStar), a general credit-assignment strategy for LLM agents. iStar jointly trains an implicit PRM with the policy model (the LLM agent) using trajectories collected online. At each training step. the policy model generates rollouts, which are ranked by an outcome reward verifier (or model) to form positive-negative trajectory pairs. The implicit PRM is optimized via multi-turn DPO objective PRM then generates implicit step rewards for each action by measuring its relative preference over the previous policy snapshot. Since this reward is computed per step, provides dense feedback to guide exploration while staying coarse enough to keep variance under control. When training the policy model, we combine two complementary advantages: an episode-level advantage from outcome rewards and a step-level advantage from the implicit step rewards, capturing both global task success and the contribution of individual actions. iStar is compatible with various existing RL algorithms, such as GRPO (Shao al.) 2024). RLOO (Ahmadian et al. 2024). and DAPO (Yu al., 2025), without relying on annotated step labels or additional rollouts.
these pairs. The updated
Our method addresses the limitations of prior work along several dimensions: (1) iStar provides step credit without annotated labels through implicit rewards derived from a multi-turn DPO objec- tive, which is guaranteed to be step-wise reward function learned from trajectory preferences (see Section ; (2) iStar stabilizes training with multi-turn purti-tturnni RL by learning implicit rewards at the step level rather than the token-level (Cui et al. 2025); (3) iStar only relies trajectory preferences that can be sourced from (un)verifiable outcome rewards even in open-ended environments, enabling unified credit assignment across different domains.
Experiments on three challenging agent benchmarks show that iStar achieves superior performance over frontier L.MS and strong RL. baselines, achieving state-of-thh-arr results in WehShop and Visu- alSokoban. In SOTOPIA, open-ended social interactions with unverifiable rewards, iStar increases goal completion by up to 14% in self-chat and 48% when chatting with GPT-4o (OpenAl 2024). Our method can also be plugged into different RL algorithms to boost their performance. Further analysis shows higher sample efficiency and training stability of our method than vanilla RL and token-level PRM (Cui et al. 2025) baselines, as well as efficient exploration with increased rewards both step- and episode-level and fewer steps to achieve a goal. Our main contributions include:
We introduce general credit assignment strategy for LLM agentic (multi-turn) RL that alternatively optimizes an implicit step-level PRM with the policy model, creating a self- reinforeing training loop.
We propose multi-turn DPO objective optimize the implicit PRM online, which is theo- rectically guaranteed to be a step-wise reward function learned from trajectory preferences.
Experiments demonstrate superior performance of our method over baselines across var- ious environments, showing higher sample efficiency, training stability and robustness to unverifiable rewards, as well as efficient exploration during multi-turn online RL.
2 PRELIMINARIES
Task formulation. We consider the LLM agent task as a multi-step decision-making process, where the agent interacts with the environment to achieve long-term goal through sequential de- cisions given a task prompt x E p(X). At each timestep t, the agent receives an observation o
2

---

from the environment and responds with a textual action a, E VL, where denotes the token vo- cabulary and L the maximum generation length[] The environment then returns a scalar reward Tr and transitions to the next state. Until the last timestep T. the full episode consists trajectory Published as a conference paper at ICLR 2026
Multi-Step Rollouts
Environment
Outcome Implicit Process Reward Model
Observations
Outcome Reward Verifier/Model
LLM Agent
Outcome Rewards
Trajectory- based DPO Update
(Policy Model)
plog syloffohnn) (eoloe
RL Update
Step Rewards
Implicit Step Rewards
Figure 1: Overview of iStar. At each training step, an LLM agent interacts with the environ- ment to generate multi-step rollouts ranked by an outcome reward verifier (or model) to construct positive-negative trajectory pairs. These pairs are used to train an implicit PRM via multi-turn DPO objective, which generates implicit step rewards for each action produced by the agent. Fi- nally, calculate step-level advantages using the implicit step rewards and episode-level advantages using outcome rewards to optimize the L.M agent (policy model) through RI..
{[o1, 01, ri),... (or, ar, r.)). However, in real-world scenarios, rewards can be sparse and delayed, such as feedback provided only at the end of a trajectory.
RL for LLMs. RL addresses the agent task by optimizing the LLM agent me (aeloraa, .). with the objective of maximizing the expected cumulative rewards during multi-turn interactions. Policy gradient methods are usually used, such as PPO (Schulman al. 2017). GRPO (Shao al. 2024 RLOO (Ahmadian et al. (2024), and REINFORCE+ (Hu et al. 2025). These RL algorithms mainly differ in the manner of estimating advantages for policy updates. For example, PPO computes advantages with learned value function using generalized advantage estimation. GRPO and RLOO are critic-free free and form relative advantages within a group of responses for the same input prompt. GRPO centers (and often normalizes) each reward by the group mean, while RLOO uses a leave- one-out mean. REINFORCE+ instead uses aatch-normalized rewards as the baseline reward.
Implicit reward modeling. Implicit rewards have shown effectiveness in reward modeling for LLM alignment by enabling models to infer rewards without explicit labels (Ethayarajh et al. 2024 Wu al. 2025| Zhang et al. (2025a). The implicit rewards used evaluate the quality of model outputs, such as DPO (Rafallvv et al. 2024b)). Further, Rafailov et al. (2024a) demonstrates that DPO can automatically learn a Q-function. Beyond the use as reward models O-functions, recent work uses implicit token-level rewards for test-time reranking (Yuan et al.) (025) or single-turn (x(l))))) where To represents the token-level reward model
RL (Cui et al. 2025); (y() Bog (n-22)) and Tie the reference model. l/s denotes the (-th token in the response y/.
METHOD
In this section, we first provide an overview of our method with definition of implicit step rewards for agentic RI. We then present theoretical analysis to justify the learning objective of the implicit PRM in iStar produces a step-wise reward function.
Note that we prompt the LLM agent to produce a reasoning process before exccuting an action, and az refers to the whole sequence consisting of both the reasoning and the action at each timestep.
3

---

Published as a conference paper at ICLR 2026
Step
Siep
Stup .. Step
Add
Te
(o
A trt
Fale
A (n5)
A" (x)
Tw(oF)
4" (ol')
(sv)
Figure 2: The credit-assignmen strategy of iSta.. In our method, episode-level advantages A computed using outcome rewards (T), while step-level advantages A are calculated based on implicit step rewards T (a) produced by the implicit PRM. The final advantages for policy up- dates is combination of these two-level advantages.
3.1 OVERVIEW
In iStar, there is an implicit PRM optimized alternately with the policy model (LLM agent), trans- forming the tendency to prefer more optimal actions into dense step rewards to guide exploration and improvement of the agent. Figure I shows the overall training pipeline of our method. The alternating optimization between the implicit PRM and the policy model creates self-reinforeing training loop to iteratively enhance each other. Below we first provide a definition of the implicit step rewards, and detail the training process for the implicit PRM and the policy model, respectively. Please refer to Appendix [A for the detailed algorithm of iStar.
Implicit step rewards. Let T
(01, 0i,... OT: oT) denote trajectory produced by the LLM agent using policy xo. For action at in the trajectory at step i, its implicit step reward is defined as
To rolo ax) = B log aa(a 01:4,
(1)
no (0l001..2)) x))
where Th represents the implicit PRM, New refers to the previous snapshot of the policy Te, and 3 E [0. 1] is temperature that scales the reward. The implicit step reward measures how much more probable the current action under the freshly learned PRM than under the old policy. Positive values indicate actions that Te believes to be responsible for recent improvements, while negative values highlight actions that should be discouraged.
Optimizing implicit PRM via multi-turn DPO online. For scalable online RL, we train the implicit PRM T on Dositive-negativv trajectory pairs sampled by the old policy model Th and derive a multi-turn DPO objective for online update:
JPRM (1) -B ,7 JoRTA
log (Blog
T xx) - B log
Ted (T x)) To ( x)) )
Tin (7+
x))
xxp((X)
where a is the logistic sigmoid, and T&. Taw as well as 3 follows Eq. [T[(x) refers to the product of action probabilities controlled by the policy (i.e., II. (a, 01:1, T.). - positive trajectory that is preferred to the negative one r-, both of which are labeled by outcome reward verifier
(or model) P Particularly, Eq. has two main differences from the standard DPO (Rafailov et al., 2024b): (1) we use the previous policy snapshot Tew as the reference model, whose parameters alter during training, instead of the initial policy model that keeps frozen; (2) we derive the DPO objective from multi-step Markov decision process (MDP) rather than one-step bandit problem (Rafailov) for for the justification that Eq. 2 is multi-turn version of standard DPO that equivalents Bradley-Teene Terry (BT) model with step-wise reward function.
jt al. 2024b). See Section 3.2
Policy learning with implicit step rewards. We use GRPO (Sao et al., 2024) as an example to illustrate how integrate our implicit step rewards into policy learning, despite that our method is compatible with variety of RI. algorithms (Ahmadian et al. 2024 Hu et al., 2025 Yu et al.)
2025 Zheng et al. 2025). As shown in Figure 2 for each task prompt x, we sample group of
2 In our experiments on WebShop and VisualSokoban, positive trajectories are those with success rates above 0, while for SOTOPIA, positive trajectories have goal completion score above 6.
4

---

Published as a conference paper at ICLR 2026
N trajectories (ri, .... TN) from the old policy Teas: and obtain its corresponding outcome rewards (ro(T)).... Nol ((TT)) through an outcome reward verifier (or model). We then compute episode-level advantages A for the group of trajectories:
(T)) = (ra(T:) mean mean((.)) (Ra)) (ld((R))
(3)
where Ro is the set of outcome rewards of N trajectories To (T( (Ti); = Next, we use the latest implicit PRM & (from the previous training step) to obtain implicit step reward for each action oj via Eq. and compute step-level advantages:
A (at) (ra(a)) mean(R.)) RRa)
where R, = (aj)), is the set of all step rewards in the N trajectories. Particularly, when we use a group of trajectories starting from the same initial state (task prompt), we can generate various counterfactual scenarios. This helps us calculate a more accurate and stable estimate of the state- value baselines, leading to a better measure of the advantage for ai. On the other hand, in a single trajectory, actions happen in different states and are influenced by noise specific the policy used by the agent, leading to advantages in high variance.
Given episodc-level advantages and step-level advantages, we combine them for policy updates: A(a:) = A E (T.) + aA4()
(5) where a is hyperparameter that balances the two-level advantages. The final advantage can differ- entiate not only between favorable and unfavorable trajectories but also beneficial and detrimental steps within a group of trajectories from the same initial state, enabling more dense rewards for policy learning in long-horizons.
Ultimately, given the advantages A(at), the policy model no is trained using surrogate objective that is widely used [Schulman et al, (2017); Shao et al. (2024):
T:
1
min (po ((a(a))A(a)))c) clip (po(a)), (1)), 1+ A(ai)) A(aE))
Jolicc (0) E (n m
(6)
NT
xxp(X)
i=1
xx(a))o(,x) where po(at) au [f] is the importance sampling ratio at the step-level, and is hyperpa- rameter that controls the clipping range of the importance. Particularly, the step-level importance sampling ratio aligns well with our implicit step rewards to ensure low-variance training noise on multi-step rollouts, similar to (Zheng et al. 2025). We also choose not to apply any KL--divergence penalty to the policy model during training. In an online, agentic RL setting, successful behavior may need to deviate substantially from what frozen language model would normally produce. Re- moving the KL penalty allows the policy model to explore the action space more freely, especially in regions that are crucial for solving the task. See Table 7 for performance comparison with and without the KL-divergence penalty.
Remarks. Alternating optimization between the implicit PRM and the policy model establishes training loop that enhances stability and accelerates convergence. In particular, using rollouts pro- duced by the current policy to train both models keeps their data distributions roughly consistent, minimizing off-policy bias and covariate shift. This consistency keeps implicit step rewards cali- brated to the agent's 's behavior, yielding dense and low- www-variance credit signals and preventing over- or underfitting the implicit PRM to ensure stable training. The result is a relf-reinforcing loop: im- proved policies yield better preference data, refining the implicit PRM, which in turn delivers more accurate implicit step rewards to guide the policy.
3.2 THEORETICAL ANALYSIS
We now justify that the learning objective of our implicit PRM in Eq. 2 is multi-turn version of standard DPO, which is equivalent to a BT model with a step-wise reward function. Formally, for any trajectory pair {T; = ((ot, ) 1 satisfying of = of (the same task prompt r), we have
T
Ta(a x)
7t
T Tz 2)
3 log P(r > T2) = o Σg
3 log
T0am (at of x)
Th.a (a of x)
T
T2
Σ
r (of. at)
-
r (oi az)
0
5

---

Published as a conference paper at ICLR 2026
where 73 denotes the optimal implicit PRM. This is multi-tumn version of standard DPO similar to Rafailov et al. (2024a), but we consider each action sequence as an optimized unit in the MDP instead of the token-level MDP. In addition, we only calculate the loss on the action tokens while excluding the log probability ratios of the tokens generated by the environment. Based on this multi- turn version, Eq. 7 shows that the learning objective of our implict PRM in Eq, 2 is equivalent to a BT model with step-wise reward function:
(3(a 01:1
(o 0z) = B
(8)
Toai (ax 01:0: 2))
This is also variant of token-level dpo objective which has been proved to be equivalent to BT model with a tosen-wise reward function (see Zhong et al. (2025) for proof details).
4 EXPERIMENTS
We evaluate iStar across a variety of agentic tasks to demonstrate: (1) its effectiveness in training LLM agents for long-horizon reasoning and acting: (2) high sample efficiency and training stability given implicit step rewards; (3) improved exploration efficiency evidenced by increased rewards and fewer steps; and (4) the core components of iStar for credit assignment in agentic RL.
Benchmarks. We evaluate LLM agents in three challenging environments: (1) WebShop (Yao) ctt al., 2022), a text-based web environment where the agent interacts with HTML-based website to search, nevigate, and purchase an item given an user instruction, requiring multi-step decision making: (2) VisualSokoban (Schrader 2018) with 6 x 6 size, a puzzle game where the agent has to push all boxes targets, requiring spatial reasoning and long-term planning over both visual and textual inputs; (3) SOTOPIA (Zhou et al. 2024), an open-ended social interaction environment where the agent interacts with another L.M agent given social scenario, role profiles and private goals, requiring reasoning over the other agent's 's real-time strategies. During training, we use scenar- ios from SOTOPIA-T (Wang et al. 2024b). See Appendix B for more details of the environments.
Baselines. We compare our method against range of competitive baselines: (1) prompting LLMs specialized in general-puroose reasoning: GPT-5 (OpenAl 2025b), Gemini-2... -25-ii (Google 2024), DeepSeek-R1 (Guo et al., 2025), and Slaude-Sonnet-44 -hinning (Anthropic. 2025): ; (2) vanil laa RL methods that only use outcome rewards: PPO (Schulman et al. 2017). GRPO (Shao et al. 2024), 2025); (3) a recent single-turn RL
RLOO (Ahmadian et al. (024) and REINFORCE++ (Hu et al.
method: PRIME (Cui et al. 2025) that introduces token-level process rewards for policy learning;
and (4) recent agentic RL algorithm: GiGPO (Feng et al. 2025) that computes step-level advan-
tages via same-state grouping.
Evaluation. For WebShop and VisualSokoban, we adopt Success Rate and Score (only for Web- Shop) as the evaluation metrics following |eng et al. (2025). These metrics are computed over validation instances and select the best score for comparison. For SOTOPIA, we report goal com- pletion score ranging from 0 to 10, which is evaluated by GPT-4o as proxy for human judgement following Zhou et al. 22024. We set the tempature to 0 for the LLM judge. Refer to Appendix C.2 for evaluation prompts used in each environment.
Implementation details. In our experiments, the implicit PRM is initialized from the base policy model by default. However, in VisualSokoban, we use Qwen2.5-VL--7 --T7--Inttt as the base model for the policy model and Qwwe2.55-BBIInstuun constant learning rate 5 x 10-? 7 for the policy model and 10- for the implicit PRM with Adam optimizer. Both the policy model and the implicit PRM use a batch size of 64 and micro-batch size
as the base model for the implicit PRM. We use a
0.05 for the implicit PRM following (Yuan et al. 2025). Performance with varying a and ( is shown in Table
We set the advantage coefficient a = 1.0 following (Feng et al. 2025) and
The rollout size is set to 8 per prompt. All methods share the identical RL configurations for fair comparison, and all experiments are run A100 GPUs. Additional training details for each environment are shown in C.I

---

Published as a conference paper at ICLR 2026
Table 1: Performance on WebShop and VisualSokoban. Qwen2.5--B-Instrruc and Qwen2.5- VL-7B-Instruct serve as the base models for the policy model in WebShop and VisualSokoban, respectively. Note that Deepseek-FR RI and PPO training not currently support multi-modal scenar- ios, and PRIME is only applicable to tasks with binary outcome rewards. Results are averaged over three random seeds.
WebShop
VisualSokoban Success
Method
Success
Score
Prompting frontier LLMs (ReAct)
GPT-5
37.5
66.1
16.6
Gemini-2.5. 2.5-Pro DeepSeek-R1
30.5
38.4
16.0
29.3
39.8
35.2
62.0
19.1
Base Model (ReAct)
21.5
47.3
14.1
PPO
78.2 + 4.5
86.6 1.1
GRPO
8.11
8
85.6 2.8
RLOO
7.14
87.6 4.7
86.3 0.6
REINFORCE++ 2025
85.8 3.9
8.8
PRIME (Cui et al.
1.8
9.6
GiGPO (Feng 2025
4..1
91.2 1.5
85.9 + 2.6
RLOO w/ iStar
8.8
93.6 - 1.0
91.7 + 1.2
4.1 MAIN RESULTS
Performance on benchmarks. Table showcases iStar's superior performance over baselines in WebShop and VisualSokoban, with notable gains on the latter, where RL algorithms struggle with irreversible mistakes and limited foresight. Similar trends oceur with much smaller base models demonstrated in Table 5 Specifically, surpasses recent multi-turn RL baseline GiGPO (Feng et al. 2025) by enabling finer-grained credit assignment, distinguishing good and bad actions with implicit rewards for each step than relying on same-state grouping. Our method also outperforms recent single-turn RL baseline PRIME (Cui et al. 2023) that uses token-level process rewards, which provides overly fine ame-grained rewards that complicate policy training in multi-turn RL with high-variance (see Figure (agure In SOTOPIA, where GiGPO and PRIME inapplicable due open-ended state space and unverifiable rewards, Table that iStar still achieves state-of-the-- the performance. Particularly, compared to vanilla RL baselines, our method improves goal completion hard social scenarios by 14% (7.92 - 8.06) in self-chat, and up to 48% (6.68 + 7.16) increase when chatting with GPT-4o. This demonstrates the generalizability of our method a wide variety of interactive environments.
method achieves ttate-of-the-art performance and
iStar using different vanilla RL algorithms. Since our method is compatible with various RL methods, we also evaluate it using different RL algorithms. including RLOO (Ahmadian et al, 2024), REINFORCE+ (Hu et al. 2025) and GRPO (Shao et al. 2024), and compare each to its vanilla version that only uses outcome rewards. As shown in Figure iStar consistently improves vanilla RL methods by integrating implicit step rewards into multi-turn n RL to improve credit assign- ment in long-horizons. For example, iStar with RLOO obtains substantial gains of 6.3% in success rate both WebShop and VisualSokoban, Similar trends occur with REINFORCE++ and GRPO, demonstrating the robustness of our method to diverse RL algorithms and environments.
4.2 IN-DEPTH ANALYSIS
Sample efficiency and training stability. Figure 4 illustrates that compared to baselines, iStar achieves faster improvement and higher final performance in validation metrics during multi-turn RL, demonstrating superior sample efficiency. Particularly, in Figure 4a), our method achieves the score of vanilla RLOO in WebShop in just 105 steps, around 2 improvement in training efficiency. By 165 steps, we reach the highest score of 94.7%, showcasing significant improvements both efficiency and performance. Notably, while PRIME exhibits comparable early-stage performance

---

Published as a conference paper at ICLR 2026
Table 2: Performance on Sotopia. 'Self-Chat" refers dialogues where the model under evaluation interacts with itself, while (GPT-44--s-Parrtne denotes interactions between the model and GPT- 4o. "Goal" refers to the goal completion rate (on scale of 0-10). The "Hard" subset comprises test scenarios in SOTOPIA that require advanced reasoning capabilities, and "All" represents the complete test set. Results are averaged over three random seeds.
Self-Chat
GPT--4-as-Partter
Method
Goal (Hard)
Goal (All)
Goal (Hard) Goal (All)
Prompting frontier LLMs (ReAct)
GPT-5
7.21
8.95
7.70
8.90
Gemini-2.5 Pro
6.74
8.27
7.43
8.41
DeepSeek-RI
6.98
8.56
7.30
8.44
6.39
8.64
7.02
8.62
Owen2.5-7B-Insttree (ReAct)
5.56
6.77
5.51
7.30
PPO
6.63 = 0.24
0.009
0.14
8.07 + 0.08
GRPO
0.24
8.31 0.06
6.42
7.84
RLOO
5.70 0.16
7.03
6.09
7.77 0.03
+REINFORCE++
6.17 + 0.30
7
7.93 + 0.09
+ -GROO w/ iStar
7.11 + 0.19
8.42 + 0.03
6.76
8.36 + 0.03
Llama3.1
(ReAct)
5.89
6.95
5.82
7.43
PPO
0.14
9.05
9
6.64 0.03
8.14 + 0.01
GRPO
7.92 0.08
9.12
6.68 +
4.04
RLOO
6.48 - 0.15
8.33
6.14
8.02 = 0.06
RREINFORC++
0.14
0.04
6.38 + 0.23
7.99 0.10
GRPO w/ iStar
8.06 = 0.11
9.20 + 0.03
7.16 + 0.14
8.45 = 0.03
WebShop
WebShop
MisualSoioban
113-
Success Rotte (%6
Success Rate (%
Score
6.7
67
100% CONTOREE wile caem
8.0% vittrr MarOOcEe NS
GERO w/tA
R100 m teer FOW OREELA witer OFE
Figure 3: Compatibility verification of iStar with various RL algorithms. Qwenn.5-7BBInntteeee and Qwen2.5-VL-77 LL-B--ntut serve as the base models for the policy model WebShop and Visu- alSokoban, respectively. Results are reported based on random seed.
on WebShop, its growth stagnates and experiences sharp fluctuations. This is because overly fine- grained process rewards token-level will complicate policy learning in long-horizon interactions, which usually involve much longer sequences than single-turn tasks. In contrast, iStar continues to improve consistently, suggesting that our implicit step rewards provide dense feedback for explo- ration while being sufficiently coarse to reduce variance for stable training.
With increased compute, our method's advantages sample efficiency and training stability not only persist but become more pronounced. As shown in Figure [T0 and Figure TT] its performance improvements remain consistently stable as training progresses. In contrast, baselines such as vanilla RLOO and GiGPO become increasingly unstable and can even degrade with more training steps. In addition, although our method introduces an implicit PRM that adds some computation per training step. this overhead relatively small since policy rollouts dominate the total training cost. Our method can also reduce per-step computation owing to its shorter trajectories (see Figure (d)-((d)) Consequently, our method generally achieves better performance than the baselines for the same amount of GPU hours as illustrated in Figure
8

---

Published as a conference paper at ICLR 2026
Veedhhpp
8,00 Dar
Succors Ratu P50
50000
Suncos
TILO vn Slat
0270
(8)
Stap
(b)
(c)
Figure 4:
Validation performance (10-step moving) during RL in WebShop and Visual-
Sokoban. Note that PRIME can only be applied tasks with binary outcome rewards.
(b1
(C)
Figure 5: Training dynamics (10-step moving) of iStar in WebShop and VisualSokoban, (a)-(b): Dynamics of the episode and implicit step rewards during RL training by our method. (c)-(d): The episode length versus training step compared baselines.
Exploration efficiency. To demonstrate the implicit step rewards provide useful guidance for pol- icy learning, we visualize the evolution of step- and episode-level rewards in Figure (5a) and (b). In particular, the implicit step reward improves very early (especially in VisualSokoban) and then the episode reward follows, indicating that our method first captures good local action heuristics and then composes them into higher-return trajectories. Even when may generate lower-quality rollouts in the carly stage of RL, our implicit PRM still remains stable and effective shown in Figure g A key consequence is that unnecessary actions reduced during multi-turn interactions, resulting in shorter episode lengths. As shown Figure 5]c) and (d), episode lengths decrease without compromising task success, evidenced by the consistent increase in episode rewards.
smaller base model that
Table 3: Ablation studies on core components of iStar. "RLO0": only outcome rewards are used compute advantages for policy updates. "w/ environmental process rewards": use raw step re- wards provided by VisualSokoban to calculate step-level advantages. "w/ merged rewards": implicit step rewards are added directly to outcome rewards before advantage computation. "w/ token-level process rewards": the implicit PRM produces rewards for each token along the entire trajectory rather than step-level rewards for each action sequence. Results are reported using one seed.
WebShop Score
VisualSokoban Success
Method
Success
RLOO
76.6
84.2
85.9
w/ environmetal process rewards
87.5
w/ merged rewards
81.3
90.7
88.3
w/ leken-level process rewards
82.0
90.0
89.1
w/ iStar
89.1
94.7
93.0
4.3
ABLATION STUDIES
presents results of ablation experiments validate the necessity of key components of our method for effective credit assignment. First, raw step penalties provided in VisualSokoban show limited improvement over vanilla RL, suggesting that the implicit step rewards learned by iStar are superior credit signals for policy learning. Second, merging the implicit step rewards into episode rewards obtains gains over vanilla RI. but the improvement are modest compared to our method.
Table 3

---

Published as a conference paper at ICLR 2026
This indicates that we should not only reward intermediate actions but also gate credit by final task success to prevent speculative reward exploitation. Therefore, combining signals at the advantage level is crucial to credit assignment in long- nong-hoizzons. Third, learning token-level process rewards is sub-optimal to multi-turn RI., suggesting that overly fine-grained rewards may introduce noise and thus increase the difficulty of policy learning.
We also compare our method to training the policy on SOTOPIA using multi-turn DPO alone. As shown in T vanilla GRPO already substantially outperforms both online and offiine multi-turn DPO. Building on this, our method further improves performance by using both step and episode rewards, enabling more effective credit assignment in multi-turn RL.
RELATED WORK
PRMs has been widely explored in single-turn tasks, such as mathematical reasoning and single- shot code generation. In these settings. PRMs are used to score intermediate steps for test-time search or reranking (Lightman et al. 2024 Wang et al. 2024a Mahan al., 2024). or for online- RL (Dou et al. 2024] [Setlur et al. 2025| Zha et al. (025). Moving to dynamic, interactive tasks, process rewards are usually constructed three ways: (I) handcrafted step rewards assigned to tool
execution (Zeng et al. 2025) or meta-reasoning tags (Zhang et al., 2025b); (2) GRMs (Liu et al.) 2025
Zou et al. 2025) that label step quality; (3) and implicit PRMS Yuan et al. 2025) that produce
leven-level process rewards.
However, manually-designed or judge-based step labels are costly and biased, suffering from reward hacking. Learning step Q-values (Choudhury| 2025) can reduce the bias but fixed PRM may poorly estimate Q-values for unseen actions during inference. PRIME(Cui et al. 2025) partially addresses these issues by jointly training implicit PRM with the generator. Yet, with overly fine-grained rewards in token-level, it introduces noises and destabilizes training in multi-turn RL. Another issue with PRIME is that it applies a cross-entropy loss to optimize the implicit PRM, which is only applicable to tasks with binary outcome rewards. Instead of learning a PRM, Feng et al.) (2025) addresses credit assignment by computing step-level advantages via same-state grouping. While effective in tasks with finite state-action space, it relies on exact state overlaps and cannot be generalized to open-ended language environments where same state rare. In contrast, our method learns implicit rewards at the step-level through multi-turn DPO objective, offering lahel-efficient step rewards with low-variance in long-horizon RL and showing robustness and generalization to (un)verifiable rewards even in open-ended environments.
6 CONCLUSION AND DISCUSSION
We propose iStar, a general credit-assignment strategy for LLM agents. In particular, we alter- natively optimize an implicit PRM with the policy model, which provides implicit step rewards to guide policy learning and results in a self reif-reinn training loop. To optimize the implicit PRM in an online setting, we propose a multi-turn DPO objective that is guaranteed to be equivalent to a BT model with step-wise reward function. In practice, our method can be plugged into a variety of RL. algorithms and generalized to (un)verifiable rewards even in open-ended environments. Em- pirical results show that iStar achieves averaged 86.5% success and 93.6% score on WebShop, and reaches 91.7% success on VisualSokoban. In SOTOPIA, our method improves goal completion in hard scenarios by up to 14% in self-chat and 48% against GPT-4o.
In the future, our method can also be validated in math problems or code generation to provide implicit step rewards for intermediate CoT steps. can also be applied to test-time scaling for search guidance. Regarding the limitations, we currently separate the impleit PRM from the policy model during training, which however, can be unified model trained with different objectives to reduce computation memory and potentially improve representation sharing. Additionally, in SOTOPIA, our implicit PRM is trained to only predict goal-completion preferences, while future work could be extended to multi-objective implicit PRMs.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
The development of autonomous LLM agents capable of independent thinking and decision-making in interactive environment including those involving human users, offer transformative potential to real-world applications but could also introduce some ethical concerns. First, if improperly de- signed or monitored, such an agent may exhibit behaviors that prioritize success metrics at the expense of human values and safety. For instance, an LLM agent operating dynamic interactions may inadvertently take actions that conflict with ethical principles, reinforce biases, or neglect the broader societal and interpersonal impacts of its decisions. These risks highlight the need for inte- grating rigorous safeguards to ensure alignment with human-centric values. Second, commitment to transparency, accountability, and inclusivity in the training and evaluation of these LLM agents is essential to ensuring they exhibit empathy, integrity, and faimess in reasoning and acting. By em- bedding ethical principles into the development process, it is expected to mitigate potential harms and harness the potential of these technologies for the benefit of individuals and the society.
REPRODUCIBILITY STATEMENT
To reproduce experiments, provide the source code at http://tttuubcooooorreeee ChaneemmChhattterrr///aaaaaaaCaa
Regarding the theoretical analysis in Sec- tion 3.2 since recent work (Zhong et al. 2025) has provided a detailed proof in token-wise setup, which is a variant of our step-wise justification, we do not elaborate more in this paper. For specific proof details, please refer to Zhong et al. (2025). With respect to training data, training pipeline relies solely on initial task prompts, as the policy model autonomously generates online rollouts during interactions with the environment (including user simulator in SOTOPIA). Therefore, we do not require a fixed offline dataset or dedicated data processing step. See Appendix C.2 for the detailed task prompt templates used in our experiments.
ACKNOWLEDGEMENTS
This work is supported by National Key R&D Program of China (2055))01220033) National Key Laboratory under Grant 241-HF-D08, and in part by the Beijing Major Science and Technology Project under Contract no. Z251 100008425009.
We gratefully acknowledge the Qwen-Characke Team at Tongyi Lab for their support in providing computing resourees and funding for this work.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in L.MSs. Lun- Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248-12267, Bangkok, Thailand, August 2024. Association for Computational Linguistics, doi: 10.18653//1/2024aacll
URL httpps://aclaatthllgyy/
2rg/2024.acl-1omm 662/
Anthropic. Introducing claude 4. Technical report, Anthropic, 2025. URL https://www.
anthropic..co//newws// wtwp///ww///aaeee4......1
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Di- giRL: Training in-the-wild device-control agents with autonomous reinforcement learning. In
The Tiigtt-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https://openneviiee wwt.ee/ette.ooor/////oxx/xxx///
Sanjiban Choudhury. Process reward models for Ilm agents: Practical framework and directions.
arXiv preprint arXiv: 10325, 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu. Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455. 2025.
11

---

Published as a conference paper at ICLR 2026
Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, and Xuan- jing Huang. StepCoder: Improving code generation with reinforcement learning from compiler feedback. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 4571-4585, Bangkok, Thailand, August 2024, Association for Computational Linguistics, doi: URL / /ocassshoooll org/2022/ acl-long. 251/
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model align- ment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning. 2024.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in group policy optimization for I1m agent training. arXiv preprint arXiv:2505. 10978, 2025.
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane rnstruetion-iietuned
Gu, and Izzeddin Gur. Multimodal web navigation with instruction finetuned foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL https://
openreview wwt.:eet/foooo
Google. Introducing gemini 2.0: our new ai model for the agentic era. Technical re-
port, Google, 2024. URL httpps//bloo. http:://bloo..ooglee///////// ttgp/////eccooolooooooooeeeeeeee
google-gemini- wateeeiiiin--uaaaeeeeeeeeeeeeeee -eeilateeddeemeer--22e http:///wtttttt.ooo.e/ee/ooo/ooa content==fceo-mmesaag
referral@utm campaign=gdmaae
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv: wrXiv:2402.04792 2024.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.0322 2025.
Bowen Jin Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling, 2025. URL https:
Aobo Kong, Wentao Ma, Shiwan Zhao, Yongbin Li, Yuchuan Wu, Ke Wang, Xiaoqian Liu, Qicheng Li, Yong Qin, and Fei Huang. SDPO: Segment-level direct preference optimization for social agents. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pp. 12403, Vienna, Austria, July 2025. Association for Com- putational Linguisties. ISBN 879-8-89118 1188889977-222 17-2251--.
doi: 10. 16653//12002.acl-ooo
URL
acl long. 607/
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2024. httphtttss///ppnreeee Forum?2idvvLL0pNEEBD
net/
Xiaoqian Liu, Ke Wang, Yongbin Li, Yuchuan WU, Wentao Ma, Aobo Kong, Fei Huang, Jian- bin Jiao, and Junge Zhang. EPO: Explicit policy optimization for strategic reasoning in L.Mss via reinforcement learning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pp. 15371-15399 Vienna, Aus- tria, July 2025, Association for Computational Linguistics. ISBN 979-8-89176-251- 1188865311222/0 1tw.1//.c22-2///// acl--log... 747. URL
doi: org 2025. acl 1on..
12

---

Published as a conference paper at ICLR 2026
Guanxing Lu, Wenkai Guo, Chubin Zhang. Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719 2025.
Dakota Mahan, Duy Phung, Rafael Rafailov, Chase Blagden, nathan lile, Louis Castricato, Jan-Philipp Franken, Chelsea Finn, and Alon Albalak. Generative reward models. ArXiv, abs/2410.12832,, 2024. URL http:https:/aat
Meta. Introducing llama 3.1: Our most capable models date. Technical report, Meta, 2024. URL
https://ai.met. meta. wwt.aaa.ooooooo///e.eee.eoooooce wwt.://oooooooo//eaaaaaaeae-eeee
OpenAl. Hello gpt-4o. Technical report, OpenAl, 2024. URL http:///gpenal.com/idde//
hello -ptoo-.
OpenAL. Introducing deep research. Technical report, OpenAI, 2025a. URL https://openai. com/ wtwpr/roddcccidddeeeeeeeeeeeeeee
OpenAl. Introducing gpt-5. Technical report, OpenAl, 2025b. URL https://openai.com
2h- Der-aansscW//meeiiiinntrddcceee
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From Sr5 to Sq Your language model is secretly a q-function. In First Conference on Language Modeling. 2024a. URL https:
//openreview. wwt wwtp://w.neeeoooomm/dd/eeEeettte
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Max-Philipp B. Schrader. gym-sokoban. httt::///ggtuub..cc/mm//crrrrrre
2018. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In The Thirteenth International Conference Learning Representations, 2025. URL http:////eeeeeeeeeeeeeeeee/er/rd
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Fahim Tajwar, Yiding Jiang, Abitha Thankaraj. Sumaita Sadia Rahman, J Zico Kolter, Jeff Schnei- der, and Russ Salakhutdinov. Training a generally curious agent. In Forty-second Interna- fional Conference on Machine Learning, 2025. URL http:////pppeeeeeeeeeeeffrrreree UeB3Hdrhda
Minzheng Wang, Yongbin Li, Haobo Wang, Xinghua Zhang, Nan Xu, Bingli Wu, Fei Huang, Haiyang Yu, and Wenji Mao. Adaptive social learning via mode policy optimization for lan- guage agents. In The Fourteenth International Conference on Learning Representations, 2026. UtRl:httpss///peeee http:///ppeeeeeeeeeeeesee httppprrwveeeetetoooooiiic/ooooo
13

---

Published as a conference paper at ICLR 2026
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- Math-shepherd: Verify and reinforce LLMs step-by-step without human anno- tations. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 1426-9439 Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v1/202444 2a1/2022...1-oon.551000 URL https:///ccaattooooggy org. wrg//024. acwpe4n..l-ooo...... 110/
Ruiyi Wang. Haofei Yu, Wenxin Sharon Zhang, Zhengyang Qi. Maarten Sap. Graham Neubig, Yonatan Bisk, and Hao Zhu. Sotopia-T: Interactive learning of socially intelligent language agents. In Annual Meeting of the Association for Computational Linguistics, 2024b.
seaat..smanticsscii org wrg. wrwg/orpgsssD....oooooo (orppss1:22837963333 268379635
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL. VjQLMeSB_J
Yuxiang Wei, Olivier Duchenne, Jade Copet. Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing Ilm reasoning via rein- forcement learning open software evolution. arXiv preprint arXiv:2502. /8449, 2025a.
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang. Bing Yin, Hyokun Yun, and Lihong Li. Webagent-rl: Training web agents via end- to-end multi-turn reinforcement learning. ArXiv, abs/2505.16421, 2025b. URL httpps://ppi.
semanticscholar wtt.:larrrrrg/rrroIIIDIII 2187:27777777
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play pref- erence optimization for language model alignment. In The Thirteenth International Conference on Learming Representations, 2025. URL https::/oppenreiieett net/ wttf:/oormmmmddammmmmma
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxin Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412.151115, 2024. URL /ttpps:/appi. semanticscholar .rw. wttpg//CrpusI:::::0 274859421
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint M0KV::25055093888, 2025.
Shunyu Yao. Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-2073 2022.
Shunyu Yao. Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL hhttps://openreview.nn www.se/.wwefooormmmiiic
WE WE_
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Honglin Yu, Weinan Dai, Yuxuan Song, Xiang Wei, Haodong Zhou, Jingjing Liu, Wei
14

---

Published as a conference paper at ICLR 2026
Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yong- Xu Wu, and Mingxuan Wang. Dapo: An open- source Ilm reinforcement learning system at scale. ArXiv, 14476,, 2025. URL https:
seattssemanticschoooar wwtp://roo/ooss:2000. 2771:4 2777104112
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. In Forty-second Interna- tional Conference on Machine Learning, 2025. URL http::///pparevieee.eet//ooommmp 8ThnPFhGm8
Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. Re- inforcing multi-turn reasoning in Ilm agents via turn-level credit assignment. arXiv preprint arXiv:2505.11821, 2025.
Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-Was Wei Hong, Duane S Boning, and Dina Katabi. RI tango: Reinforcing generator and verifier together for language reasoning. arXiv preprint arXiv:2505. 15034, 2025.
Yuheng Zhang, Dian Yu, Baolin Peng. Linfeng Song, Ye Tian, Mingyue Huo, Nan Jiang. Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning LLMs with general preferences via no-regret learning. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://pppeeeeeeeee
Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. Rlymr: Reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents. arXiv preprint wrXiv22507.22844 2025b.
Chujie Zheng, Shixuan Liu, Mingze Li, Xionghui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang. Jingren Zhou, and Junyang Lin. Group sequence policy optimiza- tion. ArXiv, abs/2507.1800 18071, 2025. URL https://api. pettss//ap..smmatiicchhoo.. wtt.:oorr...r//oouuusoo..o
280017753
Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. DPO meets PPO: Reinforced token optimization for RLHF. In Forty-second International Conference Machine Learning, 2025. httptppp///oppeeeieeeeeeeee/e//e id=IFWKVFGLFY
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi. Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SOTOPIA: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2024. URL https:/// wwt.:e//ooounnddd//rrrrbbba
Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. Reasonflux- prm: awaae prms for long chain-of-thought reasoning in 1lms. ArXiv, 2its/2556.188966 2025. URL HtLLhhtps::/// seattt/sscooarrrroo///o/oooo//0p
15

---

Published as a conference paper at ICLR 2026
A ALGORITHM
The algorithm flow iStar is detailed Algorithm
Algorithm 1: Training LLM Agents with iStar (GRPO as an example)
Input:
Task distribution p( X), language model a outcome reward verifier or model To. training steps M, rollout size N, mixing weight
Output: Optimized policy Te and PRM TTe
Initialize policy model TT f- Thmm "oata Tom PRM T f- TTmu
for iteration = 1,.., .M do
// Multi-step rollouts collection
Sample task x ~ p(X) and initialize N identical environments for ( 1,.,,, do
Sample actions {a} ( x)
Execute actions and observe next observation (0/41 ).M
// PRM training
Compute outcome rewards for N trajectories: (o(T).))
Forward pass Te based on trajectory preferences to obtain step reward ra (aj) with Eq.
Update PRM Te on trajectories using a DPO-style objective in Eq.
// Policy training
Compute episode-level advantages A (
(T)) using (T:) via Eg
Compute step-level advantages A (a)) using ra (aa) via Eq.
Combine advantages: A(a)) = A (T. + (A)
Update policy Na by maximizing objective in Eq.
Update old parameters: Ootd +
B ENVIRONMENT DETAILS
WebShop. WebShop Yao et al. 2022) simulates online shopping tasks on an e-commerce plat- form, where the agent's objective to interpret human-provided text instructions and purchase product that aligns with the given specifications. To accomplish this, the agent must interact with the website's search engine, select items review from the search results, examine their descrip- tions and details, and choose relevant options (e.g., size, color) before finalizing the purchase by clicking the "Buy" button. To identify the best product that fulfills the user's requirements, the agent may need to compare multiple products, navigate back and forth between pages, and conduct ad- ditional searches necessary. The environment includes over one million products sourced from amazon.com com, more than 12,000 crowd-soureed instructions, and rich set of semantic actions. At the end of each interaction, a binary reward is provided to indicate whether the task is successfully completed. The final score for each task is automatically calculated using programmatic matching functions that evaluate attributes, type, options, and price of the selected product.
VisualSokoban. Sokoban (Schrader 2018) consists of rooms composed of five key elements: walls, floors, boxes, box targets, and an agent. These elements may exist in different states de- pending whether they overlap with box target. Rooms are randomly generated, which helps prevent models from overfitting to specific predefined layouts. The game includes two primary ac- tions, Push and Move, which can be performed four directions: Up. Down, Left, and Right. The Move action allows the agent to proceed to an empty space in the specified direction, provided there no wall or box blocking the path. The Push action attempts to move an adjacent box, but only if the field behind the box is empty; chain pushing of multiple boxes is not allowed. If no box is adjacent, the Push action functions identically to the Move action the same direction. Success- fully completing the game by pushing all boxes onto their targets yields reward of 10 points on the final step. Additionally, pushing a box onto a target grants a reward of point, while removing a box from a target results in penalty of -1 point. Each step ineurs small penalty of -0.1 points to discourage trajectories with many steps. VisualSokoban renders visuals RGB, with the pixel size equal the grid size.
16

---

Published as a conference paper at ICLR 2026
SOTOPIA. SOTOPIA (Zhou et al. 2024) is general -omean,, open-ended platform simulate social interactions between LLM agents. The scenarios span diverse array social interaction types, such negotiation, exchange, collaboration, competition, accommodation, and persuasion. particularly challenging subset, known as SOTOPIA-hard involves scenarios requiring advanced strategic reasoning. Each agent is defined by character profiles, encompassing attributes like name, gender, personality, and profession. At the end of each dialogue, agents are evaluated by GPT-4o as proxy for human judgment across seven dimensions: Goal Completion, Believability, Knowl- edge, Secret, Relationship, Social Rules, Financial and Material Benefits. SOTOPIA-T (Wang et al., 2024b) a follow-up work that uses GPT-4 generate new set of scenarios. The social tasks in SOTOPIA-T are guaranteed to be entirely distinct from those in SOTOPIA.
C EXPERIMENT DETAILS
C.I TRAINING DETAILS
(Ynn et al. (024) as the base models for policy learning. To address invalid actions produced by LLM agents, reward penalty of -0.1 is imposed. The maximum response length 512 tokens, while the maximum prompt length is 4096 tokens in WebShop and 1024 tokens in VisualSokoban. We sample 16 different groups per rollout in WebShop, resulting in a total of 16x8 = 128 environments. In VisualSokoban, we sample 256 environments. Instead, PPO
WebShop and VisualSokoban. We use (0wnn..5-(VL)-7B-BBB
32 different groups per rollout, resulting in total of 32 uses 128 and 256 separate environments for rollouts in WebShop and VisualSokoban, respectively. The rollout temperature is set to 1.0, while the validation temperature is set to 0.4. We implement experiments in veRL (Sheng et al. 2024), each for 200 training steps.
(Yang et al. 2024) and Llama3. 1-8B-Instruct (Meta 2024) as the base models for policy learning demonstrate the robustness of our method to different model backbones. The maximum prompt length is 6144 tokens and the maximum response length is 2048 tokens. As with WebShop, sample 16 different groups per rollout in WebShop. resulting in a total of 16x8- = 128 environments (PPO uses 128 separate environments). The rollout temparature set to 0.7. Each experiment implemented in veRI. consists of 800 training steps.
SOTOPIA. We use Qwen2.5-7B-Instee
Q
Success
Score
0.5
0.05
79.7
89.4
0.8
0.05
84.4
91.3
1.0
0.05
89.1
94.7
1.2
0.05
85.9
92.6
2.0
0.05
78.1
90.7
1.0
0.1
82.8
91.8
1.0
0.5
85.2
94.7
Table Performance of our method on WebShop with varying a and 3. 1inn2..-7B-Instruue used the base model.
C.2 EVALUATION PROMPTS
We use ReAct (ao et al. 2023) as the prompting strategy, with chain-of-thought (Wei et al. 2022) generated before each action. The prompt templates used for evaluating LLM or multimodal large language model (MLLM) agents in the environments presented in Figure 88 Placeholders enclosed in curly braces(() ( represent semantic slots, which are dynamically populated at runtime.
17

---

Published as a conference paper at ICLR 2026
Prompt template for WebShop
You are expert aulonomous agent operating WebShop e-commennce environment. Your is to: (task description).
Prior this stop, you have already taken {step_ count) step(s).
Below ure the most recent thistory_ _engghhh_hhhh____ observations and the corresponding actions you took: (action history) You are now at step (current step) and your current observation is: {current observation).
Your admissible actions of the current situation are:
favailable actions]
Now it's your take action the current step. You should first reason stee-by-step about
situation,
then carefully which admissible action best advances shopping goal.
This reasoning process MUST be enclosed within <think> </think> tags.
Once you've ve finished your rensoning, you should choose an admissible action for current step. and present it within <action> </action" tags.
Figure 6: Evaluation prompts used for the LLM agent in WebShop.
Prompt template for Vokoal
You expert agent operating in the Sokoban environment.
Your goal is push all the boxes the target spots. Once all boxes are the targets, you win!
# Rules
You can only push boxes. can't pull them, so plan ahcad avoid getting stuck. You can't walk through or push boxes into walls.
To avoid traps, do not push boxes into comers or against walls where they can't be moved again.
# Visual Elements in the Image:
Character: small, green alien-like figure with antennae and black eyes. It represents you. Box: A yellow crate marked with orange "X" across its front. It is the box you need to push. Target: A black tile outlined red, with small red diamond shape center. marks the destination
box should pushed.
# Current Step
Your current observation shown in the image: <image> Your admissible actions are ["up", "down", "loft", "right").
Now it's your turn to make move (choose ONE action only for the current step).
You should first reason step-by-step about the current situation observe the positions of boxes and targets, plan path push box toward target, and avoid traps like comerr walls.
This reasoning process MUST enclosed within <think> <ithink> tags.
Once you've finished your reasoning, you should choose admissible action current step, and present within <action" </action"> tags.
Figure 7: Evaluation prompts used for the MLLM agent in VisualSokoban.
D ADDITIONAL RESULTS
D.1 RESULTS ON SMALLER LLMS
Table presents results of our method compared to baselines on WebShop using Qwen2.5-1.5B- Instruct Yang et al. 2024) as the base model. The results show that our method can also benefit base models with weaker reasoning capabilities. Furthermore, Figure demonstrates that even the weaker base model generates low-quality rollouts (indicated by low episode rewards) in the early stage of RL, our implicit PRM still remains stable and effective with steadily increasing step rewards and episode rewards throughout the training.
18

---

Published as a conference paper at ICLR 2026
Prompt template for SOTOPIA
Imaginc (agent). your task is to act/speak fagent) would, keeping in mind [agent)'s social goal. You find [agent]'s is goal (ur background) in the 'elr the context of this interaction' field. Note that {agent]'s goal is only visible you.
You should try your best achieve {agent,'s goal in way that align with their character traits. Additionally, maintaining
conversation's naturalness and realism essential (e.g., not repeat what other people has already said before).
You are at Turn [tum). You can say something to interact or just say 'left the conversation' to stop continuing. Nole: You can 'left the conversation' if I. you have achieved your social goals, 2. this conversation makes you uncomfortable, 3. find uninteresting or lose patience, 4, for other reasons you want to leave.
You should first reason step-by-step to reflect on the current state of the dialogue, then think carefully what commuiccttio and social strategies best advances your goal.
This reasoning process MUST be enclosed within <think </think>
Once you've finished your reasoning, provide your response and present within <action>> </action> lags.
Figure 8: Evaluation prompts used for LLM agents in SOTOPIA.
Table Performance on WebShop with Swwen2.5-1.5B-Isstrue the base model. Results
reported using one seed.
Method
Success
Score
Qwen2.5-1.5B-Innst (ReAct)
10.4
46.1
RLOO
71.9
85.7
GiGPO
72.7
86.8
PRIME
74.2
86.6
RLOO w/ iStar
80.5
91.5
WebShop
Implicit Step Reward (mean)
Episode Reward (mean)
n1
0.0
-01
-02
-03
-44
-0
Episode Reward
Implicit Step Reward
-0.
12s
-07
Step
Figure 9: Training dynamics (10-step moving) of iStar using Qwen2.551.5B.Insttre as the base model on WebShop. The left vertical axis illustrates the trend of trajectory-leva outcome rewards during RL, while the right vertical axis represents the progression of implicit step rewards.
D.2 RESULTS WITH MORE COMPUTE
Figures 10 and [I show that the sample-efficiency and stability advantages of our method not only persist but grow with increased compute. As training steps increase, our method yields larger and more consistent performance gains, whereas baselines such as vanilla RLOO and GiGPO become increasingly unstable and can even degrade in performance over time.
Regarding GPU hours (Figure 12). our method still outperforms the baselines in both sample ef- ficiency and training stability, even though the implicit PRM adds some computation per training
19

---

Published as a conference paper at ICLR 2026
step. This overhead is relatively small because policy rollouts dominate the total training cost, and our shorter trajectories can further reduce per-step computation (see Figure
(d)). (d))
WedrShup
Whllhhop
Vis Butbbben
Succsss Nako: 1511
1761
Sc0re 159
RL.0
Step
Stop
50op
Figure 10: Validation performance comparison (10-step moving) on WebShop and VisualSokoban with more training steps.
WeaShop
WeaShop
Visua/Solobaan
08913 ml Mal
Store (86)
Figure 11: Validation performance comparison (10-step moving) between iStar and vanilla GRPO on WebShop and VisualSokoban.
WhbShoop
WebShop
VisualSokotan
Score CN0
CaTo
GEFD
GANO
BL00 vi Blar
R100 v (Sur
1L00 Cla
1000
GPU
Figure 12: with respect to GPU hours.
Validation performance comparsion (10-step moving) on WebShop and VisualSokoban
Table 6: Performance comparison between iStar and DPO in self-chat settings of SOTOPIA. -- is used as the base model. We apply supervised fine-tuning (SFT) before multi-turn DPO, whereas GRPO and iStar do not require this step.
Method
Goal (Hard)
Goal (All)
SFT + DPO (offline)
6.40
8.29
SFT + DPO (online)
6.68
8.46
GRPO
7.92
9.12
GRPO w/ iStar
8.06
9.20
D.3 PERFORMANCE COMPARISON AGAINST USING MULTI-TURN DPO TO OPTIMIZE THE POLICY MODEL
To evaluate our weshhod-hiee integrates an implicit PRM optimized via DPO into online RL-we compare it against using multi-turn DPO alone to train the policy on SOTOPIA, In the offline setting, we use the datasets proposed by Kong et al. 2025) and follow Rafailov et al. 2024a) : Tajwar et al. 2025) to perform multi-turn DPO at the trajectory level. For online DPO, trajectory preference pairs are constructed online using outcome rewards produced by the LLM--s-judge in SOTOPIA, and optimized by multi-turn DPO following Guo al. (2024).
20

---

Published as a conference paper at ICLR 2026
Table 7: Performance comparison WebShop with and without the KL-divergence penalty. Qwen2.5-7B-Instree is used as the base model.
Method
Success
Score
RLOO w/ iStar
89.1
94.7
RLOO w/ iStar & KL
82.8
92.0
Results in shows that vanilla GRPO already significantly outperforms both online and offline multi-turn DPO. Our method further improves performance by leveraging both step and episode rewards achieve better credit assignment in multi-tur RL.
D.4 PERFORMANCE COMPARISON WITH AND WITHOUT KL-DDVERGENCE PENALTY
Table 7 compares performance WebShop with and without the KL-divergence penalty. Dropping the KL penalty improves performance, supporting our hypothesis that removing it allows the policy to better explore the action space in LLM agent tasks.
21


Reviews:
[{"review_id": "W5YNUjynQN", "reviewer": "Reviewer_gXoH", "summary": "This paper tackles the problem of sparse and unverifiable rewards for agentic LLMs. It proposed a credit-assignment strategy for LLM agents that introduces step-wise implicit step rewards learned by alternately training a PRM and the policy. The authors highlight that their approach does not need explicit reward annotations, can stabilize training with step-level rewards, and only relies on trajectory preferences.", "questions": "1. Can the authors elaborate on why dropping the KL divergence penalty when training the policy model? \n\n\n2. How sensitive is the model performance to the hyperparameters $\\alpha$ and $\\beta$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is well motivated, introducing step-level reward for agentic tasks to avoid the training instability caused by either token-level reward signal being too dense or outcome reward being too sparse.   \n\n2. The proposed method is flexible as it can be easily integrated into existing RL frameworks like GRPO, RLOO, etc.", "weaknesses": "1. The idea of step-level PRM has been studied in previous work[1]. \n\n\n2. The model introduces two hyperparameters: $\\beta$ as reward temperature and $\\alpha$ that balances step level and episode level advantages. A sensitivity sweep and guidance for setting them are important for practical adoption of the proposed method.\n\n\n[1] Yuan, Lifan, et al. \"Free process rewards without process labels.\" arXiv preprint arXiv:2412.01981 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "17S9Y1Gz1g", "reviewer": "Reviewer_Jbze", "summary": "The paper \"Agentic Reinforcement Learning with Implicit Step Rewards\" proposes a new RL-based optimization process for finetuning LLMs.   Inspired from implict rewards models that can be derivated from the DPO formulation, authors proposed a mechanism to train a step-wise function, that can be used to efficiently guide classical RL approaches, such as GRPO. Experimental results indicate the good behavior of the proposed learning scheme.", "questions": "- For multi-turn RL settings, approaches such as POAD [] propose to modulate the discount factor at the utterace level rather than at a token level, with adapted critics. Are you doing something similar ? \n\n- Although originally designed for imitation learning rather than reinforcement learning, some GAN-based methods introduce token-level discrimination mechanisms that closely mirror several aspects of the implicit reward modeling proposed here. Notably, I think to [2], which considers an objective $\\dfrac{\\pi_\\theta D_\\phi}{\\pi_{\\theta^{old}}} \\approx \\dfrac{\\pi*}{\\pi_{\\theta^{old}}} \\dfrac{\\pi_\\theta}{\\pi_{\\theta^{old}} + \\pi*}$, where $D_\\phi$ is a discriminator trained to distinguish good from bad utterances at a token-level, $\\pi*$ is the optimal policy and $\\pi_{\\theta^{old}}$ is the policy $\\pi_\\theta$ from the previous iteration. This looks very close to your training scheme. I think you should discuss the proximity with this work (for a different setting but this could be adapted) in your paper.\n\n- The theoretical analysis section is fully taken from Zhong et al. The only difference looks to lie in the use of $\\pi_{old}$ rather than $\\pi_{ref}$, but this is the same at that point. I think authors should remove that part and only mention zhong when introducing their reward function. \n\n- I understand that the reward modelling is based on works that implicitly consider reward models through policy ratios. But in your case you train a reward model based on a policy fully dedicated for this. Is it still implicit ? For me this implicit denomination should be removed from title and many places of the paper, as it is not. You explicitely build a model of the reward. \n\n- From table 3, I am not sure what is new in w/iStar row compared to the w/ token-level process rewards. Please specify. \n\n- The authors use GRPO, which does not seem particularly suitable for handling step-wise (or token-level) rewards. In particular, I have concerns with the use of a global normalization in (4), that normalize all rewards from all timesteps and all trajectories from the group. Different rewards from different situations play at different scales. I feel there is a risk that only dominating rewards (not in the sense of a advantage over other tokens in different situations, only because for instance any correct sentence must contain the corresponding token)  have an impact on the training, hindering all the contribution from other positive markers in different situations. Wouldn't it be more relevant to use PPO with a trained critic to compute advantages ? \n\n\n- you say that one of the two main difference with DPO is that \"preferences are learned from pairwise trajectories\". Going back to the DPO paper I cannot see the difference regarding that aspect. Please clarify. \n\n[1] Wen, M., Wan, Z., Zhang, W., Wang, J., & Wen, Y. (2024). Reinforcing language agents via policy optimization with action decomposition. arXiv preprint arXiv:2405.15821.\n\n[2] Lamprier, S., Scialom, T., Chaffin, A., Claveau, V., Kijak, E., Staiano, J., & Piwowarski, B. (2022, June). Generative cooperative networks for natural language generation. In International Conference on Machine Learning (pp. 11891-11905). PMLR.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Good Experimental results: Authors demonstrate the superiority of their approach on various tasks compared to different baselines. Differents ablations also allow to highlight the benefits of the different components of the approach. \n\n- Acting at a token-level is crucial for complex tasks\n\n- Well written paper, easy to follow", "weaknesses": "- Rather straightforward approach that takes most of its contribution from zhong et al (for the definition of reward models at a token level). The main novelty is to me the use of a $\\pi_{old}$ policy that is periodically updated rather than a constant reference policy. I think this is not a fully innovative contribution, as many works attempted to make evolve ths reference during training. \n\n- The use of GRPO with adavanges computed over any steps and trajectories in the group looks questionnable to me (see questions above).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZUIhBzFXV", "reviewer": "Reviewer_rhUJ", "summary": "This paper proposes iStar, a general credit-assignment strategy for LLM-based agents in reinforcement learning. The method introduces implicit step rewards, which provide dense yet stable feedback signals without explicit step-level annotations. iStar jointly trains a policy model (the LLM agent) and an implicit process reward model (PRM) through trajectory-based direct preference optimization (DPO). The PRM generates implicit step rewards by comparing current actions to those from a previous policy snapshot, providing step-wise credit signals that are aggregated with episode-level advantages. The authors theoretically show that this learning objective is equivalent to a Bradley–Terry model with a step-wise reward function. Empirical results on WebShop, VisualSokoban, and SOTOPIA demonstrate that iStar improves performance and sample efficiency over several reinforcement learning baselines (PPO, GRPO, RLOO, REINFORCE++) and achieves state-of-the-art results. Ablations confirm that implicit step rewards enhance exploration and stability compared to token-level or handcrafted process rewards.", "questions": "1. How does iStar perform when using base models with weaker reasoning capabilities, and can the implicit reward model remain stable and effective when learning from low-quality rollouts?\n2. what conditions—either theoretical or empirical—ensure that the learned implicit step rewards remain unbiased and consistent throughout training?\n3. How robust is the framework when dealing with environments where outcome rewards are highly stochastic or unverifiable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of implicit step rewards with trajectory-level DPO represents a creative approach to balancing dense feedback with low variance in long-horizon RL. It avoids the pitfalls of both sparse end-of-trajectory rewards and noisy token-level signals.\n2. The method achieves consistent improvements across diverse benchmarks, and this indicates good adaptability of the framework.\n3. A solid theoretical proof is provided to ground the method. The equivalence proof connecting the DPO objective to a step-wise Bradley–Terry model adds credibility to the approach and helps ground the intuition behind.", "weaknesses": "1.  The framework’s performance likely depends on the capability of the base model, the nature of the task environment, and the quality of outcome reward verifiers. These dependencies are not theoretically analyzed, and the paper provides limited discussion about how iStar generalizes under varying conditions. This limits the method’s theoretical generalization and reproducibility.\n2.  Some key components, such as the implicit PRM architecture, reward verifier setup, and hyperparameter tuning strategy (e.g., α, β), are insufficiently detailed. While the authors mention releasing code, more explicit descriptions or pseudo-code would strengthen reproducibility and credibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XkduyiswfQ", "reviewer": "Reviewer_m1yj", "summary": "Online Reinforcement Learning (RL) has recently emerged as a powerful technique to improve LLMs’ capabilities in a wide range of tasks ranging from single step reasoning problems like math or coding to multi-turn agentic tasks such as web navigation/tool use. For multi-turn tasks where agents need to interact with an environment multiple times to successfully complete their objective, online RL is a promising approach. A key problem in this scenario is credit assignment — many tasks often have sparse or trajectory level reward only available at the end of the trajectory, and this makes it difficult to learn whether the intermediate actions taken by the agent are good or bad. This paper proposes a method for learning implicit step level rewards and leverage it for online RL training. Specifically, the proposed method, iStar, uses DPO-style log-likelihood ratio as an implicit reward for intermediate steps, and combines it with trajectory level reward (or advantage) to produce per step learning signal. The method is validated on multi-turn tasks like WebShop and VisualSokoban, showing gains over online RL methods using just trajectory level reward signal.", "questions": "1. The implicit PRM is parametrized by $\\phi$ (denoted by $\\pi_\\phi$), whereas the policy is parametrized by $\\theta$ (denoted by $\\pi_\\theta$). In practice, what do these symbols mean? What model is used as the implicit PRM? Is $\\phi$ different from $\\theta$?\n\n2. Why not just perform multi-turn DPO using trajectory level preference pairs constructed using task success/outcome level reward, but in an online fashion? Prior works [1, 2] has already used some variant of multi-turn DPO for sequential decision making tasks, albeit in the offline manner. I would like to see both the online/offline version of this as a baseline, and compare against [1, 2].\n\n# References\n\n[1] Your Language Model is Secretly a Q-Function, https://arxiv.org/abs/2404.12358\n\n[2] Training a Generally Curious Agent, https://arxiv.org/abs/2502.17543", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is interesting. There is prior work that has used the DPO-style log-likelihood ratio as reward (and advantage), but I have not seen it being used as advantage in an online RL framework.\n\n2. The paper is well-written and easy to follow.", "weaknesses": "(**Results are not strong enough**)\n\nThe gains we see in the method are not significant enough (in my opinion) compared to the level of complexity introduced in the paper. Specifically, if we focus on the results presented in Figure 4, the difference between the much simpler RLOO baseline and iStar is not significant enough. Moreover, to me it seems like the RLOO validation curve still has a positive gradient near the end of training, and hence more training steps might flip the order of performance between RLOO and iStar.\n\nMoreover, RLOO is a less commonly used RL algorithm. Could the authors present the learning curves from GRPO and GRPO with iStar?\n\n(**Comparison at more training steps**)\n\nBased on the webshop and visual sokoban results in Figure 4, could the authors train for 300 steps (instead of 200) for all the baselines? I am not convinced that the gains may hold as one puts more compute in the baseline vs the proposed method, and would like to see longer training curves.\n\n(**Comparison at GPU hours/FLOP level**)\n\nDue to the training of the implicit PRM, which I believe is a separate model (correct me if I am wrong), iStar would spend significantly more compute compared to regular GRPO/RLOO. Hence, performance comparison at gradient step level is not a fair comparison. I would require a plot with\n\n1. X-axis: GPU hours or total FLOPs\n\n2. Y-axis: Validation performance\n\nWhere iStart shows superior performance over the baselines in order to increase my score of this paper.\n\n(**Why does PPO perform so poorly?**)\n\nIn Table 1, PPO performs notably poorly, despite using the same idea — a trained critic function to estimate step level advantage. Could the authors explain this result?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)