Group Verification-based Policy Optimization for Interactive Coding Agents
Authors: Silong Dai
Changzhi Sun
Haolun Wu
Huanran Zheng
Tao Ji
Junchi Yan
Yuanbin Wu
Dell Zhang
Xiaoling Wang
Xuelong Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model
Tool Learning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advancements in reinforcement learning from verifiable rewards (RLVR), particularly through Group Relative Policy Optimization (GRPO), have significantly improved the capabilities of large language models (LLMs) for interactive coding agents.
However, these methods overlook process-verifiable environment feedback (e.g., code execution failures), leading to inaccurate advantage estimation at each reasoning step and insufficient learning.
To address this issue, we propose Group Verification-based Policy Optimization (GVPO), a novel RL algorithm that introduces an advantage shaping framework integrating both outcome-verifiable and process-verifiable signals.
While outcome-verifiable rewards ensure alignment with long-term task objectives, process-verifiable feedback derived from intermediate execution traces (e.g., syntax errors, runtime exceptions) serves as corrective shaping terms at the step level.
By jointly leveraging these two forms of verifiability, GVPO achieves more accurate credit assignment, balancing short-term process guidance with long-term outcome alignment.
This unified formulation yields more stable optimization, faster convergence, and stronger generalization in complex interactive environments.
A 32B-parameter agent trained with GVPO in the AppWorld environment outperforms OpenAI’s o1 agent by 12.7\% on the more challenging Test-C split and surpasses the strongest 32B RL-trained state-of-the-art baseline by 3.7\%.

Published as a conference paper at ICLR 2026
GROUP VERIFICATION-BASER POLICY OPTIMIZATION FOR INTERACTIVE CODING AGENTS
Silong 1, Changzhi Sun *2.1, Haolun Wu', Huanran Zheng', Tao Ji Junchi Yan", Yuanbin Wu-.t, Dell Zhang", Xiaoling Wang".,, Xuelonn Lia.t
Department of Computer Science and Technology, East China Normal University
Institute of Artificial Intelligence (TeleAI), China Telecom Coolleee of Foreign Languages and Literatures, Fudan University
"School of Artificial Intelligence, Shanghai Jiao Tong University (sldai dstuu, ybwuêcs, xlwangecs cs} .er.ccesuuuceee..m........ ecnu. edu. {dell.z, xuelong linggllliiieeee seee.eeeooo. org
ABSTRACT
Recent advancements in reinforcement learning from verifiable rewards (RLVR). particularly through Group Relative Policy Optimization (GRPO), have signifi- cantly improved the capabilities of large language models (LLMs) for interactive coding agents. However, these methods overlook vreiess-veriiiable environment feedback (e.g., code execution failures), leading to inaccurate advantage estima- tion at each reasoning step and insufficient learning. To address this issue, we propose Group Verification-basee Policy Optimization (GVPO), a novel RL algo- rithm that introduces advantage shaping framework integrating both outcome- verifiable and process-veriie verifiable signals. While outcome- outcome-verifiabe verifiable rewards en- sure alignment with long-term task objectives, periess-verifiable feedback derived from intermediate execution traces (e.g.. syntax errors, runtime exceptions) serves corrective shaping terms the step level. By jointly leveraging these two forms of verifiability, GVPO achieves more accurate credit assignment, balancing short- term process guidance with long-term outcome alignment. This unified formula- tion yields more stable optimization, faster convergence, and stronger generaliza- tion in complex interactive environments. A 32B-parameter agent trained with GVPO in the App environment outperforms OpenAl's ol agent by 12.7% on the more challenging Test-C split and surpasses the strongest 32B RI--trained state-of-ther the-art art baseline by 3.7%.
INTRODUCTION
Large language models (LLMs) have recently demonstrated remarkable progress in understanding. reasoning, embodied interaction, and code generation, positioning them as promising candidates for interactive coding agents (Chen et al.) 2025b; Li et al. 2024 Song et al. 2025 Chen et al. 2025a), and aligning with the broader vision of AI Flow that emphasizes scalable and collaborative intelli- gence (An et al. 2026). A key challenge, however, lies in training these agents to operate reliably in complex environments where they must engage in multi-turn interactions, plan dynamically, and generate exccutable code to achieve user-specified goals. Reinforcement Icarning from verifiable rewards (RLVR) has emerged as a powerful paradigm to address this challenge (Guo et al.) 2025), as it enables scalable supervision without costly human annotations by leveraging deterministic sig- nals. Among RLVR approaches, Group Relative Policy Optimization (GRPO) (Shao et al.) (024) has proven particularly effective, significantly advancing the performance of LLM-based agents (Li
et al. 2025 Yu et al., (025).
Despite these successes, current RLVR methods still exhibit critical limitations. Most notably, they rely almost exclusively on outcome-veriee veriifaaeee rewards, such as exact answer matching, While such
Equul contribution; Work done while this author an intern TeleAT; Corresponding authors.

---

Published as a conference paper at ICLR 2026
Advantage Function
Method
Clip
Aggr.
Calcu.
Traj.
Step
Unbias.
Out.
Proc.
RLOO Ahmadian et al.
2024
Sym.
smim
GR
x
X
GRPO (Shao
2024
Sym.
GR
/
/
/
X
Dr.GROO Liu
al. 2025
Sym.
smus
GR
/
/
/
X
DAPO
Yu et al. 2025
Asy.
Im
GR
/
/
X
LOOP
Chen et al.
2025b
Sym.
smim
GR
X
X
GVPO (Ours)
Asy.
Shaping
X
V
Table Comparison of RLVR methods in LLMs. Clip: Sym. = symmetric clipping (single c for both sides); Asy. asymmetric clipping (separate high/low bounds, with explicit "clip-high" control). Aggr: loss aggregation scheme, where smim sequuence-mean-tooen---eee token-sum, and tm token-mean. Calcu. (Calculation): GR group-rclative advantage; Shaping -utcome+proeesss advantage shaping. Advantage Function: Traj. trajectory-Iovel: Step = step- level. Unbias. whether the estimator preserves the unbiased (zero-mean property. Out. whether outcome-verifiabl rewards are incorporated; Proc. = whether process-
smly meuunce-mman--
process-verifiable verifiable signals are incor- porated. GVPO is the only method that integrates both outcome- and process-level rewards through advantage shaping, achieves step-level credit assignment, and employs asymmetric clipping.
rewards faithfully capture task-level correctness, they are inherently sparse and delayed, offering little guidance during the intermediate steps of reasoning. As result, credit assignment becomes inaccurate; early-stage errors may still receive positive reinforcement if the final outcome succeeds, while partially correct reasoning may be discarded if the trajectory ultimately fails. This issue leads to unstable optimization, slow convergence, and underutilization of valuable environment feedback.
One underexplored direction is the integration of process-verifialle signals, intermediate feedback derived from execution traces such as syntax errors, runtime exceptions, partial unit test results. Unlike outcome-based signals, process feedhack dense, fine-grained, and deterministic, provid- ing rich supervision at the token or step level. However, existing RLVR methods (Yu et al. 20255) including GRPO and its variants (Liu et al. 2025). do not incorporate such signals into their learning framework, thereby missing opportunities for more precise credit assignment and error correction.
In this paper, we introduce Group Verification-basse Policy Optimization (GVPO), novel re- inforcement learning algorithm that addresses this gap through advantage shaping frame- work. GVPO extends group-baser based policy optimization by integrating both outcome-verifin verifiable and process-veriffable signals into the advantage function. Specifically, verifiblee rewards sure that learning remains aligned with long-term task objectives, while process-verifiabl signals act corrective shaping terms that adjust step-level credit assignment in real time. This design mitigates the risk of reinforcing error-prone reasoning patterns and amplifies partial successes, ef- fectively balancing short-term guidance with long-term alignment. Tab. presents comparison of RLVR methods in LLMs.
We validate GVPO in AppWorld. challenging benchmark environment where agents must solve long-horizon, multi-turn tasks spanning multiple applications and APIs. Our experiments show that a 32B-parameter agent trained with GVPO outperforms OpenAl's ol agent by 12.6% on the difficult Test-C split and surpasses the strongest 32B RL-trained state-of-the-e baseline by 3.6%. These results establish GVPO as a new milestone for RLVR-based training of interactive coding agents. In summary, this work makes the following contributions:
We identify the limitations of current RLVR approaches that rely solely on outcome-verifialle re- wards and highlight the importance of integrating imccsss-vrrfiable signals for precise credit as- signment.
We propose GVPO, the RL algorithm that unifies outcome-verifiabl and process-verifiable signals through an advantage shaping framework.
We demonstrate through extensive experiments in AppWorld that GVPO yields substantial im- provements in stability, convergence, and overall performance, outperforming both open-source and closed-source baselines.
2

---

Published as a conference paper at ICLR 2026
2
PRELIMINARY
2.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
GRPO estimates the advantage in group-relative manner. We denote user request natural lan- guage instruction q. The behavior policy Te samples a group of G individual responses {T,)er The advantage of the i-th response is then computed by normalizing its group-level reward within the sampled set:
A R; meam (
st,((R))
Note that for response 7ú, the computation of A.,: is independent of that is, all timesteps share the same advantage value. GRPO then optimizes a clipped objective:
G
[rd
min a, (0)A()) clip e,, ()
JaRro
G
(0) (T( Ts,ce)
TB (Ti, Øi Ti,ce)
where is a hyperparamettr. In this work, we drop the KI. term; beyond reducing the computational and memory cost of maintaining Tyef. this choice also supported by recent evidence that it can enhance RI Rt-Zero-style training (Liu et al. 2025).
2.2 INTERACTIVE CODING AGENTS
Interactive Coding Agents (ICA) represent new paradigm of intelligent agents that accomplish tasks through iterative interaction with external APIs by executing code snippets (e.g.. Python). Similar to ReAct (Yao al. (023), ICA decomposes the process into two components: reasoning and action. However, unlike ReAct where actions are natural language commands, ICA employs executable code as actions, making more suitable for tool-rich environments (Fig.
Formally, let denote language model, let I be code interpreter that executes code written by Ty to realize tool calls, and let be an input query. Let t denote the i-th sampled trajectory; for notational simplicity we suppress the subscript i. We construct the partial reasoning trajectory at step as
b|k] T1. 01, Tkk Ck, Ok. where r denotes natural language reasoning. çj denotes generated code, and oj is the execution result of cj. The iterative generation process for trajectory 7 follows:
(Tk, (x) =o(- 1 ) Tík - 1]), Ok I(xx), T(k] = r[k ( r L C 0o Here, + indicates sequence concatenation. This cycle continues until the model produces a final answer or the maximum number of reasoning steps K Kmax is reached, with each step informed by previous code-execution results.
Next, we clarify some notation to facilitate the subsequent exposition.
Definition (Index sets over a trajectory). Given trajectory TG define I4 (1...,,1 For each j E I, let Ti.j denote the j-th token in trajectory т;. We partition I, into two disjoint subsets:
I& II, I (. =0,
where I collects the indices of generation tokens (including both reasoning and code tokens), and I collects the indices of observation tokens (execution feedbuck returned by the interpreter). In addition, we introduce two index sets:
Iauco -I!ail== I, nI =
The set Iquce collects the indices of reasoning and code tokens whose corresponding step execution succeeded (no message), while Ihhil collects those whose execution failed (with error mes- sages). Observation indices T are not included in this splis. Thus, every reasoning/code token in the trajectory belongs either to Iauce or to Tfail, but not both.
We distinguish between timestep t, which denotes the t-th generated token (token-level granularity). and reasoning step h, which denotes the k-th reasoning cycle in ICA, consisting of reasoning, code generation, and execution (rk, Ck: o.).

---

Published as a conference paper at ICLR 2026
Bic
Skep
Reword
q
Figure 1: Overview of the proposed GVPO. For each question g. multiple trajectories are sampled to interact with the environment and yield outcome-bast based rewards Rg Outcome-verifiabl advantages A.t derived from these rewards, while process process-verifiable feedback B,, captures step-level suc- cesses and failures. A shaping function f(-) integrates both signals to produce the final advantages Ai. enabling outcome alignment with step-level correction.
APPROACH
In this section, first present the proposed algorithm (Sec. 3.1 followed by a description of the outcome-based reward functions (Sec. 3.2) and the process-verifiable functions (Sec. 3.3 within the AppWorld environment. An overview GVPO is illustrated Fig.
3.1 GROUP VERIFICATION-BASER POLICY OPTIMIZATION (GVPO)
For each question g. GVPO samples group of outputs {T and optimizes the policy via the following objective:
1r {min
(o) G
clip (ar,r Élow, 1 + thigh) Ard (1)
1=1
(i) no( Q: Ti,<t)
Taaa (Te, 9: Ti,<t)
Loss Aggregation. The objective in Eq. aggregates learning signals hierarchically across both the group and sequence dimensions. GVPO adopts the sequence-mean-tee -e---mmen--tttemaal- (smtm) scheme: within each trajectory, token-level contributions first averaged to obtain token mean, and these are then averaged across sequences to yield the final objective.
For comparison, GRPO also employs smim, whereas Dr.GRPO (Liu al. 2025 mean-token-sum (smis) scheme, in which token contributions are summed within each sequence before averaging across sequences, thereby amplifying the influence of longer trajectories. An al- ternative variant, smim, averages at the sequence level first and then distributes uniformly to tokens, yielding a more length-invariant signal. By contrast, DAPO (Yu al. (025) applies a simpler token-mean (fm) strategy, directly averaging token-level signals without additional sequence-level aggregation.
the sequence-
Advantage Shaping. both outcome-verifiable and process-verifialee signals into the advantage estimation:
We introduce the notion of an advantage shaping function, which integrates
Art = (A,, B,,), A, R, mean (f [R. (R)).
where Å is the outcome-verifiable advantage, defined relative to group-based rewards without std normalization terms (Liu al., 2023), and B., denotes process-level feedback derived from execution signals (e.g., compilation status, runtime exceptions, or partial unit-test outcomes). The shaping function f((-) provides general mechanism to modulate A, using deterministic process feedback, thereby calibrating the policy's 's credit assignment. is important to note that after shaping, A no longer preserves the unbiased property (i.e., E[ 0 no longer holds).
Intuition. cutcome-verifiable rewards ensure alignment with final task objectives, while process- verifiable signals serve corrective shaping terms that guide learning finer granularity. When
4

---

Published as a conference paper at ICLR 2026
execution feedback indicates etrly-stage failures, f(-) introduces negative corrections to reduce the likelihood of reinforeing error-prone patterns. Conversely, partial successes yield positive correc- tions, amplifying behaviors that show promising progress before final outcomes are observed.
By shaping the advantage with both outcome-level and process-level information, the policy benefits from more accurate credit assignment across trajectories. This unified formulation ensures that learning is not only guided by final correctness but also by intermediate execution quality, leading to more stable optimization, faster convergence, and stronger performance complex problem-solving environments.
3.2 OUTCOME- VERIFIABLE REWARD FUNCTIONS
In AppWorld, each task is associated with of unit tests that check whether the agent's generated code correctly produces the desired state changes without introducing unintended side effects. We leverage these unit tests to construct an outcome-based reward signal (Chen al. 2025b).
Formally, for a trajectory produced in response to query let {ug}عm denote the M unit tests associated with the task. Each unit test 2 returns a binary result pass( T.) E {0, {0,1}. indicating whether the output passes the test. The outcome-based reward is defined as the fraction of passed tests:
M
R
pass(uj, Tz).
M
where R. [0, 1]. This scalar reward provides direct measure of the final correctness of the generated output. is sparse in nature-nonzero signals are only available once execution has completed-be it captures the ultimate task objective faithfully.
3.3 PROCESS- PROCESS-VERIHIABLE FUNCTIONS
While outcome-based rewards provide only sparse signals at the end of execution, process-verifiable functions deliver dense step-level feedback by differentiating between successful and failed reason- ing/code tokens within trajectory т;. Formally, let I,-er and Iffall denote the index sets corre- sponding to successful and failed reasoning/code tokens, respectively. Based on these index sets, we define process-verifiabl bonus B that adaptively adjusts the step-level advantage Agz:
E Taucc
A
t Isucc
-b
Ifail
A = =0,
A
E Ifail
Å
Bi,
=
ALe
b A
E fhal
A <0,
(1+
-
E Ifail Å O,
- Ai,
otherwise.
0
otherwise.
Here, > fixed penalty coefficient (set 0.2 our experiments). The adjustment mechanism can be interpreted as follows:
Successful tokens (those associated with correct reasoning code execution) retain their outcome-
based advantage Å,, unchanged.
Failed tokens are penalized in a manner sensitive to the trajectory's 's outcome advantage:
When A,, . = 0, a fixed penalty -b is imposed.
When 4 < 0, the negative outcome advantage amplified proportionally.
In all other cases, the advantage is set to zero.
Observation tokens represent environment feedback and are excluded from updates, hence set to zero. Li et al. 2025).
When the advantage is negative, apply multiplicative adjustment to penalize failed trajectories, while also exploring an additive variant ablations. This yields more robust credit assignment under noisy misleading signals.
5

---

Published as a conference paper at ICLR 2026
Scalability. An important advantage of the process veriifabee paradigm is that it is inherently rule- based and does not require training an additional reward model. This property makes it highly scal- able: whenever the environment provides deterministie feedback signals, such as compilation status, runtime errors, constraint checks, or state-transition validations, process-verifialle rewards can be applied directly. While instantiate this idea in the coding domain (AppWorld), the paradigm is not limited to program synthesis. Any environment capable of emitting reliable process-level feed- back can naturally support this form of supervision, enabling efficient scaling across diverse tasks without costly human annotation or reward-mode training.
EXPERIMENTS
Dataset. We evaluate on AppWorld (Trivedi et al., 2024), a benchmark for interactive coding agents that requires multi-turn planning and executable code generation in stateful Python envi- ronment. It integrates nine simulated consumer apps (e.g., email, payments, shopping, file system), exposing 457 APIs across realistic digital activities. The benchmark defines 750 tasks from 250 scenarios, split into train (35/105), dev (20/60), test-normal (56/168), and test-challenge (139/417), with the latter involving unseen apps and more complex planning.
Evaluation. We report results using two key metrics: Task Goal Completion (TGC), the percent- age of tasks in which the agent passes all evaluation tests, and Scenario Goal Completion (SGC), the percentage of scenarios in which the agent succeeds on every associated task.
Implementation. We adopt Qwen2.5- 5-BB-55-BBB--- Instruct as the base model and train it with the veRI. (Sheng et al., 2024) framework, using vL..M (Kwon et al.] 2023) for efficient batched in- ference. Training is restricted to difficulty-1/2 1/2 tasks in AppWorld (72 samples, 24 scenarios) with 8 rollouts per sample, a maximum of 40 interaction turns for training and 50 for evaluation, tempera- ture 1.0 during training (exploration) and 0 at evaluation (deterministic execution). More details can be found in Appendix A.I
Baselines. We compare GVPO against both zero-shot LLMs (GPT-4o, GPT-4 Trb, Llama-3 70B, OpenAl ol, and Qwen2.5-32B-Instteeee Instrucct and RL-trained models (RLOO, GRPO, GSPO, GRPO, and LOOP), all optimized with unit-test- rewards R; E [o. 1]. We select the checkpoint achiev- ing the highest TGC score on the development set. Detailed descriptions of the baselines are pro- vided in Appendix A.2
4.1 RESULT
Main Results. Tab. 2 compares zero-shot prompting models and RI. fine-tuned methods on the AppWorld benchmark. Among the zero-shot models, OpenAl ol achieves the highest performance (61.9 TGC 41.1 SGC on Test-N), but still struggles on the more challenging Test-C split, RL fine- tuning consistently improves performance over prompting. In particular, our method, GVPO, sets a new state-of- of-the-rtt across both splits, achieving 72.6 TGC / 55.4 SGC on Test-N and 49.4 TGC / 28.8 SGC on Test-C. Notably, GVPO surpasses the previously strongest 32B RL baseline, LOOP, by 3.7 points TGC and 2.2 points SGC on Test-C, demonstrating the effectiveness of incorporating mrocess-verifiable signals for credit assignment. While LOOP performs competitively on Test-N, falls short on Test-C, highlighting GVPO's superior generalization to unseen apps and longer multi-step planning tasks, More analyses of the validation set results can be found in Appendix A.3
Entropy Trajectories. Fig. compares the entropy trajectories of GRPO, DAPO, GSPO, and GVPO during training. GSPO shows the fastest entropy decay, collapsing near-deterministic policy, which indicates premature convergence and insufficient exploration.
GRPO and DAPO both maintain moderate entropy levels, but their trajectories still decline steadily, suggesting exploration diminishes over time and leading to potential suboptimal local minima. In contrast, GVPO consistently preserves higher entropy and avoids collapse, reflecting its ability to sustain exploration throughout training. This stability stems from its integration of process-verifiie shaping and asymmetric clipping, which penalize incorrect trajectories without discouraging diver-

---

Published as a conference paper at ICLR 2026
Test-N
Test-C
Para.
Std.
Aggr.
Out.
Proc.
TGC
SGC
TGC
SGC
Prompting with LLM
GPT-4o
48.8
32.1
30.2
13.0
GPT-4 Trb
26.8
12.5
17.5
5.8
OpenAl ol
61.9
41.1
36.7
19.4
LlaMA3
70B
24.4
17.9
7.0
4.3
Qwen2.5
32B
34.5
16.1
18.9
7.9
Fine-tuning with RL
GRPOw/
32B
smim
X
62.5
42.9
38.6
25.2
GRPO
32B
smim
X
61.3
39.3
38.5
21.6
GSPO
32B
smim
X
56.5
33.9
37.6
17.3
DAPO
32B
Un
X
57.1
35.7
38.8
18.0
Dr.GRPO
32B
SMIS
X
63.7
44.6
40.5
18.7
RLOO
32B
X
SmIm
X
60.7
39.3
40.1
20.9
LOOP
32B
SmIm
X
71.3
53.6
45.7
26.6
GVPO
32B
X
72.6
55.4
49.4
28.8
Table 2: Test performance is reported on both the normal (Test-N) and challenge (Test-C) splits of AppWorld, using TGC (Task Goal Completion) and SGC (Scenario Goal Completion). Para.: model parameter scale. Std.: whether std normalization applied advantage computation. Aggr.: loss aggregation scheme, where smam sequence- sequence-mean-tonee mean-token- mean, smts sequence mean-token- sum, and mm token-mean, Out.: use outcome-veriiee verifiable signals. Proc.: use process-verifante signals. GRPOwn denotes GRPO trained with KL regularization. In DAPO, omit dynamic sampling and overlong reward shaping. The best results are shown in bold, and the second-best results are underlined.
sity. As result, GVPO achieves better balance between exploration and exploitation than GRPO, DAPO, and GSPO, contributing to its superior robustness and final performance.
Aug. interaction per task
Feec. fuilures per stap
018
21.30
0.08
0.375
DAPO
17.00
008
0.350
G5PO
0.0
004
0.06
0,02
0.325
002
0tho
CH29F5
0/90
5/o
0.300
free, per step
freq, step
0.38
0.10
0.250
0.25
018
0.09
0.225
00e
0.200
008
Training
(b) Aggregate changes in agent behavior be- tween GRPO, Dr.GRPO, and GVPO, aged over three i.dd rollouts per dev task.
(a) Entropy trajectories of GRPO, DAPO, GSPO, and GVPO during training.
Figure 2: Training dynamics and behavioral comparisons across RL methods.
4.2 ANALYSIS
Shaping Coefficient We further investigate the effect of the shaping coefficient b, which con- trols the relative strength of step-level penalty in GVPO. To analyze its sensitivity, conduct hyperparameter sweep over b e {0.1, 0.2, 0.4) on both Test-N and Test-C benchmarks, and report the results in Tab. 3

---

Published as a conference paper at ICLR 2026
Test-N
Test-C
b
TGC
SGC
TGC
SGC
0.1
66.7
39.3
53.1
30.9
0.2
72.6
55.4
49.4
28.8
0.4
67.9
44.6
48.0
28.1
Table 3: Sensitivity analysis of the shaping coefficient b in GVPO on Test-N and Test-C. Perfor- mance comparison under different values of b, showing that b = 0.2 achieves the best overall and most stable results across both benchmarks.
Overall, we observe that 6 0.2 achieves the best and most consistent performance across all set- tings, and is therefore adopted our main experiments. When b = 0.1, the step-level correction becomes too weak, leading to insufficient credit assignment and degraded scenario-level perfor- mance. contrast, a larger value = 0.4 tends to over-penalize failed intermediate steps, which introduces mild training instability and does not yield further improvement in final task success.
Across the evaluated range, GVPO exhibits relatively stable performance, indicating that the method moderately sensitive to and does not rely fine-grained tuning of this hyperparameter.
Dev
Test-N
Test-C
Setting
TGC
SGC
TGC
SGC
TGC
SGC
GVPO
84.2
73.7
72.6
55.4
49.4
28.8
tm
73.7
52.6
58.3
35.7
34.6
15.1
Aggr.
smiS
75.4
63.6
64.9
46.4
42.8
23.7
Clip
Sym.
71.9
52.6
60.1
39.3
40.5
21.6
Std.
77.2
68.4
70.8
48.2
42,8
23.7
mbs # gs
77.2
62.5
66.7
42.9
43.1
25.9
Other
additive shaping only
73.7
57.9
64.9
46.4
39.8
23.0
Table 4: Ablation study of GVPO. Performance under different design choices is reported on the AppWorld Dev, Test-N, and Test-C splits. Variants include alternative loss aggregation schemes asymmetric clipping, std nor-
(mu=token-meet smtts=squuence seaeeee-mmatttok---uuue malization (Std.), and additional settings such as mismatched micro-batch/es batro-bbtch/group sizes (mbs + gs), and additive shaping only (i.e., A,e = A,e - b, e Ifail A A, < 0).
symmetric
Ablation Result. Tab. presents the ablation results of GVPO across the App dev, Test- N, and Test-C splits. The full GVPO consistently achieves the strongest performance, confirming the effectiveness of its combined design. Alternative aggregation schemes (token-mean, sequence- mean-token-ssum) lead to noticeable drops in Test-C, suggesting that GVPO's sequence-mean-ttkee- mean formulation provides better trade-off between stability and credit assignment. Symmetric clipping underperforms, validating the benefit of the asymmetric "elip-higher" strategy for main- taining exploration. Similarly. applying std normalization (Std.) degrades performance, suggesting that removing variance scaling avoids optimization bias and better preserves the reward signal.
For mismatched micro-batch/group sizes (mbs=8, gs=6), we observe a larger performance drop here compared prior math-reasoning tasks, likely because AppWorld tasks involve diverse API calls and state transitions where imbalance in batch-wise normalization amplifies variance in gradient estimates, making training less stable.
In GVPO, for failed tokens with negative outcome advantages we apply multiplicative shaping,i.e., A,, = (1+b)- (11+))-A,, A,, for t E Zffiil AA < 0. In the additive-only variant, this scaling is replaced by constant penalty, i.e., A,, A. b for the same index set. This substitution weakens the balance between outcome- and process-level signals and leads to poorer generalization, underscoring that the choice of advantage shaping critical; we leave deeper investigation to future work.
8

---

Published as a conference paper at ICLR 2026
Task: Label all email threads my Gmail inbox from notifications@--app>.com with the label of the respective app. Ignore spam and archived ones.
GRPO
Dr.RPOO
GVPO
Read apps & api descriptions
Check account & password
Read apps & api descriptions
Gmail login with fake password
Gmail login with wrong api
Check account & password
Check account & password
Read apps & api descriptions
Read apps api deseriptions
Gmail login with wrong para.
Read api docs
Gmail login
Read upi does
Gmail login
Read api docs
Gmail login
Read api docs
Check email
Read api docs
Check email
Read api docs
Label email
Figure 3: Case study on the Gmail labeling task. Compared with GRPO and Dr.GRPO, the GVPO- trained agent exhibits more cautious strategy by extensively consulting API descriptions and doc- umentation before executing concrete actions.
Agent Behaviors. Fig. 2 summarizes the behavioral characteristics of agents trained with GRPO, Dr.GRPO, and GVPO across four measures: (i) average number of interactions per task, (ii) exe- cution failure probability per step, (iii frequency of show api does calls, and (iv) frequency of show api descriptiin ons calls. The results reveal clear behavioral differences among the methods. GVPO achieves the lowest failure probability, while simultaneously exhibiting the high- est frequency of documentation queries. In fact, nearly half of the steps taken by GVPO agents suggesting that step-
involve consulting either s show_api ddoc or show www.wap.cco wew.ddescriptioon level penalties discourage risky or poorly informed actions and instead incentivize information gath- ering before execution. This behavior reflects a clear shift toward a gquery-before-acts act"" strategy: as invalid or high-risk actions are explicitly penalized gain relative utility, reducing the need for corrective loops caused by earlier mistakes. Importantly, this more cautious decision-making does not incur excessive interaction overhead. GVPO requires slightly more steps than Dr. GRPO but fewer than GRPO, which often accumulates longer trajecto- ries due to repeated error recovery.
the step level, documentation- -ueroynuuuuuuuuueeeeee actions
From a credit assignment perspective, fine-grained process feedback allows GVPO to identify and penalize specific failure modes more precisely than scenario-level rewards alone, enabling more targeted policy updates. Overall, these results indicate that GVPO suppresses unsafe behavior, pro- motes structured information seeking, and shifts the agent toward more deliberate, verify-then-act reasoning, while preserving sufficient interaction efficiency and diversity.
Case Study. Building on these quantitative findings, we next provide qualitative case study in Fig. 3 In contrast to GRPO and Dr.GRPO, which often make invalid attempts such as logging into Gmail with incorrect parameters cautious strategy. Nearly half of its tool invocations are devoted to querying documentation be- fore committing concrete actions (e.g., Gmail login, reading, labeling emails). This behavior directly aligns with the trends observed in Fig. 2b offering complementary evidence that GVPO encourages agents to act more carefully and reliably by consulting external resources prior to exe- cution.
invoking the wrong API, the GVPO adopts markedly more
5 RELATED WORK
), research the RLVR paradigm has accelerated significantly (Wen et al. 2025 Xie et al. 2025). This line of work covers diverse dimensions such as training data curation (Wang al. 2024). objective formulation (Liu et al. 2025). hyperparameter
RLVR.
Since DeepSeek-RR R1 (Guo et al.)
2025),

---

Published as a conference paper at ICLR 2026
optimization Yu al. 2025 base model selection (Hu et al. 2025 and empirical insights Yue jet al. 2025). Verified rewards in prior work are typically derived from deterministic outcome checks (e.g., exact match in math (Cobbe et al. (02T)) , unit tests in coding (Austin et al.) 202T). based
verification with tools Li et al. (202)): Qian et al. (2025), LLMM-bsed verifiers (Wen et al. 2025
Chen al. 2024). logic-based verifiers (Wang al.) 2025) domain-specific reward models (Su
et al. 2025 The related work GiGPO (Feng et al. 2025) employs an additive shaping function to integrate step-relative advantages but does not leverage intermediate process feedback. In contrast, we introduce more general advantage shaping framework that unifies both additive and multiplica- tive formulations, and validate its effectiveness on the more challenging App World benchmark.
RL for LLM agents. A complementary line of research investigates tool-use learning Yao al. where agents are trained to interact with external environments through APIs (Qin et al.
2023
2024
code execution (Li al. 2025), multi-turn reasoning (Wei et al. 2025 Xi et al. 2025
Da et al. 2025 Mai et al. 2025). ). The applications span text-based games (Narasimhan et al,
2015 Yao et al. 2020 Carta et al. 2023), web navigation and shopping (Yao al. 2022), mobile
device interaction (Bai et al. 2024), and embodied tasks (Zhai et al. (024). yet our work focuses
on AppWorld (Trivedi et al. 2024), significantly more challenging benchmark that requires long- horizon, multi-app interactions. The most related effort is LOOP (Chen et al. (0255), but unlike LOOP, GVPO incorporates signals from intermediate execute feedback, enabling more accurate credit assignment and improved robustness.
6 CONCLUSION
We propose Group Verification- baadee Policy Optimization (GVPO), a reinforcement learning algo- rithm that unifies autcome-verifiable and process-verifiabl verifiable signals through an advantage shaping framework. By leveraging intermediate execution feedback alongside final task outcomes, GVPO achieves more accurate credit assignment, resulting in greater training stability, accelerated conver- gence, and improved generalization in complex interactive environments. Empirical results on the challenging App World benchmark demonstrate that GVPO not only surpasses strong RL basclines but also closes the gap with much larger proprictary systems, highlighting its potential scalable approach for training stateful, multi-turn LLM interactive agents.
Limitations. GVPO currently relies on deterministic environments with well-defined process sig- nals, and its effectiveness on tasks with ambiguous or noisy feedback remains underexplored. Ex- tending GVPO to broader domains and integrating it with richer supervision sources are important directions for future work.
ACKNOWLEDGMENTS
The authors wish to thank the reviewers for their helpful comments and suggestions. This work was supported by National Key R&D Program of China (No. 20252D1801501) NSFC grant (No. 62136002 and 62477014), Ministry of Education Research Joint Fund Project (809) BB042299,, Shanghai Knowledge Service Platform Project (No. ZF1213).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker, Back to basics: Revisiting reinforce-style optimization for learn- ing from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248 12267, 2024.
Hongjun An, Wenhan Hu, Sida Huang, Siqi Huang, Ruanjun Li, Yuanzhi Liang, Jiawei Shao, Yil- iang Song, Zihan Wang. Cheng Yuan, et al. Ai flow: Perspectives, scenarios, and approaches. Vicinagearth, 3(1):1, 2026.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint 0000:2222877 2021.
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in 2024.
Neural Information Processing Systems, 37:12461-111
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Y Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International conference on machine learning, pp. 307633777171 PMLR, 2023.
Jinyang Chen, Haolun Wu, Jianhong Pang, Yihua Wang, Dell Zhang, and Changzhi Sun. Tool learning with language models: a comprehensive survey methods, pipelines, and benchmarks. Vicinagearth, 2(1):16, 2025a.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with llms. arXiv preprint arXiv:2412.18925. 2024.
Kevin Chen, Marco Aasumano-Townett Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive IIm agents. arXiv preprint arXiv:2502.01600 2025b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint a 10. /4168, 2021.
Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent-rlvr: Training software engineering agents via guidance and environment rewards. arXiv preprint arXiv:2506. 110.2500.11425, 2025.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in irouupin-group group policy optimization for Ilm agent training. arXiv preprint 1005922505.10978 2025.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501.12945 2025.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Heung-Yeung Shum. Open-reasoner-zeero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXhv:2503.242944 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 61 1-666, 2023.
Xinyi Li. Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey Ilm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9, 2024.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint 2xXiv:2503.23383 2025.
11

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503,2078: 2025.
Xinji Mai, Haotian Xu, Weinong Wang, Jian Hu, Yingying Zhang, Wenqiang Zhang, et al. Agent rl scaling law: Agent rl with spontaneous code execution for mathematical problem solving. arXiv preprint arXiv:2505.07773. 2025.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1-11, 2015.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tüi, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint urXiv:2504.33956 2025.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In ICLR, 2024.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Mohit Shridhar, Xingdi Yuan, Alec-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint ar XXxv:2010.00766. 2020.
Honghao Song, Liang Wang, Xiaozhen Qiao, Yifan Chen, Da Sun, and Zhe Sun. Embodied intelli- gence for robot manipulation: development and challenges. Vicinagearth, 2(1):8, 2025.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.2388 2025.
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16022- 16076, 2024.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426-9439, 2024.
Xinyu Wang, Changzhi Sun, Lian Cheng, Yuanbin Wu, Dell Zhang, Xiaoling Wang, and Xuelong Li. Logic-regularized verifier elicits reasoning from 1ms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32617- 32630, 2025.
Yuan Wei, Xiaohan Shan, Ran Miao, and Jianmin Li. Agent": An agent-geneeaates-agent framework for reinforcement learning automation. arXiv preprint urXiv:2509. 13368, 2025.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llss. arXiv preprint arXiv:2506. /4245, 2025.
12

---

Published as a conference paper at ICLR 2026
Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Jun- jie Ye, Jiazheng Zhang, Wenxiang Chen, et al. Agentgym-rl: Training llm agents for long- long-horizon decision making through multi-turn reinforcement learning. arXiv preprint arXtv:2509.08755, 2025.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint 10XXi:::502.1770000 2025.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games. In Proceedings of the 2020 Confer- on Empirical Methods in Natural Language Processing (EMNLP), pp. 8736-8767777 8754, 2020.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35-20744-20757, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint MxXv:2503.114470 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504. 130.:25504.3837,, 2025.
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong. Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems, 37:110935-110971, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 HYPERPARAMETERS AND TRAINING SETUP
Setting
Value
Base setup
Base model
Swen2.5533B-Instruu veRL FSDP2 + vLLM (batched inference)
Framework
Training tasks
AppWorld difficulty- 1/2 (72 samples, scenarios)
Rollouts per sample
Max turns (train / eval)
40 50
Temperature (train / eval)
1.0 1.0/0.0
GVPO hyperparameters
Group size G
Clipping range (Ciow; Enigh)
(0.2, 0.28)
Learning rate
1 x 10 10-5(comm (constannt))
Batch size
4 8 512 tokens
Max sequence length per
Entropy coefficient
Advantage shaping penalty
0.2
Optimizer
AdamW
Training steps
144
Mirco batchsize
Table 5: Hyperparameters and training setup for GVPO experiments in AppWorld.
Training setup. We conduct all experiments GPUs. Our training framework is based on verl with FSDP for efficient distributed training. For data generation, we employ vLLM to perform rollouts. After collecting trajectories, we recompute the per-token log-probabilities under the generating policy, rather than using the values directly reported by vIILM.
single machine equipped with 8 NVIDIA H100
Agent setup. All agents are prompted in a ReAct-style format, which includes one in-context example of successful task execution. The agent receives as input the results of code execution API API call outputs or exception traces), together with the original task instruction. At each tur,, the agent is allowed generate up to 512 tokens in total. This limit covers both reasoning tokens echain-of-thoughi and code. If an API response exceeds 3K tokens, it is truncated, and the agent is provided with a short note indicating that truncation has occurred.
World setup. During training, we launch 32 independent AppWorld backend services in ad- vance. The training framework communicates with these AppWorld backends via a Redis message queue, which serves the central communication hub. Each backend service is bound to umique port, and port numbers are used to associate each App World instance with its corresponding tra- jectory. This design ensures that multiple trajectories be executed in parallel while avoiding interference across different AppWorld backends.
Training Hyperparameters. We use a constant learning rate of 10-5 and clip the gradient norm to in all experiments. Each training sample produces 8 rollouts with temperature 1.0. To accelerate training, we apply early stopping to rollout collection: rollout generation terminated once at least 6 rollouts have been collected for each task and 90% of the total rollouts have been collected. Concretely, we consider two stopping conditions. First, within each group of 8 rollouts for given task, rollout collection ends once 6 rollouts have finished. Second, across the 32 sampled tasks, rollout collection terminates once 30 tasks have completed. Early stopping is only applied after the model has generated at least 30 steps of rollout, ensuring sufficient exploration before termination. We allow up to 40 interactions between the agent and the environment during training and up to 50 for evaluation. Any episode that does not complete within this interaction budget considered failure. If the sequence reaches the model's context window limit, the rollout is terminated (Tab. 5
14

---

Published as a conference paper at ICLR 2026
Dev
Para.
Std.
Aggr.
Out.
Proc.
TGC
SGC
Fine-tuning with RI.
GRPO
32B
smim
75.4
63.2
GRPO
32B
/
SmRtn
/
X
77.2
57.9
GSPO
32B
SmRtm
J
63.2
42.1
DAPO
32B
TI
J
X
68.4
47.4
Dr.GRPO
32B
X
STRIS
77.2
68.4
RLOOw/R
32B
X
STRIm
X
68.4
57.9
GVPO
32B
X
STRim
84.2
73.7
Table 6: Comparison of RL fine-tuning methods the AppWorld Dev set. Para.: model parameter scale. Std.: whether std normalization is applied advantage computation. Aggr. loss aggregation scheme, where smim sequence mean- weean-token-ment smis sequence-ment seaeeee-mma-ttk---ummme and rm token-mean. Out.: use of outcome-verifiable signals. Proc.: use of process-verifiable signals. GRPO denotes GRPO trained with KL regularization. The best results shown in bold, and the second-best results are underlined.
A.2 BASELINES
This section provides a detailed overview of the reinforcement learning (RI) algorithms evaluated in our experiments. Each method represents different strategy for variance reduction, credit signment, or stability enhancement policy gradient optimization.
1. RLOO (Reinforce Leave-One-Out). Builds upon the REINFORCE estimator by introducing each rollout group. This design reduces variance in advantage estima- tion compared to vanilla REINFORCE, leading to more stable updates without requiring additional learned value functions.
leave-one-out baseline within
2. GRPO (Group Relative Policy Optimization). Computes relative advantages by normalizing re- wards within rollout group, thereby stabilizing training against reward scale fluctuations. GRPO has become standard RLVR approach for L.LM fine-tuning. Variants may additionally incorporate KL regularization with respect to the base model to control divergence.
3. GSPO (Group Sequence Policy Optimization). Moves from token-level to sequence-level opti- mization by defining the importance ratio the trajectory level. It applies sequence-level clipping, which simplifies optimization and reduces variance. This approach has demonstrated strong perfor- mance and efficiency, particularly in recent Qwen3 models.
4. Dr.GRPO (GRPO Done Right). Addresses biases in GRPO by (i) removing the normalization with group standard deviation during advantage computation, and (ii) modifying the loss aggrega- tion scheme to smts. These changes improve token efficiency and reduce optimization bias, while preserving the reasoning capability of the model.
5. DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization). Combines clip-higher for exploration and token-level loss for fin-grainee credit assignment; we omit dynamic sampling and overlong reward shaping.
6. LOOP Leave-One-Our PPO). Extends RLOO by adopting a PPO-style optimization procedure, applying multiple epochs of clipped updates per batch. This combination improves exploration and enhances policy robustness not open-sourced, rely the reported results from the publication, which are available only for the Test set.
interactive tasks. As the original implementation was
A.3 VALIDATION RESULT
Tab. presents the updated validation results on the Dev split for all RL fine-tuning methods. Our method, GVPO, achieves the best overall performance, reaching 84.2 TGC and 73.7 SGC, out- performing all baselines by a clear margin. The improvements are substantial: compared with the strongest baseline Dr.GRPO (77.2 TGC / 68.4 SGC), GVPO yields gains of +7.0 TGC and +5.3 SGC. This demonstrates that incorporating process- level verifiable signals provides consistent ben- efits both task-level completion and scenario-level consistency.
15

---

Published as a conference paper at ICLR 2026
Among baselines, Dr.GRPO delivers the strongest performance, tying for the highest TGC (77.2) and achieving a relatively high SGC (68.45 4), indicating that its modified loss aggregation scheme and the removal of standard deviation normalization in advantage computation contribute more stable and effective optimization. Standard GRPO achieves competitive TGC (77.2) but exhibits noticeably lower SGC (57.9), suggesting that outcome-only supervision is insufficient to ensure scenario-level coherence. Adding KL regularization to GRPO slightly reduces TGC (75.4) but improves SGC (63.2), reflecting trade-off between exploration and stability. Similarly, GSPO and DAPO achieve comparatively lower results. GSPO replaces token-level importance sampling with sequence-level importance sampling. and DAPO adopts token-ment loss aggregation with clip-high strategy; these design choices lead to more moderate improvements on this setting.
Overall, the Dev results consistently show that explicitly leveraging process-verifiabe feedback leads to stronger and more balanced improvements across both metrics. The clear margin on the development set further confirms that GVPO provides a more informative and stable optimization signal during training, which aligns with the trends observed on the Test split.
A.4 GENERALIZATION BEYOND APPWORLD
To further examine the generalization ability of GVPO beyond the AppWorld environment, we con- duct additional experiments on two widely used interactive reinforcement learning benchmarks: ALFWorld and WebShop. These environments differ substantially from AppWorld in task structure, action space, and intermediate feedback signals, providing a rigorous test of cross-domain robust- ness.
Experimental Setup For both ALFWorld (Shridhar et al. 2020) and WebShop (aao et al. 2022 we adopt unified experimental setting. We train Qwen2.5 BBInntruct using GVPO and GRPO for 100 training steps under identical hyperparameters. For each method, the best-performing check- point is selected based on validation performance and evaluated on corresponding test set. Dur- ing interaction, if the environment returns "nothing happens" or the model produces invalid or improperly formatted action, the step is treated as a failure.
Results on ALFWorld Tab. reports the performance on ALFWorld across different task cate- gories. GVPO consistently outperforms GRPO by large margin in overall success rate as well as in most individual task types. Notably, GVPO achieves strong gains on challenging categories such as Clean, Heat, and Pick2. Moreover, GVPO substantially surpasses strong proprietary systems, including Gemini-2.5-Pa and GPT-4o, in overall performance.
These results indicate that GVPO effectively leverages step-level feedback to improve long-horizon decision making in embodied interaction tasks.
Pick
Look
Clean
Heat
Cool
Pick2
All
GPT-40
75.3
60.8
31.2
56.7
21.6
49.8
48
Gemini-2.5-pm
92.8
63.3
62.1
69
26.6
58.7
60.3
GRPO
85.3
86.7
80
15.4
65
56.2
70.3
GVPO
89.2
57.1
95.8
80
85.7
82.4
84.4
Table 7: Experiment result ALFWorld
Results WebShop We further evaluate GVPO on the WebShop benchmark, which involves web-based shopping interactions with sparse and noisy feedback. As shown in Tab. g GVPO again outperforms GRPO and leading proprietary models. In particular, GVPO achieves higher suc- cess rate than both GPT-4o and Gemini-2. 2.5-Poo, demonstrating consistent advantages in a markedly different interaction setting.
Analysis The above results suggest that the effectiivness of GVPO is not limited to coding- oriented or synthetic environments. Many real-world interactive tasks naturally produce step-level failure signals, such invalid navigation actions, failed API calls, or inconsistent environment
16

---

Published as a conference paper at ICLR 2026
Score
Success rate
GPT-4o
31.8
23.7
Gemini-2.5-pro
42.5
35.9
GRPO
79.7
70.3
GVPO
90.2
80.5
Table 8: Experiment result on Webshop
states. GVPO explicitly exploits these signals through step-level advantage shaping, enabling more accurate credit assignment and safer policy updates.
Overall, these additional experiments confirm that GVPO generalizes robustly across agents, datasets, and domains, and provides consistent performance gains beyond the original AppWorld setting.
A.5 EXAMPLE OF APPWORLD
Fig. illustrates representative trajectory from the AppWorld Spotify task, showcasing how the agent interacts with APIs in a multi-step reasoning process. The agent first queries the supervisor's s profile and stored account passwords, then uses this information successfully authenticate with the Spotify API and obtain an access token. With the valid token, it proceeds to call the recommendation API to retrieve music suggestions. This example highlights how the agent effectively integrates intermediate signals, such execution success, to guide subsequent actions, ensuring correctness and reliability in a complex, stateful environment.
A.6 LLM USAGE
In accordance with the ICLR policy on the use of large language models (LLMs), we report that LLMs were used exclusively for language polishing and grammar refinement in this paper. LLMs were not used for designing experiments, conducting analyses, or generating research content. All research ideas, implementations, and results are the work of the authors.
17

---

Published as a conference paper at ICLR 2026
User Instruction:
Name the artist most recommended to me on Spotify.
Okay, let's find the password to login Spotify.
supervisor_profile show_profiile secevissor_psssordss = ip("acccontt_name"" p(" passwoor""" for in show account (asrount_passwords]]]
... 'Suuurrisss'''s profile')...
Ok, now we have the profile of supervisor and get the password, we can login now.
access _okessstoke-e-e = spotify.login(
usemameessueervioooooooio "atpttroooooooeeeiii""
passwordssperrsssrrpassssssssss ["aacess__okee)
Execution success
Next, let's find the artist recommended to supervisor on Spotify.
rrcmmeeddtiossoooo.rrcmmmmmmtdds to serraccess_token
API
Figure 4: Example trajectory from AppWorld's Spotify task.
18


Reviews:
[{"review_id": "IZ8Nn9nDdF", "reviewer": "Reviewer_1bHw", "summary": "This paper proposes Group Verification-Based Policy Optimization (GVPO), a new reinforcement learning (RL) algorithm designed to improve the training of large language model (LLM)–based interactive coding agents. GVPO extends prior Group Relative Policy Optimization (GRPO) by introducing an advantage shaping framework that combines two complementary reward types:\n\nOutcome-verifiable rewards -- reflecting final task correctness (e.g., unit-test results);\n\nProcess-verifiable signals -- reflecting intermediate feedback such as syntax errors or runtime exceptions.\n\nThe method shapes per-step advantages using both signals to achieve finer credit assignment and more stable optimization. Evaluations on AppWorld, a challenging multi-turn code-execution benchmark, show that a 32B-parameter GVPO agent outperforms OpenAI’s o1 by 12.6% on the hardest test split and the strongest 32B RL baseline (LOOP) by 3.6%. Analyses indicate better exploration stability, reduced execution errors, and more cautious agent behavior.", "questions": "1. Can this method be applied to other problems beyond coding agent RL training?\n2. How sensitive is GVPO to the choice of shaping coefficient b? Could adaptive or learned shaping improve stability across domains?\n3. Could the asymmetric clipping mechanism interact with shaping to introduce bias in long trajectories?\n4. How well does GVPO scale beyond 32B parameters or to more diverse environments such as web or embodied agents?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The integration of process-verifiable signals addresses the sparsity and delay of traditional outcome-only RL signals, providing a principled improvement to credit assignment.\n2. GVPO achieves state-of-the-art performance among open 32B-scale agents on AppWorld, significantly surpassing both outcome-only RL methods and prompting-based models.\n3. Ablation and behavior analyses (entropy, cautiousness, failure rates) convincingly support the claimed benefits in stability and exploration.\n4. Since process-verifiable feedback is rule-based, GVPO avoids additional reward-model training and can generalize to other deterministic environments.", "weaknesses": "1. Experiments focus solely on the AppWorld coding environment; it remains unclear how GVPO performs in non-deterministic or partially verifiable domains (e.g., reasoning or natural dialogue tasks).\n2. Lack of comparison to LLM-based verifiers: Recent RLVR works using LLM verifiers are not empirically compared, which limits positioning relative to that line of research.\n3. While the paper observes “cautious behavior,” it lacks deeper qualitative discussion of how process feedback influences reasoning patterns.\n4. The method assumes reliable step-level feedback (syntax/runtime correctness), which may restrict applicability to highly structured environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FLwW3JOjBY", "reviewer": "Reviewer_N8jC", "summary": "This paper introduces Group Verification-based Policy Optimization (GVPO), a novel RL from Verifiable Rewards (RLVR) algorithm designed to enhance the training of interactive coding agents. The core insight is that existing RLVR algorithm rely solely on sparse, outcome-verifiable rewards (e.g., final unit test results), leading to inaccurate credit assignment across an agent's reasoning trajectory. GVPO addresses this by integrating dense, process-verifiable feedback according to the advantage estimation directly into the optimization objective. Experiments shows that GVPO outperforms existing RLVR algorithms on AppWorld, and Qwen2.5-32B-Instruct trained with RLVR outperforms than zero-shot commercial LLMs.", "questions": "- Why can GVPO prevent the entropy decay in Figure 2?\n\n- Considering that the number of training samples is only 35, I wonder how many gradient steps the training has?\n\n- The proposed shaping function for $B_{i,t}$ and the resulting $A_{i,t}$ is complex and contains several case-based rules. Designing such function for each secnario is diffcult and may cause reward hacking.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow and well-organized.\n\nThe paper is well-motivated. Constructing step-level signals are of great importance for RLVR algorithms, especially for long-term hard tasks.\n\nSome empirical results are compelling. Achieving a new SOTA on a complex benchmark and outperforming a model like OpenAI's o1 by a significant margin (12.6% on Test-C) with a 32B model is a strong testament to the method's efficacy.", "weaknesses": "One of the main claimed contribution, process-verifiable feedback derived from intermediate execution traces (e.g., syntax errors, runtime exceptions), are not implementated in the algorithm.\n\nThe proposed shaping function for $B_{i,t}$ and the resulting $A_{i,t}$ is complex and contains several case-based rules. Designing such function for each secnario is diffcult and may cause reward hacking [1].\n\nThe experiments are limited to one agent type(CodeAct), one scenario (AppWorld), and one model (Qwen2.5-32B-Instruct), which can not prove the generalization of GVPO. No evidence is given that GVPO helps in other widely-used agent domains (e.g., WebArena, SWEBench).\n\nAll experimental results are point estimates from a single run. It is unclear whether the +3.6 % TGC gain over LOOP is statistically significant.\n\nReference: \n[1] Guo D, Yang D, Zhang H, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning[J].Nature, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YncweZErTN", "reviewer": "Reviewer_JpF7", "summary": "This paper proposes Group Verification-based Policy Optimization (GVPO), a novel reinforcement learning algorithm that integrates outcome-verifiable rewards with process-verifiable feedback. By leveraging both long-term task alignment signals and intermediate corrective signals, GVPO improves credit assignment, optimization stability, and generalization in complex interactive environments. Experimental results demonstrate that a 32B-parameter agent trained with GVPO outperforms strong baselines, including OpenAI's o1 agent, particularly on challenging benchmarks like Test-C in the AppWorld environment.", "questions": "1. How does GVPO scale computationally when applied to larger models or more complex environments? Are there practical limitations or bottlenecks?\n2. Could GVPO benefit from additional architectural modifications or optimizations specific to larger-scale models (e.g., 64B or beyond)?\n3. Would GVPO’s advantage shaping framework generalize to other interactive environments beyond AppWorld? Any insights into potential domain-specific challenges?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed GVPO framework is novel and well-motivated, with a clean and intuitive formulation that combines outcome-verifiable and process-verifiable feedback.\n2. By incorporating step-level successes and failures through process-verifiable signals, GVPO effectively constrains optimization, leading to more reasonable and guided learning. The experimental results convincingly support this claim.\n3. The paper presents a comprehensive experimental setup, including appropriate ablations and a sufficiently large model size (32B), which adds credibility to the findings.", "weaknesses": "1. The paper does not sufficiently discuss the computational cost of GVPO, particularly in terms of scalability to larger models or environments.\n2. The experiments are limited to a 32B-parameter model, leaving open the question of how GVPO would perform with larger-scale models, especially given the trend toward scaling in reinforcement learning research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fuhw4xzCGK", "reviewer": "Reviewer_34BC", "summary": "The paper proposes GVPO, a reinforcement learning method for interactive code agents. The approach integrates outcome-verifiable signals and process-verifiable signals (e.g., syntax errors, runtime exceptions, partial test passes) into a shaped advantage function, and further stabilizes training through asymmetric clipping and sequence-mean–token-mean aggregation, which improve stability and length fairness.\nIn the AppWorld environment, GVPO achieves consistent improvements over baselines such as GRPO, Dr.GRPO, DAPO, and LOOP, and provides several ablation studies.", "questions": "- How is the shaping coefficient b chosen, and how sensitive is performance to it? \n\n\n- How sensitive are the results to the chosen upper and lower bounds in asymmetric clipping?\n\n\n- The authors note that the KL term is removed for GRPO (only comparing with GRPO-w/kl), but the effect of GVPO + KL has not been evaluated.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The stability design is well thought out: asymmetric clipping and two-level aggregation reduce gradient explosion and sequence-length bias.\n\n2. Experiments are relatively comprehensive within a single environment, including main tables and multiple ablations with consistent trends, and the setup appears reproducible.", "weaknesses": "- I am not an expert in this subfield, but from my perspective the conceptual novelty is limited: combining intermediate and final feedback for reward shaping is rather straightforward within the RLVR framework (Code Agent).\n\n- The theoretical analysis is insufficien, the paper lacks quantitative discussion of bias/variance changes and convergence properties introduced by advantage shaping.\n\n- Empirically, GVPO performs comparably to LOOP on the normal splits and shows moderate advantage on the challenge split.\n\n- The related-work section does not provide a sufficiently comprehensive overview of prior works, making the claimed contributions difficult to clearly position.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)